Vision-assisted prosthetic control method, device, prosthesis and storage medium
Through visual assistive technology, point cloud information and motion parameters are used to generate terrain contour maps, adjust joint angles, solve the problem of the lower limb prosthesis kicking obstacles when walking, and improve walking stability.
Patent Information
- Application Number
- CN202310968147.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-31
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2043-07-31
AI Technical Summary
When walking, the lower limb prosthesis is limited in contact and feedback, and it is easy to kick obstacles, resulting in low walking stability.
Using a visual assisted control method, the environmental information and prosthetic motion parameters are obtained through the point cloud acquisition device, the topographic profile map is generated, the actual distance between the connecting rod and the obstacle is calculated, and the joint angle is adjusted to avoid collision.
Effectively avoid prosthetic kicks into obstacles and improve walking stability.
Smart Images

Figure CN116869713B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of computer technology, and in particular, to a prosthesis control method, device, prosthesis and storage medium based on vision assistance. Background Art
[0002] In recent years, with the advancement of science and technology, lower limb prosthetic technology has been actively developed. Lower limb prostheses have provided great help for amputees to restore their walking ability and improved their quality of life.
[0003] In related technologies, the design of lower limb prostheses usually focuses on drawing on biological structures and functions, and is designed biomimetically by imitating the bone and muscle structure of the human leg. However, the feedback on contact perception is often limited. When amputees walk with lower limb prostheses, there is a risk that the lower limb prostheses will kick obstacles, resulting in lower walking stability. Summary of the invention
[0004] The embodiments of the present application provide a prosthesis control method, device, prosthesis and storage medium based on vision assistance, which can solve the problem in the related art that the lower limb prosthesis has limited feedback on contact perception, there is a risk of kicking obstacles, and the walking stability is low.
[0005] A first aspect of an embodiment of the present application provides a prosthesis control method based on vision assistance, comprising:
[0006] Acquire environmental point cloud information of the environment in which the prosthesis is located from the point cloud acquisition device, and acquire motion parameters of the prosthesis and a rotation angle of the point cloud acquisition device from the sensor component, wherein the motion parameters include force parameters of the prosthesis and current joint angles of joints on the prosthesis;
[0007] Generate terrain contour map based on rotation angle and environmental point cloud information;
[0008] If the motion parameter indicates that the prosthesis is in a swinging state, then determining the actual distance between each prosthesis link and the target obstacle according to the terrain contour map, the current joint angle of the joint and the length of each prosthesis link, wherein the target obstacle is the obstacle indicated in the terrain contour map;
[0009] If there are prosthetic links whose actual distances are less than or equal to the preset distance threshold, the target joint angles of the joints are determined according to the actual distances corresponding to the prosthetic links, and the joints are controlled to adjust from the current joint angles to the target joint angles.
[0010] A second aspect of an embodiment of the present application provides a prosthetic control device based on visual assistance, comprising:
[0011] An information acquisition unit is used to acquire environmental point cloud information of the environment in which the prosthesis is located from a point cloud acquisition device, and to acquire motion parameters of the prosthesis and a rotation angle of the point cloud acquisition device from a sensor component, wherein the motion parameters include force parameters of the prosthesis and current joint angles of joints on the prosthesis;
[0012] A contour generation unit, used for generating a terrain contour map according to the rotation angle and the environmental point cloud information;
[0013] a distance determination unit, for determining, if the motion parameter indicates that the prosthesis is in a swinging state, actual distances between each prosthesis link and a target obstacle according to the terrain contour map, current joint angles of the joints, and lengths of each prosthesis link, wherein the target obstacle is an obstacle indicated in the terrain contour map;
[0014] The angle adjustment unit is used to determine the target joint angle of the joint according to the actual distance corresponding to each prosthetic link if there is a prosthetic link whose actual distance is less than or equal to the preset distance threshold, and to control the joint to adjust from the current joint angle to the target joint angle.
[0015] A third aspect of an embodiment of the present application provides a prosthesis, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, each step of the visual-assisted prosthesis control method provided in the first aspect is implemented.
[0016] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the visual-assisted prosthesis control method provided in the first aspect are implemented.
[0017] The implementation of the visual-assisted prosthesis control method, device, prosthesis and storage medium provided in the embodiments of the present application has the following beneficial effects: when the motion parameters of the prosthesis indicate that the prosthesis is in a swinging state, the actual distance between each prosthetic link and the target obstacle is obtained through the terrain contour map, the current joint angle of the joint in the prosthesis and the length of each prosthetic link; when there is a prosthetic link whose actual distance is less than or equal to a preset distance threshold, the target joint angle of the joint is determined through the actual distance corresponding to each prosthetic link, and the joint is controlled to adjust from the current joint angle to the target joint angle, so that the joint in the prosthesis moves at the target joint angle, which can prevent the prosthesis from kicking obstacles and help improve walking stability. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0019] Figure 1 is a flowchart of an implementation of a prosthesis control method based on visual assistance provided in an embodiment of the present application;
[0020] Figure 2 is a schematic diagram of a prosthesis provided in one embodiment of the present application;
[0021] Figure 3 is a schematic diagram of a scene of a prosthetic connecting rod and a target obstacle provided by an embodiment of the present application;
[0022] Figure 4 is a flowchart of an implementation of a prosthesis control method based on vision assistance provided by another embodiment of the present application;
[0023] Figure 5 is a schematic diagram of the corresponding relationship between the thigh angle and the knee joint angle provided by an embodiment of the present application;
[0024] Figure 6 This is a schematic diagram of a scene of climbing stairs provided in an embodiment of the present application;
[0025] Figure 7 This is a schematic diagram of a scene of going down stairs provided in an embodiment of the present application;
[0026] Figure 8 is a flowchart of a prosthesis control method based on visual assistance provided by another embodiment of the present application;
[0027] Fig. 9 It is a schematic diagram of the bionic angle adjustment relationship under various environmental terrains provided by an embodiment of the present application;
[0028] Fig.10 This is a flowchart of an implementation of generating a terrain contour map provided by an embodiment of the present application;
[0029] Fig.11 is a structural block diagram of a prosthetic control device provided in an embodiment of the present application;
[0030] Fig.12 It is a structural block diagram of a prosthesis provided in one embodiment of the present application. DETAILED DESCRIPTION
[0031] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0032] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0033] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.
[0034] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0035] In order to illustrate the technical solution of the present application, the following embodiments are used for illustration.
[0036] See also Figure 1 , Figure 1 This is a flowchart of an implementation of a prosthesis control method based on vision assistance provided in an embodiment of the present application. The flowchart may include the following steps 101 to 104.
[0037] Step 101 , obtaining environmental point cloud information of the environment in which the prosthesis is located from a point cloud acquisition device, and obtaining motion parameters of the prosthesis and a rotation angle of the point cloud acquisition device from a sensor component.
[0038] Wherein, the above-mentioned prosthesis may include at least one joint, and the two ends of each joint are respectively connected to a prosthesis connecting rod. In practice, the above-mentioned prosthesis may be a prosthesis suitable for thigh amputees, in which case the prosthesis includes a knee joint and an ankle joint, the two ends of the knee joint are respectively connected to the thigh prosthesis connecting rod and the calf prosthesis connecting rod, and the two ends of the ankle joint are respectively connected to the calf prosthesis connecting rod and the sole prosthesis connecting rod. The above-mentioned prosthesis may also be a prosthesis suitable for tibial amputees, in which case the prosthesis includes an ankle joint, and the two ends of the ankle joint are respectively connected to the calf prosthesis connecting rod and the sole prosthesis connecting rod. In practice, in order to reduce the impact of the ground on the foot during walking, the sole prosthesis connecting rod may include a first connecting rod from the ankle joint to the toe, and a second connecting rod from the toe to the heel.
[0039] See also Figure 2 , Figure 2 Schematic diagram of a prosthesis provided by an embodiment of the present application. Figure 2 As shown, the sole prosthesis connecting rod includes a first connecting rod 2051 and a second connecting rod 2052 , and the prosthesis includes a knee joint 201 and an ankle joint 202 , the knee joint 201 is connected to the thigh receiving cavity 203 and the calf prosthesis connecting rod 204 , and the ankle joint 202 is connected to the calf prosthesis connecting rod 204 and the first connecting rod 2051 .
[0040] The prosthesis may be provided with a point cloud acquisition device and a sensor assembly. The point cloud acquisition device is usually a device for collecting point cloud information, such as a depth camera, a laser radar, etc. The sensor assembly is usually a device for collecting various parameters, such as an inertial sensor, an angle sensor, a pressure sensor, etc. The above-mentioned motion parameters may include the force parameters of the prosthesis and the current joint angle of the joint. The rotation angle of the above-mentioned point cloud acquisition device is usually the rotation angle of the camera coordinate system of the point cloud acquisition device relative to the ground coordinate system.
[0041] In this embodiment, the execution subject of the above-mentioned visual-assisted prosthesis control method is usually a prosthesis, and specifically may be a controller in the prosthesis.
[0042] In practice, the above-mentioned execution subject can obtain the environmental point cloud information of the prosthesis environment from the point cloud acquisition device through the network, obtain the force parameters of the prosthesis from the pressure sensor, obtain the current joint angle of the joint from the inertial sensor set at each joint, and obtain the rotation angle of the point cloud acquisition device from the inertial sensor set on the point cloud acquisition device. Here, the above-mentioned execution subject can also obtain the current joint angle of each joint through the motor encoder at each joint.
[0043] Step 102: Generate a terrain contour map according to the rotation angle and the environmental point cloud information.
[0044] Among them, the above-mentioned terrain contour map is usually used to indicate the contour of the environmental terrain.
[0045] In practice, the environmental point cloud information collected by the point cloud acquisition device is usually point cloud information in the camera coordinate system. The above-mentioned execution subject can obtain the rotation matrix between the camera coordinate system and the ground coordinate system through the rotation angle, and use the rotation matrix to convert the environmental point cloud information from the camera coordinate system to the ground coordinate system.
[0046] Afterwards, the above-mentioned execution entity can input the environmental point cloud information in the ground coordinate system into the pre-trained contour map generation model to obtain a terrain contour map. Among them, the contour map generation model is used to characterize the correspondence between the environmental point cloud information and the terrain contour map. Here, the contour map generation model can be a model obtained by training the initial model (such as a convolutional neural network (CNN), a residual network (ResNet), etc.) based on training samples using machine learning methods. The above-mentioned execution entity can also use a fitting algorithm, such as a random sampling consistency algorithm, a least squares method, etc., to fit the environmental point cloud information to obtain a terrain contour map.
[0047] Step 103, if the motion parameter indicates that the prosthesis is in a swinging state, then the actual distance between each prosthesis link and the target obstacle is determined according to the terrain contour map, the current joint angle of the joint and the length of each prosthesis link.
[0048] The actual distance mentioned above is usually the minimum distance between the prosthetic connecting rod and the target obstacle.
[0049] See also Figure 3 , Figure 3 Schematic diagram of a scene of a prosthetic connecting rod and a target obstacle provided by an embodiment of the present application. Figure 3 As shown, the actual distance between the lower leg prosthesis link 204 and the target obstacle 206 is d1, the actual distance between the first link 2051 and the target obstacle 206 is d2, and the actual distance between the second link 2052 and the target obstacle 206 is d3. The target obstacle is usually an obstacle indicated in the terrain contour map, such as a step surface perpendicular to the ground.
[0050] In practice, the states of the prosthesis include a swinging state and a standing state. The above-mentioned executive entity can collect the force parameters of the prosthesis through a pressure sensor installed at the ankle joint of the prosthesis, and then compare the collected force parameters with a pre-stored force threshold. If the force parameter is greater than or equal to the force threshold, it indicates that the prosthesis is in a standing state; if the force parameter is less than the force threshold, it indicates that the prosthesis is in a swinging state.
[0051] When the prosthesis is in a swinging state, the above-mentioned execution subject can obtain the joint position of the joint through the inertial sensor set at the joint, use the joint position, the current joint angle and the length of the prosthesis connecting rod connected to the joint to determine the position of the prosthesis connecting rod connected to the joint, and then calculate the actual distance between the prosthesis connecting rod and the target obstacle in the terrain contour. The above-mentioned execution subject can also input the obstacles indicated by the terrain contour map, the current joint angle of the joint and the length of each prosthesis connecting rod into the pre-trained prosthesis forward kinematic model to obtain the actual distance between each prosthesis connecting rod and the target obstacle. Among them, the prosthesis forward kinematic model is used to characterize the corresponding relationship between the obstacles indicated by the terrain contour map, the current joint angle of the joint and the length of each prosthesis connecting rod, and the actual distance between each prosthesis connecting rod and the target obstacle. Here, the prosthesis forward kinematic model can be a model obtained by training the initial model (such as a convolutional neural network (CNN), a residual network (ResNet), etc.) based on the training sample using a machine learning method.
[0052] Step 104, if there are prosthetic links whose actual distances are less than or equal to the preset distance threshold, determine the target joint angles of the joints according to the actual distances corresponding to the prosthetic links, and control the joints to adjust from the current joint angles to the target joint angles.
[0053] The preset distance threshold is usually a pre-set distance value. In practice, when the actual distance between the prosthetic connecting rod and the target obstacle is greater than the preset distance threshold, it means that the distance between the prosthetic connecting rod and the target obstacle is relatively far, and there is no risk of the prosthesis kicking the target obstacle during walking; when the actual distance between the prosthetic connecting rod and the target obstacle is less than or equal to the preset distance threshold, it means that the distance between the prosthetic connecting rod and the target obstacle is relatively close. If the joint angle of the prosthesis is not adjusted, there is a risk of the prosthesis kicking the target obstacle during walking.
[0054] In practice, the execution subject may use the actual distances of the prosthetic links to find the joint angles corresponding to the actual distances of the prosthetic links from a pre-established distance-angle correspondence table. The distance-angle correspondence table may be a correspondence table pre-established by the execution subject that stores a plurality of correspondences between distances and angles.
[0055] In practice, the above execution subject can also establish an objective function that describes the motion cost, and obtain the corresponding joint angle when the value of the objective function of the motion cost is minimized through iteration, and determine the joint angle as the target joint angle. For example, for a prosthesis including a knee joint and an ankle joint, the following motion cost objective function can be established:
[0056]
[0057] Among them, θ k is the angle of the knee joint, θ a is the angle of the ankle joint, are the target joint angles of the knee joint and the ankle joint, is the motion cost function, represents the motion cost function The angle of the knee joint and the angle of the ankle joint corresponding to the minimum value. The cost function is determined by the following formula:
[0058]
[0059] Among them, f repulse is the obstacle avoidance cost function, i is the prosthetic link, k r is the obstacle avoidance cost weight, d i is the actual distance between the prosthetic link i and the target obstacle, d 0 is the preset distance threshold.
[0060] For each joint, after obtaining the target joint angle of the joint, the execution subject can use the target joint angle of the joint to subtract from the current joint angle to obtain the angle difference, and then use the angle difference to find the target current corresponding to the angle difference from a pre-established angle difference-current correspondence table. Among them, the angle difference-current correspondence table can be a correspondence table pre-established by the execution subject, which stores a plurality of correspondences between angle differences and currents. Afterwards, the execution subject can use the obtained target current to control the motor of the joint to adjust the joint from the current joint angle to the target joint angle.
[0061] The visual-assisted prosthesis control method provided in the present embodiment obtains the actual distance between each prosthetic link and the target obstacle through the terrain contour map, the current joint angle of the joint in the prosthesis and the length of each prosthetic link when the motion parameters of the prosthesis indicate that the prosthesis is in a swinging state; when there are prosthetic links whose actual distance is less than or equal to a preset distance threshold, the target joint angle of the joint is determined through the actual distance corresponding to each prosthetic link, and the joint is controlled to adjust from the current joint angle to the target joint angle, so that the joint in the prosthesis moves at the target joint angle, which can prevent the prosthesis from kicking obstacles and help improve walking stability.
[0062] See also Figure 4 , Figure 4 This is a flowchart of an implementation of a prosthesis control method based on vision assistance provided in another embodiment of the present application. The flowchart may include the following steps 401 to 402.
[0063] In some embodiments, the motion parameters further include a thigh angle. After determining the target joint angle of the joint according to the actual distances corresponding to the prosthetic links, the following may also be included:
[0064] Step 401, if the thigh angle at the current moment belongs to the stretching angle range at the current moment, then determine the stretching adjustment angle according to the environmental terrain indicated by the terrain contour map.
[0065] Among them, the above-mentioned motion parameters include the thigh angle, which is usually the angle of the thigh raised relative to the standing state during walking. In practice, for prostheses including knee joints and ankle joints, an inertial sensor or an angle sensor can be set in the thigh receiving cavity to collect the thigh angle. For prostheses that do not include knee joints but only ankle joints, an inertial sensor or an angle sensor can be set at the user's thigh or knee to collect the thigh angle, for example, the inertial sensor can be attached to the user's thigh.
[0066] The above extension angle range is usually the range of thigh angles corresponding to the prosthesis in the extension phase. In practice, the swing state of the prosthesis includes the extension phase and the flexion phase. The extension phase is usually the phase from the sole of the foot leaving the ground to the thigh angle reaching the maximum in the gait cycle, and the flexion phase is usually the phase from the sole of the foot leaving the ground to the thigh angle reaching the maximum in the gait cycle. The gait cycle usually refers to the process of walking when the same foot steps out from the heel leaving the ground to the heel touching the ground again.
[0067] Among them, the stretching angle range at the current moment can be determined by the thigh angle at the previous moment and the preset angle shaking range. The preset angle shaking range is usually a preset angle shaking range, and the preset angle shaking range usually includes an upper limit shaking angle and a lower limit shaking angle. In practice, the above-mentioned neutral subject can use the thigh angle at the previous moment to add the upper limit shaking angle and the lower limit shaking angle to obtain the stretching angle range at the current moment. For example, the thigh angle at the previous moment is 10 degrees, the upper limit shaking angle in the preset angle shaking range is 0.02 degrees, and the lower limit shaking angle is -0.04 degrees. The stretching angle range at the current moment can be determined as [9.96, 10.02].
[0068] The above-mentioned environmental terrain may include any of the following: flat ground, uphill, downhill, up steps, down steps.
[0069] In practice, the above-mentioned execution subject can judge whether the thigh angle at the current moment belongs to the extension angle range at the current moment through the thigh angle at the current moment and the determined extension angle range at the current moment, that is, judge whether the prosthesis is in the extension stage at the current moment.
[0070] When the thigh angle at the current moment belongs to the stretching angle range at the current moment, the execution subject may use the environmental terrain to find the stretching adjustment angle corresponding to the environmental terrain from the pre-established terrain-angle correspondence table. The terrain-angle correspondence table may be a correspondence table pre-established by the execution subject and storing correspondences between multiple terrains and angles.
[0071] Step 402, adjusting the target joint angle according to the stretch adjustment angle, and switching the target joint angle to the adjusted target joint angle.
[0072] In some application scenarios, the execution subject may calculate and adjust the target joint angle based on the stretch adjustment angle to obtain an adjusted target joint angle.
[0073] In other application scenarios, the above-mentioned execution subject can also establish an objective function that describes the motion cost, and obtain the corresponding joint angle when the value of the objective function of the motion cost is minimized through iteration, and determine the joint angle as the target joint angle. For example, for a prosthesis including a knee joint and an ankle joint, the following motion cost objective function can be established:
[0074]
[0075] Among them, the cost function Determined by the following formula:
[0076]
[0077] Among them, f repulse is the obstacle avoidance cost function, f footplacement is the smooth landing cost function. Obstacle avoidance cost function f repulse The determination formula of is the same as that of the previous embodiment. For details, please refer to the determination formula of the previous embodiment, which will not be repeated here.
[0078] Among them, i is the prosthetic connecting rod, k r is the obstacle avoidance cost weight, d i is the actual distance between the prosthetic link i and the target obstacle, d 0 is the preset distance threshold. Smooth landing cost function f footplacement Determined by the following formula:
[0079]
[0080] Among them, k a is the weight of the stable landing cost, θ k is the angle of the knee joint, θ k,g Adjust the angle for knee extension, θ a is the angle of the ankle joint, θ a,gAdjust the angle for ankle extension.
[0081] For each joint, after obtaining the adjusted target joint angle, the execution subject may switch the target joint angle to the adjusted target joint angle.
[0082] In this embodiment, the extension angle range at the current moment is determined by the thigh angle at the previous moment and the preset angle jitter range, and the extension angle range is used to judge the swing state of the prosthesis. This can avoid the influence of slight jitter when the human leg swings on the judgment of the swing state of the prosthesis, which helps to improve the stability of the judgment of the swing state of the prosthesis; determining the extension adjustment angle through the environmental terrain, and using the extension adjustment angle to adjust the target joint angle, helps to improve the accuracy of the target joint angle of the prosthetic joint, further avoid the prosthesis kicking obstacles, and improve walking stability.
[0083] In some optional implementations of this embodiment, the above-mentioned determination of the stretch adjustment angle according to the environmental terrain indicated by the terrain contour map may include the following steps 1 and 2.
[0084] Step 1: If the environmental terrain indicated by the terrain contour map is any one of flat ground, uphill and downhill, the stretch adjustment angle is determined according to the thigh angle and the current joint angle of the joint.
[0085] In practice, during the stretching phase of walking, and when the terrain contour indicates that the environmental terrain is any of flat ground, uphill and downhill, there is a certain angle correspondence between the thigh angle and the knee joint angle. The above-mentioned execution subject can use the thigh angle and the angle correspondence to obtain the stretch adjustment angle of the knee joint, and obtain the stretch adjustment angle of the ankle joint through the stretch adjustment angle of the knee joint and the thigh angle.
[0086] See also Figure 5 , Figure 5 Schematic diagram of the corresponding relationship between the thigh angle and the knee joint angle provided by an embodiment of the present application. Figure 5 As shown, the horizontal axis is the thigh angle in the gait cycle, and the vertical axis is the knee joint angle in the gait cycle. A straight line is used to fit the corresponding relationship between the thigh angle and the knee joint angle in the gait cycle, and the fitted straight line is y=-x+0.1468. Based on this result, the extension adjustment angle of the knee joint can be set as:
[0087] θ k,g =-θ h +0.1468
[0088] Among them, θ k,g Adjust the angle for knee extension, θ h The angle of the thigh.
[0089] The ankle extension angle can be set to:
[0090] θ a,g =min{0,-θ a -θ k,g}=min{0,-θ a +θ h -0.1468}
[0091] Among them, θ a,g Adjust the angle for ankle extension, θ a is the current joint angle of the ankle joint.
[0092] Step 2: If the terrain indicated by the terrain contour map is an upward step or a downward step, the extension adjustment angle is determined according to the terrain contour map and the length of each prosthetic connecting rod.
[0093] In practice, when going up or down stairs, the landing point of the prosthesis is usually limited by the shape of the stairs. The above-mentioned execution body can calculate the extension adjustment angle through the terrain contour map and the length of each prosthesis connecting rod.
[0094] As an example, see Figure 6 , Figure 6 is a schematic diagram of a scene of climbing stairs provided by an embodiment of the present application. Figure 6 , ΔW is the horizontal distance between the nearest step corner point in the direction of the prosthesis and the prosthetic knee joint, ΔH is the vertical distance between the nearest step corner point in the direction of the prosthesis and the prosthetic knee joint, h is the height of the step, and w s The pre-set safe distance between the prosthesis and the step. At this time, the extension adjustment angle of the knee joint can be set to:
[0095]
[0096] The extension adjustment angle of the ankle joint is:
[0097]
[0098] Among them, ΔW is the horizontal distance between the nearest step corner point in the direction of the prosthesis and the prosthetic knee joint, ΔH is the vertical distance between the nearest step corner point in the direction of the prosthesis and the prosthetic knee joint, h is the height of the step, and w is the height of the step. s is the pre-set safe distance between the prosthesis and the step, L1 is the length of the first connecting rod, L2 is the length of the second connecting rod, θ heel is the angle between the first connecting rod and the second connecting rod.
[0099] As another example, see Figure 7 , Figure 7 is a schematic diagram of a scene of going down stairs provided in an embodiment of the present application. Figure 7 , ΔW is the horizontal distance between the nearest step corner point in the direction of prosthesis advancement and the prosthetic knee joint, ΔH is the vertical distance between the nearest step corner point in the direction of prosthesis advancement and the prosthetic knee joint, w is the width of the step, w s The pre-set safe distance between the prosthesis and the step. At this time, the extension adjustment angle of the knee joint can be set to:
[0100]
[0101] The extension adjustment angle of the ankle joint is:
[0102]
[0103] Where L1 is the length of the first connecting rod, θ heel is the angle between the first connecting rod and the second connecting rod.
[0104] In this embodiment, different methods are used to determine the stretch adjustment angle corresponding to the joint for different environmental terrains, which helps to improve the accuracy of the stretch adjustment angle. Using an accurate stretch adjustment angle to adjust the target joint angle helps to improve the accuracy of the target joint angle.
[0105] See also Figure 8 , Figure 8 This is a flowchart of an implementation of a prosthesis control method based on vision assistance provided by another embodiment of the present application. The flowchart may include the following steps 801 to 803.
[0106] Step 801: If the thigh angle at the current moment does not belong to the extension angle range, the bionic angle adjustment relationship is determined according to the environmental terrain indicated by the terrain contour map.
[0107] Among them, the bionic angle adjustment relationship is used to describe the relationship between the thigh angle and the bionic adjustment angle.
[0108] In practice, when the thigh angle at the current moment does not belong to the extension angle range, the execution subject may use the environmental terrain indicated by the terrain contour map to find the bionic angle adjustment relationship corresponding to the environmental terrain from the pre-established terrain-adjustment relationship correspondence table. The terrain-adjustment relationship correspondence table may be a correspondence table pre-established by the execution subject and storing a plurality of correspondences between terrains and adjustment relationships.
[0109] Step 802, according to the relationship between the thigh angle and the bionic angle adjustment, determine the bionic adjustment angle corresponding to the thigh angle, the bionic adjustment angle includes the bionic angle corresponding to the joint.
[0110] In practice, the above-mentioned execution subject can determine the bionic adjustment angle corresponding to the thigh angle from the obtained bionic angle adjustment relationship.
[0111] See also Fig. 9 , Fig. 9 It is a schematic diagram of the bionic angle adjustment relationship under various environmental terrains provided by an embodiment of the present application. Fig. 9 The first row from top to bottom shows the change in thigh angle in one gait cycle when the environment terrain is flat, and shows the corresponding bionic angle adjustment relationship when the environment terrain is flat. The second row shows the change in thigh angle in one gait cycle when the environment terrain is ascending stairs, and shows the corresponding bionic angle adjustment relationship when the environment terrain is ascending stairs. The third row shows the change in thigh angle in one gait cycle when the environment terrain is descending stairs, and shows the corresponding bionic angle adjustment relationship when the environment terrain is descending stairs. The fourth row shows the change in thigh angle in one gait cycle when the environment terrain is uphill, and shows the corresponding bionic angle adjustment relationship when the environment terrain is uphill. The fifth row shows the change in thigh angle in one gait cycle when the environment terrain is downhill, and shows the corresponding bionic angle adjustment relationship when the environment terrain is downhill.
[0112] Taking the first row as an example, the first figure from left to right in the first row shows the change of thigh angle in a gait cycle when the environment is flat. The second figure shows the bionic angle adjustment relationship between the thigh angle and the knee joint angle when the environment is flat, and the third figure shows the bionic angle adjustment relationship between the thigh angle and the ankle joint angle when the environment is flat.
[0113] Similarly, Fig. 9 The understanding of the figures in the second to fifth rows is basically the same as that of the figures in the first row, so I will not go into details here.
[0114] Combination Fig. 9 It can be seen that after determining the environmental terrain, the above-mentioned execution subject can adopt the thigh angle to obtain the bionic angle of the knee joint corresponding to the thigh angle from the fitting curve corresponding to the bionic angle adjustment relationship between the thigh angle and the knee joint angle; adopt the thigh angle to obtain the bionic angle of the ankle joint corresponding to the thigh angle from the fitting curve corresponding to the bionic angle adjustment relationship between the thigh angle and the ankle joint angle.
[0115] Step 803, controlling the joint to adjust from the current joint angle to the corresponding bionic angle.
[0116] In practice, for each joint, after obtaining the bionic angle of the joint, the above-mentioned execution subject can use the bionic angle of the joint to make a difference with the current joint angle to obtain the angle difference, and then use the angle difference to find the target current corresponding to the angle difference from the pre-established angle difference-current correspondence table. Among them, the above-mentioned angle difference-current correspondence table can be a correspondence table pre-established by the execution subject, which stores a plurality of correspondences between angle differences and currents. Afterwards, the above-mentioned execution subject can use the obtained target current to control the motor of the joint to adjust the joint from the current joint angle to the bionic angle.
[0117] In this embodiment, when the thigh angle at the current moment is not within the extension angle range, the joint is controlled to adjust from the current joint angle to the corresponding bionic angle, which helps to coordinate the swinging movement of the connecting rod of the prosthesis in the flexion phase naturally, avoids unnatural limb swinging, and helps to improve walking stability.
[0118] In some embodiments, when it is detected that the difference between adjacent stride lengths of the prosthesis meets a preset stable stride length condition, the extension angle range at the current moment can be determined in the following manner: the extension angle range is determined based on the target thigh angle and a preset angle jitter range.
[0119] The preset stable step length condition may be that the difference between adjacent step lengths is less than a preset step length difference threshold, and the preset step length difference threshold is usually a preset step length difference threshold, such as 0.1.
[0120] Among them, the above-mentioned target thigh angle is a thigh angle that meets a preset reference condition. The preset reference condition may include selecting the maximum thigh angle within a preset reference time as the target thigh angle. The preset reference time is usually a pre-set time, for example, 3 minutes.
[0121] In practice, the execution subject can obtain the acceleration of the prosthesis from the inertial sensor or acceleration sensor provided on the prosthesis, and determine the starting point and the ending point of the step by analyzing the acceleration. For example, the peak value or the zero crossing point of the acceleration signal is detected. When the acceleration signal changes from positive to negative, it indicates the beginning of the step; when the acceleration signal changes from negative to positive, it indicates the end of the step. Afterwards, the execution subject can integrate the acceleration in the acceleration interval between the starting and ending points of the step to obtain the speed, and integrate the acceleration over time to obtain the speed change curve over time. Then, after obtaining the speed change curve over time, the execution subject can integrate the speed over time to obtain the displacement change curve over time, and determine the starting point and the ending point of each step according to the displacement change curve over time, calculate the step length corresponding to each step, and subtract the adjacent step lengths to obtain the difference between the adjacent step lengths.
[0122] When the difference between adjacent stride lengths of the prosthesis meets the preset stable stride length condition, the above-mentioned execution entity can add the target thigh angle to the upper and lower limit angles of the preset angle shaking range to obtain the extension angle range at the current moment.
[0123] In this embodiment, when the difference between adjacent stride lengths of the prosthesis meets the preset stable stride length condition, the extension angle range at the current moment is determined by the target thigh angle and the preset angle jitter range, which can avoid calculating the extension angle range at each moment, help reduce the amount of calculation, and improve the efficiency of determining the extension angle range.
[0124] In some embodiments, the above-mentioned motion parameters may also include joint angular velocity, and the above-mentioned prosthesis control method based on vision assistance may also include the following first step to second step.
[0125] In the first step, the first-order derivative of the joint angular velocity and the second-order derivative of the joint angular velocity are determined according to the joint angular velocity.
[0126] In practice, the execution subject may obtain the joint angular velocity of the joint from an angular velocity sensor provided at the joint, or directly obtain the joint angular velocity from a motor encoder at the joint. The execution subject may then derive the joint angular velocity to obtain the first-order derivative and the second-order derivative of the joint angular velocity.
[0127] In the second step, the target joint angle is adjusted according to at least one of the joint angular velocity, the first-order derivative of the joint angular velocity and the second-order derivative of the joint angular velocity, and the target joint angle is switched to the adjusted target joint angle.
[0128] In practice, the above-mentioned execution entity can set the joint angular velocity range to constrain the joint angular velocity to prevent the joint angular velocity from exceeding the joint angular velocity range, set the first-order derivative range of the joint angular velocity to constrain the first-order derivative of the joint angular velocity to prevent the first-order derivative of the joint angular velocity from exceeding the first-order derivative range of the joint angular velocity, and set the second-order derivative range of the joint angular velocity to constrain the second-order derivative of the joint angular velocity to prevent the second-order derivative of the joint angular velocity from exceeding the second-order derivative range of the joint angular velocity.
[0129] In practice, the above-mentioned execution entity can adjust the target joint angle through at least one group of joint angular velocity and joint angular velocity range, first-order derivative of joint angular velocity and first-order derivative range of joint angular velocity, second-order derivative of joint angular velocity and second-order derivative range of joint angular velocity, and then switch the target joint angle to the adjusted target joint angle.
[0130] In practice, the above execution subject can establish an objective function that describes the motion cost, and obtain the corresponding joint angle when the value of the objective function of the motion cost is minimized through iteration, and determine the joint angle as the target joint angle. For example, for a prosthesis including a knee joint and an ankle joint, the following motion cost objective function can be established:
[0131]
[0132] Among them, the motion cost function Determined by the following formula:
[0133]
[0134] Among them, f repulse is the obstacle avoidance cost function, f footplacement is the motion smoothing cost function. Obstacle avoidance cost function f repulse The determination formula is the same as that in the above embodiment. For details, please refer to the determination formula in the above embodiment, which will not be repeated here.
[0135] Smooth landing cost function f smooth Determined by the following formula:
[0136]
[0137] Among them, k c is the motion smoothness cost weight, n is the prosthetic joint, w n is the joint angular velocity, w m is the preset angular velocity threshold, a n is the joint angular velocity, a m is the preset first-order derivative threshold of angular velocity, j n is the joint angular velocity, j m is a preset second-order derivative threshold of angular velocity.
[0138] In this embodiment, by adjusting the target joint angle through the joint angular velocity, smooth rotation of the prosthetic joint can be achieved, and the prosthetic joint can be prevented from rotating too fast or too slow, which helps to improve walking stability.
[0139] In some embodiments, in order to comprehensively consider obstacle avoidance, smooth landing, bionics and smooth motion, the above-mentioned execution subject can establish an objective function that describes the motion cost, and obtain the corresponding joint angle when the value of the objective function of the motion cost is minimized through iteration, and determine the joint angle as the target joint angle. For example, for a prosthesis including a knee joint and an ankle joint, the following motion cost objective function can be established:
[0140]
[0141] Among them, the cost function Determined by the following formula:
[0142]
[0143] Among them, the obstacle avoidance cost function f repulse , smooth landing cost function f smooth , motion smoothing cost function f footplacement The formula for determining is the same as that in the above embodiment. For details, please refer to the formula in the above embodiment, which will not be described here. bio Determined by the following formula:
[0144]
[0145] Among them, k b is the bionic cost weight, g k (θ h ) is the thigh angle θ h The bionic angle of the knee joint, g k (θ h ) is the thigh angle θ h The bionic angle of the ankle joint.
[0146] In this embodiment, by comprehensively considering obstacle avoidance, smooth landing, bionics and smooth movement, the prosthesis can avoid kicking obstacles when walking in various environmental terrains, land smoothly, have a bionic swing trajectory, and move smoothly, which helps to improve the stability of the prosthesis during walking.
[0147] See also Fig.10 , Fig.10 This is a flowchart for implementing the generation of a terrain contour map provided by an embodiment of the present application. The flowchart may include the following steps 1001 to 1003.
[0148] Step 1001: adjust the environment point cloud information from the camera coordinate system to the ground coordinate system according to the rotation angle.
[0149] In practice, the above-mentioned execution entity can establish a coordinate system transformation matrix from the camera coordinate system to the ground coordinate system by rotating the angle, and for each point cloud in the environmental point cloud information, use the coordinate system transformation matrix to convert the coordinates of the point cloud from the camera coordinate system to the ground coordinate system.
[0150] In practice, the above-mentioned execution subject can also input the rotation angle and the environmental point cloud information into a pre-trained coordinate transformation model to obtain the environmental point cloud information in the ground coordinate system. The coordinate transformation model is used to characterize the correspondence between the environmental point cloud information, the rotation angle and the environmental point cloud information in the ground coordinate system. Here, the coordinate transformation model can be a model obtained by training the initial model (such as a convolutional neural network (CNN), a residual network (ResNet), etc.) based on the training samples using a machine learning method.
[0151] Step 1002 , determining the environmental terrain according to the point cloud features of the adjusted environmental point cloud information.
[0152] Among them, the above-mentioned point cloud features are usually used to describe the attributes or characteristics of point cloud information, such as the boundaries of point cloud information.
[0153] In some application scenarios, the above-mentioned execution entity can match the point cloud features with the pre-stored terrain features of various terrains, and use the terrain whose corresponding terrain features match the point cloud features as the environmental terrain of the prosthesis.
[0154] In other application scenarios, the above-mentioned execution subject can also input the point cloud features of the environmental point cloud information into a pre-trained terrain classification model to obtain the environmental terrain corresponding to the environmental point cloud information. Among them, the terrain classification model is used to characterize the correspondence between the point cloud features and the environmental terrain. Here, the terrain classification model can be a model obtained by training the initial model (such as Convolutional Neural Network (CNN), Residual Network (ResNet), etc.) based on the training samples using machine learning methods.
[0155] Step 1003, fitting the environmental point cloud information using a fitting method corresponding to the environmental terrain to obtain shape feature parameters and contour points of the environmental terrain, and generating a terrain contour map according to the shape feature parameters and contour points.
[0156] The above fitting methods may include a straight line fitting method for fitting flat land, upslope and downslope, and a plane fitting method for fitting up steps and down steps. The shape feature parameters are used to describe the geometric shape of the environment terrain.
[0157] In practice, the execution subject may first reduce the dimension of the environmental point cloud information to reduce the amount of calculation. When the environmental terrain is any one of flat land, uphill and downhill, the execution subject may use the following formula to fit the environmental point cloud information:
[0158]
[0159] Among them, k is the kth point in the environmental point cloud information, (x k , z k ) is the coordinate of the kth point in the environmental point cloud information after dimensionality reduction, is the average coordinate value of all points in the environmental point cloud information after dimensionality reduction, m is the number of points in the environmental point cloud information, and θ is the slope of the environmental terrain.
[0160] When the environmental terrain is any one of flat land, upslope and downslope, the characteristic parameters of the environmental terrain include the slope θ of the environmental terrain, the contour points of the environmental terrain are The straight line used to describe the environmental terrain is
[0161] In practice, when the terrain environment is an upward step or a downward step, the execution subject can use a plane fitting method to fit the step multiple times to obtain the shape characteristic parameters and contour points of the step. Here, when fitting the step multiple times, one step is fitted each time. For example, when the environment terrain includes two steps, the execution subject can use the following formula to fit the step twice:
[0162]
[0163]
[0164]
[0165] Among them, P inliers1 is the set of points in the step surface for the first fitting, (x k ,z k ) is the coordinate of the kth point in the environmental point cloud information after dimensionality reduction, d is the preset internal point distance threshold, P 2D is the two-dimensional point cloud after dimensionality reduction, z inliers1 is the z coordinate of the point on the step surface of the first fitting, |z k -z inliers1 | is the coordinate z k The absolute value of the difference between the point cloud and the z coordinate of the point on the step surface of the first fitting, |P inliers1 | is the number of points in the step surface of the first fitting, P inliers2 is the set of points in the step surface for the second fitting, P 2D -P inliers1 The point cloud remaining after deleting the first fitted step surface point set in the two-dimensional point cloud, z inlierr2 is the z coordinate of the point on the step surface of the second fitting, |P inliers2 | is the number of points in the step surface of the second fitting, |z k -z inliers2 | is the coordinate zk The absolute value of the difference between the point cloud and the z coordinate of the point on the step surface of the second fitting, x 1max is the maximum x-coordinate of the points on the step surface of the first fitting, x 2max is the maximum value of the x coordinate of the point in the step surface of the second fitting, (x l ,z l ) is the coordinate of the point on the step surface of the first fitting, (x j ,z j ) is the coordinate of the point on the step surface of the second fitting, h is the height of the step, and w is the width of the step.
[0166] When the environment terrain is an upward step or a downward step, the characteristic parameters of the environment terrain include the height h of the step, the width w of the step, the contour point (x anchor ,z anchor )=(x 1max ,z inliers1 ), step surface S = {(x,z)|z = z anchor +iw,x anchor +(i-1)w <x<x anchor +iw,i∈{-1,0,1}}∪{(x,z)|x=x anchor +iw,z anchor +ih <z<z anchor +(i+1)h,i∈{-1,0,1}}.
[0167] Afterwards, the execution entity may connect the contour points in sequence based on the shape feature parameters to obtain a terrain contour map.
[0168] In this embodiment, the terrain contour map is generated by using shape feature parameters and contour points, which helps to improve the accuracy of generating the terrain contour map.
[0169] See also Fig.11 , Fig.11 1 is a structural block diagram of a prosthetic control device 1100 provided in an embodiment of the present application, including:
[0170] The information acquisition unit 1101 is used to acquire environmental point cloud information of the environment in which the prosthesis is located from the point cloud acquisition device, and to acquire motion parameters of the prosthesis and the rotation angle of the point cloud acquisition device from the sensor component, wherein the motion parameters include force parameters of the prosthesis and current joint angles of joints on the prosthesis;
[0171] A contour generating unit 1102 is used to generate a terrain contour map according to the rotation angle and the environmental point cloud information;
[0172] a distance determination unit 1103, for determining, if the motion parameter indicates that the prosthesis is in a swinging state, actual distances between each prosthesis link and a target obstacle according to the terrain contour map, the current joint angle of the joint and the length of each prosthesis link, wherein the target obstacle is an obstacle indicated in the terrain contour map;
[0173] The angle adjustment unit 1104 is used to determine the target joint angle of the joint according to the actual distance corresponding to each prosthetic link if there is a prosthetic link whose actual distance is less than or equal to the preset distance threshold, and control the joint to adjust from the current joint angle to the target joint angle.
[0174] In some embodiments, the angle adjustment unit 1104 includes an extension adjustment module and an angle switching module (not shown in the figure).
[0175] A stretch adjustment module, for determining a stretch adjustment angle according to the environmental terrain indicated by the terrain contour map if the thigh angle at the current moment belongs to the stretch angle range at the current moment, wherein the stretch angle range at the current moment is determined by the thigh angle at the previous moment and a preset angle jitter range;
[0176] The angle switching module is used to adjust the target joint angle according to the extension adjustment angle, and switch the target joint angle to the adjusted target joint angle.
[0177] In some embodiments, the extension adjustment module is specifically used to: if the environmental terrain indicated by the terrain contour map is any one of flat ground, uphill and downhill, determine the extension adjustment angle according to the thigh angle and the current joint angle of the joint; if the environmental terrain indicated by the terrain contour map is up stairs or down stairs, determine the extension adjustment angle according to the terrain contour map and the length of each prosthetic link.
[0178] In some embodiments, the angle adjustment unit 1104 further includes a relationship determination module, a bionic determination module, and a bionic adjustment module.
[0179] a relationship determination module, for determining a bionic angle adjustment relationship according to the environmental terrain indicated by the terrain contour map if the thigh angle at the current moment does not belong to the extension angle range, wherein the bionic angle adjustment relationship is used to describe the relationship between the thigh angle and the bionic adjustment angle;
[0180] A bionic determination module, used to determine a bionic adjustment angle corresponding to the thigh angle according to the relationship between the thigh angle and the bionic angle adjustment, wherein the bionic adjustment angle includes a bionic angle corresponding to the joint;
[0181] The bionic adjustment module controls the joint to adjust from the current joint angle to the corresponding bionic angle.
[0182] In some embodiments, the device also includes a range determination unit (not shown in the figure), which is used to: determine the stretching angle range based on the target thigh angle and the preset angle jitter range, wherein the target thigh angle is a thigh angle that meets a preset reference condition, and the preset reference condition includes selecting the maximum thigh angle within a preset reference time length as the target thigh angle.
[0183] In some embodiments, the device further includes a derivative determination unit and an angle switching unit (not shown in the figure).
[0184] A derivative determination unit is used to determine the first-order derivative of the joint angular velocity and the second-order derivative of the joint angular velocity according to the joint angular velocity; an angle switching unit is used to adjust the target joint angle according to at least one of the joint angular velocity, the first-order derivative of the joint angular velocity and the second-order derivative of the joint angular velocity, and to switch the target joint angle to the adjusted target joint angle.
[0185] In some embodiments, the contour generation unit 1102 includes a point cloud adjustment module, a terrain determination module, and a terrain generation module (not shown in the figure).
[0186] A point cloud adjustment module, used to adjust the environmental point cloud information from the camera coordinate system to the ground coordinate system according to the rotation angle;
[0187] A terrain determination module, used for determining the environmental terrain according to the point cloud features of the adjusted environmental point cloud information;
[0188] The terrain generation module is used to fit the environmental point cloud information using a fitting method corresponding to the environmental terrain, obtain the shape feature parameters and contour points of the environmental terrain, and generate a terrain contour map based on the shape feature parameters and contour points, wherein the shape feature parameters are used to describe the geometric shape of the environmental terrain.
[0189] The prosthetic control device provided in the present embodiment obtains the actual distance between each prosthetic link and the target obstacle through the terrain contour map, the current joint angle of the joint in the prosthesis and the length of each prosthetic link when the motion parameters of the prosthesis indicate that the prosthesis is in a swinging state; when there are prosthetic links whose actual distance is less than or equal to a preset distance threshold, the target joint angle of the joint is determined through the actual distance corresponding to each prosthetic link, and the joint is controlled to adjust from the current joint angle to the target joint angle, so that the joint in the prosthesis moves at the target joint angle, which can prevent the prosthesis from kicking obstacles and help improve walking stability.
[0190] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the visual-assisted prosthetic control method in the present application. Their specific functions and technical effects can be found in the embodiment section of the visual-assisted prosthetic control method and will not be repeated here.
[0191] See also Fig.12 , Fig.12 1 is a block diagram of a prosthesis 1200 provided in an embodiment of the present application. The prosthesis 1200 in this embodiment includes: at least one processor 1201 ( Fig.12 Only one processor is shown in the figure), a memory 1202, and a computer program 1203 stored in the memory 1202 and executable on at least one processor 1201, such as a prosthesis control program. When the processor 1201 executes the computer program 1203, the steps in the above-mentioned embodiments of the prosthesis control method based on visual assistance are implemented. When the processor 1201 executes the computer program 1203, the functions of the modules / units in the above-mentioned embodiments of the apparatus are implemented, for example, Fig.11 The functions of the information acquisition unit 1101 to the angle adjustment unit 1104 are shown.
[0192] Exemplarily, the computer program 1203 may be divided into one or more units, one or more units are stored in the memory 1202, and are executed by the processor 1201 to complete the present application. One or more units may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program 1203 in the prosthesis 1200. For example, the computer program 1203 may be divided into an information acquisition unit, a contour generation unit, a distance determination unit, and an angle adjustment unit. The specific functions of each unit have been described in the above embodiments and will not be repeated here.
[0193] The prosthesis 1200 may be a computing device such as a prosthesis, a desktop computer, a tablet computer, a cloud prosthesis, and a mobile terminal. The prosthesis 1200 may include, but is not limited to, a processor 1201 and a memory 1202. Those skilled in the art will appreciate that Fig.12 This is merely an example of the prosthesis 1200 and does not constitute a limitation of the prosthesis 1200 , and may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the prosthesis may also include input and output devices, network access devices, buses, etc.
[0194] The processor 1201 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0195] The memory 1202 may be an internal storage unit of the prosthesis 1200, such as a hard disk or memory of the prosthesis 1200. The memory 1202 may also be an external storage device of the prosthesis 1200, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the prosthesis 1200. Optionally, the memory 1202 may also include both an internal storage unit of the prosthesis 1200 and an external storage device. The memory 1202 is used to store computer programs and other programs and data required by the prosthesis 1200. The memory 1202 may also be used to temporarily store data that has been output or is to be output.
[0196] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0197] If the integrated module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and the computer program that can be completed by instructing the relevant hardware through a computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.
[0198] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0199] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A prosthetic control method based on vision assistance, It is characterized in that Applied to a prosthesis, the prosthesis includes a joint, both ends of the joint are respectively connected to a prosthesis connecting rod, and a point cloud collection device and a sensor assembly are provided on the prosthesis. The method includes: Acquire environmental point cloud information of the environment in which the prosthesis is located from the point cloud acquisition device, and acquire motion parameters of the prosthesis and a rotation angle of the point cloud acquisition device from the sensor assembly, wherein the motion parameters include force parameters of the prosthesis and a current joint angle of the joint; Generate a terrain contour map according to the rotation angle and the environmental point cloud information; If the motion parameter indicates that the prosthesis is in a swinging state, then determining the actual distance between each prosthesis link and a target obstacle according to the topographic contour map, the current joint angle of the joint and the length of each prosthesis link, wherein the target obstacle is an obstacle indicated in the topographic contour map; If there is a prosthetic link whose actual distance is less than or equal to the preset distance threshold, determining the target joint angle of the joint according to the actual distance corresponding to each prosthetic link, and controlling the joint to adjust from the current joint angle to the target joint angle; Determining the target joint angle of the joint includes: establishing an objective function that describes the motion cost, and determining the joint angle corresponding to when the value of the objective function of the motion cost is minimized through iteration as the target joint angle, wherein the objective function is: in, is the angle of the knee joint, is the angle of the ankle joint, are the target joint angles of the knee joint and the ankle joint, is the motion cost function, Denotes the motion cost function The angle of the knee joint and the angle of the ankle joint corresponding to the minimum value; the motion cost function is determined by the following formula: in, is the obstacle avoidance cost function, For prosthetic connecting rod, is the obstacle avoidance cost weight, Connecting rod for prosthesis The actual distance to the target obstacle, is the preset distance threshold.
2. The prosthesis control method based on vision assistance according to claim 1, It is characterized in that The motion parameters also include a thigh angle, and after determining the target joint angle of the joint according to the actual distances corresponding to the prosthetic links, the method further includes: If the thigh angle at the current moment belongs to the stretching angle range at the current moment, determining the stretching adjustment angle according to the environmental terrain indicated by the terrain contour map, wherein the stretching angle range at the current moment is determined by the thigh angle at the previous moment and the preset angle jitter range; The target joint angle is adjusted according to the stretch adjustment angle, and the target joint angle is switched to the adjusted target joint angle.
3. The prosthesis control method based on vision assistance according to claim 2, It is characterized in that The step of determining the stretch adjustment angle according to the environmental terrain indicated by the terrain contour map includes: If the environmental terrain indicated by the terrain contour map is any one of flat ground, uphill and downhill, determining a stretch adjustment angle according to the thigh angle and the current joint angle of the joint; If the environmental terrain indicated by the terrain contour map is an upward step or a downward step, the extension adjustment angle is determined according to the terrain contour map and the length of each prosthetic link.
4. The visually assisted prosthetic control method according to claim 2, It is characterized in that After determining the target joint angle of the joint according to the actual distances corresponding to the prosthetic links, the method further includes: If the thigh angle at the current moment does not belong to the extension angle range, determining a bionic angle adjustment relationship according to the environmental terrain indicated by the terrain contour map, wherein the bionic angle adjustment relationship is used to describe the relationship between the thigh angle and the bionic adjustment angle; Determine a bionic adjustment angle corresponding to the thigh angle according to the thigh angle and the bionic angle adjustment relationship, wherein the bionic adjustment angle includes a bionic angle corresponding to the joint; The joint is controlled to adjust from the current joint angle to a corresponding bionic angle.
5. The visually assisted prosthetic control method according to claim 2, It is characterized in that When it is detected that the difference between the adjacent step lengths of the prosthesis meets the preset stable step length condition, the extension angle range at the current moment is determined in the following manner: The stretching angle range is determined according to the target thigh angle and the preset angle jitter range, wherein the target thigh angle is a thigh angle that meets a preset reference condition, and the preset reference condition includes selecting the maximum thigh angle within a preset reference time length as the target thigh angle.
6. The visually assisted prosthetic control method according to claim 1, It is characterized in that The motion parameters also include joint angular velocity, and the method further includes: Determining a first-order derivative of the joint angular velocity and a second-order derivative of the joint angular velocity according to the joint angular velocity; The target joint angle is adjusted according to at least one of the joint angular velocity, the first-order derivative of the joint angular velocity, and the second-order derivative of the joint angular velocity, and the target joint angle is switched to the adjusted target joint angle.
7. The prosthesis control method based on vision assistance according to any one of claims 1 to 6, It is characterized in that The step of generating a terrain contour map according to the rotation angle and the environmental point cloud information comprises: Adjusting the environmental point cloud information from the camera coordinate system to the ground coordinate system according to the rotation angle; determining the environmental terrain according to the point cloud features of the adjusted environmental point cloud information; The environmental point cloud information is fitted using a fitting method corresponding to the environmental terrain to obtain shape feature parameters and contour points of the environmental terrain, and the terrain contour map is generated according to the shape feature parameters and the contour points, wherein the shape feature parameters are used to describe the geometric shape of the environmental terrain.
8. A prosthetic control device based on visual assistance, It is characterized in that Applied to a prosthesis, the prosthesis includes at least one joint, both ends of the joint are respectively connected to a prosthesis connecting rod, and the prosthesis is provided with a point cloud collection device and a sensor assembly, including: an information acquisition unit, configured to acquire environmental point cloud information of the environment in which the prosthesis is located from the point cloud acquisition device, and to acquire motion parameters of the prosthesis and a rotation angle of the point cloud acquisition device from the sensor assembly, wherein the motion parameters include force parameters of the prosthesis and current joint angles of the joints on the prosthesis; A contour generating unit, used for generating a terrain contour map according to the rotation angle and the environmental point cloud information; a distance determination unit, configured to determine, if the motion parameter indicates that the prosthesis is in a swinging state, actual distances between each prosthesis link and a target obstacle according to the topographic contour map, current joint angles of the joints, and lengths of each prosthesis link, wherein the target obstacle is an obstacle indicated in the topographic contour map; an angle adjustment unit, for determining a target joint angle of the joint according to the actual distances corresponding to the respective prosthetic links, and controlling the joint to adjust from the current joint angle to the target joint angle if there are prosthetic links whose actual distances are less than or equal to a preset distance threshold; Determining the target joint angle of the joint includes: establishing an objective function that describes the motion cost, and determining the joint angle corresponding to when the value of the objective function of the motion cost is minimized through iteration as the target joint angle, wherein the objective function is: in, is the angle of the knee joint, is the angle of the ankle joint, are the target joint angles of the knee joint and the ankle joint, is the motion cost function, Denotes the motion cost function The angle of the knee joint and the angle of the ankle joint corresponding to the minimum value; the motion cost function is determined by the following formula: in, is the obstacle avoidance cost function, For prosthetic connecting rod, is the obstacle avoidance cost weight, Connecting rod for prosthesis The actual distance to the target obstacle, is the preset distance threshold.
9. A prosthesis comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the vision-assisted prosthesis control method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the prosthesis control method based on vision assistance as described in any one of claims 1 to 7 is implemented.
