An image recognition-based personalized gait intention recognition method and system

Abstract

The application provides an image recognition-based personalized gait intention recognition method and system, and relates to the technical field of gait intention recognition.The application collects an infrared image pair and a distorted infrared image through a binocular infrared camera integrated with an equipment, respectively calculates and generates two depth maps, and performs weighted fusion to obtain a three-dimensional depth map of an environment under the feet of a user; multi-scale geometric analysis is carried out on the three-dimensional depth map, geometric features are extracted, and a multi-scale environment geometric feature tensor is generated; the tensor is input into a multi-task processing link, and user landing area semantic segmentation is executed in parallel, foot trajectory prediction is combined with real-time posture and historical gait of the user; according to the matching relationship between the semantic segmentation graph and the trajectory prediction sequence, the exploratory or confirmatory landing intention of the user is judged, corresponding intention instructions are generated, the personalized gait landing intention of an exoskeleton wearer can be accurately and robustly recognized, and reliable instruction basis is provided for intelligent control of the exoskeleton.

Description

Technical Field

[0001] This application relates to the field of gait intention recognition technology, and in particular to a personalized gait intention recognition method and system based on image recognition. Background Technology

[0002] Personalized gait intention recognition based on image recognition, which uses visual perception to acquire environmental and gait information to determine the user's foot placement intention, is a core human-computer interaction technology for exoskeletons and mobility assistive robots. This technology can adapt to the gait habits of different users, improve the coordination and safety of intelligent mobility assistive devices, and has broad application prospects in the fields of rehabilitation and elderly care.

[0003] Existing technologies of this kind mostly use monocular vision or conventional depth cameras to acquire environmental images, obtain spatial data through only a single depth calculation method, and then combine it with basic gait features to complete intent prediction. Due to its simplicity and controllable cost, this approach is the mainstream technical path for exoskeleton gait intent recognition.

[0004] Single-mode depth perception is susceptible to interference from the environment and terrain, resulting in insufficient depth information accuracy and inadequate environmental feature extraction, which can easily lead to deviations in foot placement and trajectory prediction. Intent recognition results suffer from poor stability, high misjudgment rates in complex scenes, and an inability to reliably adapt to personalized gait and changing walking environments. Therefore, existing technologies suffer from low accuracy in gait intent recognition and weak environmental adaptability. Summary of the Invention

[0005] The purpose of this application is to provide a personalized gait intention recognition method and system based on image recognition, so as to solve the problems of low accuracy and weak environmental adaptability of gait intention recognition in the prior art.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a personalized gait intent recognition method based on image recognition, comprising:

[0007] The device acquires infrared image pairs from the left and right perspectives and distorted infrared images with coded patterns using a binocular infrared camera integrated into the exoskeleton device.

[0008] Phase is calculated from the distorted infrared image to generate a first depth map, and parallax is calculated from the infrared image pair to generate a second depth map.

[0009] The first depth map and the second depth map are fused together with pixel-level confidence weighting to output a 3D depth map of the environment under the user's feet;

[0010] Multi-scale geometric analysis is performed on the three-dimensional depth map to capture environmental geometric features, thereby generating a multi-scale environmental geometric feature tensor. The environmental geometric features include the edge, planar and height variation features of the terrain structure.

[0011] The environmental geometric feature tensor is input into a multi-task processing process to perform semantic segmentation of the user's footing area and foot trajectory prediction based on the user's real-time posture and historical gait in parallel.

[0012] Based on the matching relationship between the semantic segmentation map obtained from semantic segmentation and the trajectory prediction sequence obtained from foot trajectory prediction, the user's foot placement intention is determined and a corresponding intention instruction is generated. The foot placement intention includes exploratory intention or confirmatory intention.

[0013] Optionally, the multi-task processing process includes a first processing branch and a second processing branch that are executed in parallel;

[0014] The first processing branch is used to perform pixel-by-pixel classification of the multi-scale environmental geometric feature tensor, divide each pixel position into different semantic regions, and generate a semantic segmentation map of the user's landing area. The semantic regions include safety planes, step edges, and dangerous pits.

[0015] The second processing branch is used to combine multi-scale environmental geometric feature tensors, joint angle sequences collected in real time by the exoskeleton device, and historical gait cycle data, and through a deductive structure with temporal memory function, to predict the continuous position points of the user's feet in three-dimensional space over a period of time in the future, and generate a trajectory prediction sequence.

[0016] Optionally, the step of determining the user's foot placement intention based on the matching relationship between the semantic segmentation map obtained from semantic segmentation and the trajectory prediction sequence obtained from foot trajectory prediction, and generating a corresponding intention command, includes:

[0017] The trajectory prediction sequence is mapped onto the semantic segmentation map to obtain semantic category information corresponding to the predicted location points;

[0018] The first discrimination criterion is obtained based on the semantic category corresponding to the endpoint location point in the trajectory prediction sequence;

[0019] The velocity change characteristics of the trajectory prediction sequence in the segment near the endpoint are analyzed to obtain a second discrimination criterion. The velocity change characteristics include the velocity descent gradient and velocity fluctuation value. The segment near the endpoint refers to the trajectory part of a preset length before the endpoint position point.

[0020] Based on the first and second criteria, the user's landing intention is determined to be either exploratory or confirmatory according to preset rules;

[0021] Based on the determined landing intention, intention commands for controlling the exoskeleton device are generated.

[0022] Optionally, the multi-scale geometric analysis of the three-dimensional depth map is performed to capture environmental geometric features to generate a multi-scale environmental geometric feature tensor. These environmental geometric features include the edges, planar features, and height variation features of the terrain structure, including:

[0023] The 3D depth map is scanned pixel by pixel using three different window sizes: large, medium, and small.

[0024] In a small window, the degree of difference in depth values ​​at each point within the window is calculated as an edge feature;

[0025] Within a medium-sized window, the plane equation of the depth values ​​within the window is fitted to extract planar features;

[0026] In a large window, the overall height relationship between the window center point and the depth values ​​of the surrounding area is calculated as a height variation feature.

[0027] The edge features, planar features, and height variation features obtained at the same pixel location under three different window sizes are merged into a multi-dimensional feature vector.

[0028] Arrange the multi-dimensional feature vectors of all pixel locations according to the image coordinates to form a multi-scale environmental geometric feature tensor.

[0029] Optionally, the preset rule means that when the semantic category corresponding to the first discrimination criterion is a safe plane semantic region, and the speed descent gradient in the second discrimination criterion is greater than the first threshold and the speed fluctuation value is less than the second threshold, then the user's landing intention is judged as a confirmatory intention; otherwise, it is judged as an exploratory intention.

[0030] Optionally, the step of predicting the continuous position points of the user's foot in three-dimensional space over a future period through a deductive structure with temporal memory function, and generating a trajectory prediction sequence, includes:

[0031] It receives multi-scale environmental geometric feature tensors, user joint angle sequences, and historical gait cycle data;

[0032] In the extrapolation structure, based on the temporal memory function, the received data is temporally modeled to infer the foot movement state at future moments using data from the current and historical moments;

[0033] The inference is performed iteratively to continuously generate predicted positions of the foot in three-dimensional space at multiple future moments, forming a trajectory prediction sequence.

[0034] Optionally, the step of calculating the phase from the distorted infrared image and generating a first depth map, and simultaneously calculating the disparity from the infrared image pair and generating a second depth map, includes:

[0035] The coded pattern at each pixel in the distorted infrared image is periodically analyzed, and the phase value corresponding to each pixel is obtained by calculating the brightness change sequence of the coded pattern point by point.

[0036] The phase value of each pixel is mapped to the corresponding distance value to form the first depth map;

[0037] For the left-view infrared image and the right-view infrared image in the infrared image pair, the corresponding pixel region in the right-view infrared image is found by pixel-by-pixel region matching, and the horizontal offset between the two is calculated as the disparity value.

[0038] Based on the fixed spacing and imaging focal length of the binocular infrared camera, the disparity value of each pixel is converted into a distance value to form a second depth map.

[0039] Secondly, this application provides a personalized gait intention recognition system based on image recognition, comprising:

[0040] The acquisition module is used to acquire infrared image pairs from the left and right perspectives and distorted infrared images with coded patterns through a binocular infrared camera integrated into the exoskeleton device.

[0041] The first generation module is used to calculate the phase from the distorted infrared image and generate a first depth map, and at the same time calculate the disparity from the infrared image pair and generate a second depth map.

[0042] The fusion module is used to perform pixel-level confidence-weighted fusion of the first depth map and the second depth map to output a three-dimensional depth map of the user's environment.

[0043] The analysis module is used to perform multi-scale geometric analysis on the three-dimensional depth map, capture environmental geometric features, and generate a multi-scale environmental geometric feature tensor, wherein the environmental geometric features include the edge, planar and height variation features of the terrain structure.

[0044] The processing module is used to input the environmental geometric feature tensor into the multi-task processing process and perform semantic segmentation of the user's foot landing area and foot trajectory prediction based on the user's real-time posture and historical gait in parallel.

[0045] The second generation module is used to determine the user's foot placement intention based on the matching relationship between the semantic segmentation map obtained by semantic segmentation and the trajectory prediction sequence obtained by foot trajectory prediction, and to generate a corresponding intention instruction, wherein the foot placement intention includes exploratory intention or confirmatory intention.

[0046] Thirdly, this application provides an electronic device, comprising:

[0047] Memory, used to store computer programs;

[0048] A processor, configured to implement the steps of the image recognition-based personalized gait intention recognition method as described in the first aspect above when executing the computer program.

[0049] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the image recognition-based personalized gait intention recognition method described in the first aspect above.

[0050] The personalized gait intent recognition method based on image recognition provided in this application uses a binocular infrared camera to acquire corresponding images, which can meet the acquisition requirements of dual-mode depth calculation. The dual paths generate depth maps separately, which can enrich the source of depth data. Pixel-level confidence weighted fusion can improve the accuracy and anti-interference of the 3D depth map. Multi-scale geometric analysis can effectively extract and characterize the core geometric features of the terrain. Multi-task parallel processing can simultaneously and efficiently acquire the semantic information of the landing area and foot trajectory information. Based on the segmentation map and trajectory sequence matching, the landing intent can be accurately determined and the corresponding instructions can be output.

[0051] Furthermore, the environmental geometric feature tensor is input into a multi-task processing procedure, which performs semantic segmentation of the user's landing area and foot trajectory prediction based on the user's real-time posture and historical gait in parallel. The multi-task processing procedure includes two parallel processing branches. The first branch classifies the multi-scale environmental geometric feature tensor pixel by pixel, dividing semantic regions such as safety planes, step edges, and dangerous pits to generate a semantic segmentation map of the landing area. The second branch combines the feature tensor, exoskeleton joint angle sequences, and historical gait cycle data to predict the continuous three-dimensional position points of the user's feet through temporal memory deduction structures, generating a trajectory prediction sequence. This dual-branch parallel specialized processing improves the execution efficiency of semantic segmentation and trajectory prediction. Accurate segmentation of different terrain semantic regions clarifies the safety attributes of the landing environment, and combining temporal and posture data enhances the accuracy and continuity of foot trajectory prediction, providing more accurate and reliable basic data support for subsequent landing intention determination. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 A flowchart illustrating a personalized gait intent recognition method based on image recognition provided in this application embodiment;

[0054] Figure 2 A flowchart illustrating another personalized gait intent recognition method based on image recognition provided in this application embodiment;

[0055] Figure 3 This is a schematic diagram of the structure of a personalized gait intention recognition system based on image recognition, provided in an embodiment of this application. Detailed Implementation

[0056] Existing image-based gait intention recognition technologies mostly employ a single depth perception approach, which is susceptible to interference from walking environment and terrain changes. The accuracy and reliability of depth information are insufficient, and environmental geometric features are not fully extracted, leading to deviations in foot trajectory prediction and misjudgments of foot placement areas. Such solutions exhibit poor stability and a high false-judgment rate in complex terrain, making it difficult to adapt to the personalized gait of different users and varied walking scenarios, and thus failing to provide safe and reliable intention command support for exoskeleton devices.

[0057] To address the aforementioned issues, this application proposes a personalized gait intention recognition method based on image recognition. It acquires two types of images using a binocular infrared camera and generates dual-path depth maps. These maps are then fused using pixel-level confidence weighting to obtain a high-precision 3D environmental depth map. Multi-scale geometric analysis extracts core terrain geometric features, and multi-task parallel processing simultaneously performs semantic segmentation of the landing area and foot trajectory prediction. Finally, the landing intention is determined based on the matching relationship between the two. This scheme enhances environmental perception robustness through dual-depth fusion, and optimizes recognition accuracy through multi-scale features and multi-task collaboration. It effectively reduces the probability of misjudgment in complex scenes and enhances environmental adaptability, fundamentally solving the problems of low recognition accuracy and poor environmental adaptability in existing technologies. This provides stable and reliable personalized gait intention commands for exoskeleton devices.

[0058] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] The core of this application is to provide a personalized gait intention recognition method based on image recognition, and a flowchart of one specific implementation is shown below. Figure 1 As shown, the method includes:

[0060] S101. By using a binocular infrared camera integrated into the exoskeleton device, infrared image pairs from the left and right perspectives and distorted infrared images with coded patterns are acquired.

[0061] The binocular infrared camera is an imaging device integrated into the exoskeleton device, featuring two shooting angles (left and right) to capture images in the infrared band. An infrared image pair refers to two sets of infrared images acquired by the camera from the left and right perspectives respectively. A distorted infrared image with a coded pattern refers to an infrared image containing specific coded markers and exhibiting slight image distortion; the coded pattern is used to assist in subsequent depth information calculation.

[0062] In one specific implementation, this step achieves image acquisition through a binocular infrared camera fixedly integrated into a suitable position on the exoskeleton device. After the camera is activated, the left and right shooting perspectives work and acquire images simultaneously. On one hand, through the two independent left and right perspectives, infrared images of the user's feet and the surrounding environment are captured respectively. These two sets of images correspond to each other and together constitute an infrared image pair. On the other hand, infrared images with preset coded patterns are acquired simultaneously. These images will be slightly distorted due to the optical imaging characteristics of the camera. Finally, the desired two types of images are simultaneously acquired: the infrared image pair of the left and right perspectives and the distorted infrared image with the coded pattern.

[0063] S102. The phase is calculated from the distorted infrared image and a first depth map is generated. At the same time, the parallax is calculated from the infrared image pair and a second depth map is generated.

[0064] In this context, phase refers to the periodic representation of the brightness changes of the coded pattern in the distorted infrared image, used to indirectly reflect the distance of pixels to the camera. Parallax refers to the horizontal position offset of corresponding pixel regions in the left and right viewpoints of the infrared image pair, and is a core parameter for calculating spatial distance. The first depth map is an image representing the distance of each pixel in the user's environment to the camera, obtained based on the phase solution of the coded pattern. The second depth map is an image similarly representing the distance information of environmental pixels, obtained based on the parallax calculation of the infrared image pair.

[0065] S102 specifically includes:

[0066] S1021. Perform periodic analysis on the coded pattern at each pixel in the distorted infrared image, and obtain the phase value corresponding to each pixel by calculating the brightness change sequence of the coded pattern point by point.

[0067] Periodic analysis refers to the analysis of the recurring brightness and darkness variations in the coded pattern. The brightness and darkness variation sequence refers to the continuous change process of the coded pattern at each pixel, from bright to dark and then back to bright. The phase value is a parameter that quantifies the position of the brightness and darkness variation cycle in the coded pattern, and its value corresponds to the distance from the pixel to the camera.

[0068] S1022. Map the phase value of each pixel to the corresponding distance value to form the first depth map.

[0069] Mapping refers to converting phase values ​​into actual spatial distance values ​​between pixels and the binocular infrared camera through a preset correspondence. In the first depth map, the grayscale or color value of each pixel corresponds to a distance value, intuitively presenting the distance distribution of the user's surroundings.

[0070] S1023. For the left-view infrared image and the right-view infrared image in the infrared image pair, find the corresponding pixel region in the right-view infrared image for each pixel region in the left-view infrared image by pixel-by-pixel region matching, and calculate the horizontal offset between the two as the disparity value.

[0071] Pixel-by-pixel region matching refers to comparing tiny pixel regions of the same size in the left-view infrared image and the right-view infrared image one by one to find regions with consistent features and corresponding positions. Horizontal offset refers to the difference in position of corresponding pixel regions in the horizontal direction between the left and right views. This difference is the disparity value, and its magnitude is negatively correlated with the distance from the pixel to the camera.

[0072] S1024. Based on the fixed spacing and imaging focal length of the binocular infrared camera, the disparity value of each pixel is converted into a distance value to form a second depth map.

[0073] The fixed spacing refers to the fixed distance between the left and right lenses of the binocular infrared camera. The imaging focal length refers to the focal length by which the camera lens converges light into a clear image; both are inherent parameters of the camera and are used to establish the conversion relationship between parallax values ​​and actual distances. The second depth map has the same function as the first depth map, both used to characterize the distribution of environmental distances, only differing in their generation methods.

[0074] In one specific implementation, this step generates the depth map through two parallel paths. The core objective is to balance the computational efficiency and accuracy of depth information to meet the exoskeleton device's need for real-time perception of the user's surroundings. When the exoskeleton wearer walks, they need to quickly acquire environmental distance information to support gait intention judgment. Parallel processing avoids the latency issues caused by single-path processing. Specifically, the two processing paths are carried out simultaneously without interference. The first path specifically processes the distorted infrared image with the coded pattern obtained earlier, and its core is to convert the distance by parsing the phase information of the coded pattern. The second path specifically processes the infrared image pairs from the left and right perspectives, and its core is to convert the distance by calculating the disparity information of the binocular images. The two paths proceed simultaneously, each completing the preprocessing and distance conversion of its corresponding image, ultimately generating a first depth map and a second depth map, forming two independent sets of environmental distance information. This provides a foundation for the subsequent fusion of the two depth maps and, through the complementarity of the two paths, reduces the errors that may occur in single-path processing, further improving the reliability of environmental distance perception.

[0075] As an example, this example is based on an exoskeleton wearer walking indoors. The environment underfoot includes a flat area and small steps about 10cm high. The fixed distance between the binocular infrared cameras is preset to 15cm, and the imaging focal length is preset to 8mm. The encoding pattern uses periodic black and white stripes to facilitate subsequent phase calculation. All operations are consistent with the actual use scenario of the exoskeleton, ensuring the practicality and reference value of the example.

[0076] First, for the distorted infrared image with black and white stripe coding acquired in this scene, step S1021 performs periodic analysis on the black and white stripe coding at each pixel in the distorted infrared image, capturing the brightness change sequence of the coding pattern of each pixel. Here, it is first clarified that one complete brightness cycle includes one bright stripe and one dark stripe. For example, for a pixel in a flat road area, its coding pattern shows a continuous change of "bright stripe-dark stripe-bright stripe-dark stripe", forming two complete brightness cycles; while for a pixel at the edge of a small step, because it is closer to the camera, the coding pattern is compressed, showing one complete brightness cycle of "bright stripe-dark stripe". By calculating the number of such brightness change cycles and the position of each pixel in the current cycle, the phase value corresponding to each pixel is obtained. The specific calculation process is as follows: the total phase of a single brightness cycle is set to 2π, and the phase value is used to characterize the position of the pixel in the brightness cycle, with a value range of 0~2π. The phase value of a pixel is equal to its position ratio in the brightness cycle multiplied by 2π. In the flat area, pixels are farther from the camera, resulting in more extended coded stripes that correspond to two complete brightness cycles. Taking a single cycle of 10 pixels as an example, a pixel in the flat area falls at the 8th pixel position in the first cycle, with a proportion of 0.8 calculated based on the pixel's absolute position from the start of the cycle. However, in the phase calculation of infrared coded stripes, the phase characteristic of the pixel at the end of the cycle is determined by the distance to the remaining pixels until the end of the cycle. This calculation method matches the logic of flat areas having smaller phase values ​​and corresponding to farther distances. Here, the proportion is calculated using the remaining pixels, resulting in a proportion of 0.2. Therefore, the phase value is 2π × 0.2 = 0.4π. Pixels at the edge of the small step are closer to the camera, resulting in compressed coded stripes that correspond to only one complete brightness cycle. Taking a single cycle of 10 pixels as an example, a pixel at the edge of the small step falls at the 3rd pixel position within the cycle, with a position proportion of 0.3. Therefore, the phase value is 2π × 0.3 = 0.6π.

[0077] Secondly, using the phase values ​​of all pixels obtained in step S1021, the phase value of each pixel is mapped to an actual spatial distance value through a preset phase-distance mapping rule. This rule is pre-calibrated based on parameters such as the fixed spacing and imaging focal length of the binocular infrared camera, with the aim of converting abstract phase values ​​into intuitive actual spatial distances. The specific mapping rule and logic are as follows:

[0078] The default mapping rule is that the smaller the phase value, the larger the corresponding distance value; and the larger the phase value, the smaller the corresponding distance value. The core logic is that the farther away from the camera, the smoother the phase change of the coded pattern, and the smaller the phase value; the closer to the camera, the more compressed the coded pattern, the more drastic the phase change, and the larger the phase value. For example, a phase value of 0.4π maps to a distance of 1.0 meter; a phase value of 0.6π maps to a distance of 0.8 meters. It should be noted that 1.0 meters corresponds to a flat road and 0.8 meters corresponds to a step because the step is 10cm higher than the flat road, and the straight-line distance of the raised part from the camera is closer than the straight-line distance from the flat road, perfectly matching the spatial relationship of the actual scene. The distance values ​​corresponding to all pixels are integrated to form the first depth map. This depth map distinguishes distance differences through different grayscale values, clearly showing the height difference and distance distribution between the flat road and the small step, facilitating subsequent capture of environmental geometric features.

[0079] Simultaneously, for the left and right view infrared image pairs acquired in this scenario, step S1023 is executed concurrently, using a block matching algorithm to perform pixel-by-pixel region matching on the left and right view infrared images. A tiny 3×3 pixel region surrounding each pixel is selected as a matching unit. The brightness and contour features of each matching unit in the left view are compared with those of all matching units in the right view to find corresponding units with completely identical features. For example, a matching unit representing a flat road in the left view has a corresponding matching unit in the right view, and the horizontal position difference between the two is calculated to be 11 pixels. This horizontal offset is the disparity value of the matching unit and its corresponding pixel. Similarly, a matching unit representing the edge of a small step in the left view has a horizontal offset of 13 pixels in the right view, and its disparity value is 13 pixels.

[0080] Finally, based on the disparity values ​​of all pixels, combined with the fixed spacing of 15cm and the imaging focal length of 8mm for the binocular infrared camera, step S1024 converts the disparity value of each pixel into its corresponding spatial distance value according to the conversion logic between disparity values ​​and actual distance. First, the core conversion rule is clarified: the binocular camera ranging formula is... ,in, It is the distance value from the target to the camera. The baseline for binoculars is 0.15m in this example. The focal length is 0.008m in this example, and d is the physical offset of the parallax, where d = parallax value × physical size of a single pixel. In this example, the physical size of a single pixel of the infrared camera is 0.0001m.

[0081] For example, if the disparity value of a pixel representing a flat road area is 11, then its physical offset d is 11 × 0.0001 = 0.0011m. Substituting this into the binocular camera ranging formula, the distance from that pixel to the camera can be calculated. If the disparity value of a pixel representing the edge of a small step is 13, then its physical offset d is 13 × 0.0001 = 0.0013m. Similarly, the distance from this pixel to the camera can be calculated. The distance values ​​of all pixels are integrated to form a second depth map. This depth map complements the first depth map, both clearly showing the distance distribution of the flat road and small steps underfoot. Only due to the different generation paths, there are slight differences in the distance values ​​of some pixels, providing complementary data for subsequent depth map fusion.

[0082] The above example is only one example of this application. In practical applications, it can be flexibly set according to the usage scenario of the exoskeleton, camera parameters, and coding pattern type. This application does not limit it in this regard.

[0083] This application generates first and second depth maps respectively. The dual-path complementarity makes up for the shortcomings of single depth calculation, improves the accuracy and reliability of depth information, effectively reduces distance calculation errors caused by environmental interference, and provides high-quality basic data support for the subsequent fusion generation of three-dimensional depth maps of the user's ground environment.

[0084] S103. Perform pixel-level confidence-weighted fusion of the first depth map and the second depth map to output a three-dimensional depth map of the environment under the user's feet.

[0085] In one specific implementation, the first and second depth maps are first preprocessed to ensure that the pixel positions of the two images are completely aligned. That is, pixels at the same coordinates in the two images correspond to the same position in the user's real-world environment, laying the foundation for pixel-by-pixel fusion. Subsequently, the confidence level of each pixel in the two depth maps is calculated: for the first depth map (based on phase resolution), the clarity of the coded pattern is evaluated. If the coded pattern of a pixel is clear and the brightness variation is distinct, its depth value error is small, and the confidence level is set to a higher value. For the second depth map (based on parallax calculation), the matching degree of the pixel regions in the left and right viewpoints is evaluated. The more accurate the matching and the more stable the horizontal offset calculation, the higher the confidence level of the pixel.

[0086] Next, fusion weights are assigned based on the confidence level of each pixel, following the principle of "the higher the confidence level, the greater the weight." That is, if the confidence level of a pixel is significantly higher in one depth map than in another, the pixel value of the high-confidence depth map is used as the primary value, and the pixel value of the low-confidence depth map is used as the secondary value, thereby reducing the impact of low-confidence data on the fusion result.

[0087] For example, continuing with the indoor walking scenario mentioned earlier, for a pixel in a flat area, the first depth map shows a depth of 1.0 meter with a confidence level of 0.85, while the second depth map shows a depth of 1.1 meter with a confidence level of 0.90. Assuming that the weight of the second depth map is assigned as 0.58 and the weight of the first depth map is assigned as 0.42 based on the confidence levels of the two depth maps, the fused pixel depth value is approximately 1.06 meters after weighted calculation (1.0 × 0.42 + 1.1 × 0.58). For a pixel on the edge of a small step, the first depth map shows a depth of 0.8 meter with a confidence level of 0.92, while the second depth map shows a depth of 0.9 meter with a confidence level of 0.83. The weight of the first depth map is assigned as 0.52 and the weight of the second depth map as 0.48. After calculation, the fused pixel depth value is determined to be approximately 0.85 meters.

[0088] Finally, the fused depth values ​​of all pixels are integrated and combined with the spatial positioning parameters of the binocular infrared camera to reconstruct the three-dimensional structure of the environment beneath the user's feet, generating a 3D depth map. This 3D depth map can clearly present the height difference between flat roads and small steps, while avoiding the blurring and error problems that may occur with single depth maps. It fully restores the edges, planes, and height changes of the environment, providing clear and reliable stereo data support for subsequent multi-scale geometric analysis.

[0089] This application effectively integrates the advantages of two depth maps through pixel-level confidence weighted fusion in this step, making up for the errors and deficiencies of single depth map calculation, improving the accuracy, stability and anti-interference of environmental depth information, and generating a three-dimensional depth map that can realistically and comprehensively restore the three-dimensional features of the environment under the user's feet, providing high-quality core data support for subsequent extraction of environmental geometric features and determination of gait and foot placement intentions.

[0090] S104. Perform multi-scale geometric analysis on the three-dimensional depth map to capture environmental geometric features and generate a multi-scale environmental geometric feature tensor.

[0091] Multi-scale geometric analysis refers to a method that uses analysis windows of different sizes to perform layered scanning and feature extraction on 3D depth maps, taking into account both subtle terrain features and overall structure. Environmental geometric features are the core information representing the spatial structure of the terrain beneath the user's feet, including the edge, planar, and height variation features of the terrain structure. Edge features correspond to terrain contours, such as the edge of a small step; planar features correspond to flat areas, such as a flat road; and height variation features correspond to the overall elevation differences of the terrain, such as the height difference between a flat road and a step. The environmental geometric feature tensor is a multi-dimensional data set formed by integrating the various geometric features of all pixels in image coordinate order, facilitating rapid access by subsequent multi-task processing modules.

[0092] S104 specifically includes:

[0093] S1041. Using windows of three different sizes—large, medium, and small—the three-dimensional depth map is scanned pixel by pixel.

[0094] In this context, a window refers to a rectangular area used to define the local analysis scope. The three sizes—large, medium, and small—correspond to different analysis scales. Pixel-by-pixel scanning means that the window moves in steps of one pixel, sequentially covering every pixel of the 3D depth map to ensure that no feature of any area is missed: small windows are used to capture subtle terrain features, such as the edges of steps; medium windows are used to identify medium-sized flat areas, such as local flat roads; and large windows are used to perceive the overall terrain structure, such as the overall height relationship between flat roads and steps.

[0095] S1042. In a small window, calculate the degree of difference in depth values ​​at each point within the window as an edge feature.

[0096] The degree of difference in depth values ​​refers to the dispersion of depth values ​​among all pixels within the window. The greater the difference, the more distinct the terrain outline of the area. Edge features are characteristic information representing the boundaries of terrain outlines, corresponding to convexities, depressions, or boundaries in real-world terrain, and are used to distinguish different types of terrain regions.

[0097] S1043. In a medium-sized window, fit the plane equation of the depth values ​​within the window and extract the plane features.

[0098] The plane equation is a mathematical expression used to describe the spatial location of a plane. By fitting the depth value within the window, it can be determined whether the area is a flat plane. Plane features are characteristic information that characterizes the flatness of the terrain. They correspond to areas in reality with no obvious changes in elevation, such as flat roads indoors. The closer the fitted plane equation is to the depth value within the window, the flatter the area is, and the more significant the plane features are.

[0099] S1044. In a large window, calculate the overall height relationship between the window center point and the depth values ​​of the surrounding area as a height variation feature.

[0100] Among them, the overall elevation relationship refers to the difference between the depth value of the center point of the window and the average depth value of all pixels within the window. A positive difference indicates that the terrain at the center point is relatively low, while a negative difference indicates that the terrain at the center point is relatively high. The elevation variation feature is characteristic information representing large-scale terrain undulations, used to reflect the overall terrain trend of the environment underfoot, such as the overall elevation difference between a flat road and small steps.

[0101] S1045. The edge features, planar features, and height variation features obtained at the same pixel location under three different window sizes are merged into a multi-dimensional feature vector.

[0102] Among them, the multi-dimensional feature vector is a one-dimensional data vector formed by integrating the three types of geometric features corresponding to a single pixel in a preset order. Each dimension corresponds to a quantized value of a feature, which can completely characterize the spatial structural attributes of the terrain location corresponding to the pixel, making it convenient for subsequent unified processing.

[0103] S1046. Arrange the multi-dimensional feature vectors of all pixel locations according to the image coordinates to form a multi-scale environmental geometric feature tensor.

[0104] Image coordinates refer to the two-dimensional position coordinates of each pixel in the 3D depth map. Arranging by image coordinates means arranging the multi-dimensional feature vectors of all pixels in order according to their actual positions in the depth map, forming a multi-dimensional data set that combines spatial location information and feature information, namely the environmental geometric feature tensor. This tensor can completely preserve the spatial distribution features and detailed information of the terrain.

[0105] In one specific implementation, this step generally follows a process of multi-scale scanning, sub-scale feature extraction, single-pixel feature merging, and full-image feature integration. The core is to comprehensively capture the subtle features and overall structure of the terrain through complementary analysis at different scales, ensuring that the extracted environmental geometric features are comprehensive and accurate, and adaptable to the real-time processing requirements of the exoskeleton without significant delay. Specifically:

[0106] First, three preset sizes of rectangular windows are used to scan the 3D depth map pixel by pixel to ensure that each pixel can be covered and analyzed by the three types of windows;

[0107] Then, within the small window, the edge features of each pixel are obtained by calculating the variance of all depth values ​​within the window; within the medium window, the least squares method is used to fit the plane equation of the depth values ​​within the window, and the significance of the plane features is judged by the equation fit degree.

[0108] Within a large window, the difference between the depth value at the center point of the window and the average depth value within the window is calculated to obtain the height variation feature of each pixel.

[0109] Then, the three features of edge, plane and height variation of the same pixel are merged into a three-dimensional feature vector in a preset order to fully represent the terrain attributes of the pixel. Finally, the three-dimensional feature vectors of all pixels are arranged in order of their horizontal and vertical coordinates in the three-dimensional depth map to form a multi-scale environmental geometric feature tensor for subsequent multi-task processing.

[0110] As an example, this example continues the indoor walking scenario of the exoskeleton wearer mentioned earlier. The environment underfoot includes a flat area and a small step about 10cm high. The 3D depth map clearly shows the distance and height difference between the two. The three window sizes are set as small window 3×3 pixels, medium window 5×5 pixels, and large window 7×7 pixels. The specific process is as follows:

[0111] First, in step S1041, the 3D depth map is scanned pixel by pixel using windows of three different sizes. The small window scans the entire depth map in steps of one pixel, focusing on scanning fine terrain areas such as the edges of small steps; the medium window scans simultaneously, focusing on covering medium-sized flat areas such as roads; and the large window advances simultaneously, focusing on capturing the overall spatial relationship between roads and small steps, ensuring that the analysis at all three scales is comprehensive.

[0112] Secondly, in step S1042, the variance of the depth values ​​of each pixel within a small 3×3 window is calculated to measure the degree of depth difference and serve as an edge feature. For example, when the small window scans to the edge of a small step, the window contains pixels above the step (depth 0.85 meters) and on the flat road (depth 1.06 meters). The calculated variance is relatively large, indicating that the depth difference in this area is significant, and the corresponding edge features are obvious, which can characterize the edge of the step. When the small window scans to the flat road area, the depth values ​​of all pixels in the window are around 1.06 meters, the variance is small, and the edge features are not significant, indicating that there is no obvious contour boundary in this area.

[0113] Next, in step S1043, the least squares method is used to fit the plane equation of all pixel depth values ​​within a 5×5 medium-sized window to extract plane features. For example, when the medium window completely covers the flat road area, the depth values ​​of all pixels within the window fluctuate little, the fitted plane equation has a high degree of fit and a small fitting error, indicating that the terrain in this area is flat and the plane features are significant; when the medium window partially covers the edge of the steps, the depth values ​​within the window fluctuate greatly, the fit is low, and the plane features are not significant, indicating that this area is not a flat plane.

[0114] Then, step S1044 is executed. Within a large 7×7 window, the difference between the depth value of the pixel at the center of the window and the average depth value of all pixels within the window is calculated to characterize the height variation. For example, when the center of the large window is located in a flat area, the difference between the center depth value of 1.06 meters and the average depth of 1.05 meters within the window is small, indicating that the overall height of the area is stable and the height variation is not significant. When the center of the large window is located above the edge of a small step, the difference between the center depth value of 0.85 meters and the average depth of 0.96 meters within the window is large, indicating that the overall height of the area changes significantly, and the height variation is significant, which can characterize the height difference between flat roads and steps.

[0115] Then, in step S1045, the edge features, planar features, and height variation features obtained at the same pixel location under three different window sizes are merged in the order of "edge features, planar features, and height variation features" to form a three-dimensional multi-dimensional feature vector. For example, for a pixel in a flat road area, the variance of the edge features is small, the fit of the planar features is high, and the difference in the height variation features is small. The feature vector formed after merging can clearly represent the flat terrain corresponding to this pixel. For a pixel at the edge of a step, the variance of the edge features is large, the fit of the planar features is low, and the difference in the height variation features is large. The feature vector formed after merging can clearly represent the terrain contour boundary corresponding to this pixel.

[0116] Finally, step S1046 is executed, which arranges the three-dimensional multi-dimensional feature vectors of all pixels in the three-dimensional depth map in the order of the horizontal and vertical coordinates of each pixel to form a multi-scale environmental geometric feature tensor. This tensor contains not only the subtle terrain features of each pixel, but also the flat features of a medium range and the overall height features of a large range, thus completely preserving the terrain structure information of the environment underfoot.

[0117] The above example is only one example of this application. In practical applications, the specific size of the window and the specific method of feature calculation, such as gradient calculation for edge features and other fitting algorithms for plane fitting, can be flexibly adjusted according to the usage scenario and environmental complexity of the exoskeleton. This application does not limit these aspects.

[0118] This application realizes multi-scale geometric feature extraction of 3D depth maps, which can comprehensively and accurately capture the edge, plane and height change features of the environment underfoot, fully restore the terrain spatial structure, and generate multi-scale environmental geometric feature tensors. This provides high-quality and comprehensive feature support for subsequent parallel execution of semantic segmentation of the landing area and foot trajectory prediction, ensuring the accuracy and reliability of subsequent multi-task processing.

[0119] S105. Input the environmental geometric feature tensor into the multi-task processing process, and perform semantic segmentation of the user's foot landing area and foot trajectory prediction based on the user's real-time posture and historical gait in parallel.

[0120] The multi-task processing process includes a first processing branch and a second processing branch that are executed in parallel.

[0121] The first processing branch performs pixel-by-pixel classification of the multi-scale environmental geometric feature tensor, dividing each pixel location into different semantic regions to generate a semantic segmentation map of the user's landing area. These semantic regions include safety planes, step edges, and dangerous pits.

[0122] The second processing branch combines multi-scale environmental geometric feature tensors, joint angle sequences collected in real time by the exoskeleton device, and historical gait cycle data. Through a deductive structure with temporal memory function, it predicts the continuous position points of the user's feet in three-dimensional space over a period of time and generates a trajectory prediction sequence.

[0123] In the second processing branch, a deductive structure with temporal memory function is used to predict the continuous position points of the user's feet in three-dimensional space over a future period, generating a trajectory prediction sequence, specifically including:

[0124] It receives multi-scale environmental geometric feature tensors, user joint angle sequences, and historical gait cycle data; in the inference structure, based on the temporal memory function, it performs temporal modeling on the received data, and infers the foot movement state at future moments using data from the current and historical moments; it iteratively performs inference, continuously generating predicted positions of the foot in three-dimensional space at multiple future moments to form a trajectory prediction sequence.

[0125] The purpose of this step is to process two core tasks in parallel, simultaneously acquiring the safety attributes of the landing area and the future movement path of the foot. This provides two core data supports for subsequently determining the user's landing intention, clarifying both "where the user can land" and "where the user wants to land," balancing processing efficiency and data reliability, and adapting to the real-time needs of exoskeleton intelligent control.

[0126] In one specific implementation, this step generally follows a process of feature input, dual-branch parallel processing, and separate output of results. The core is to improve processing efficiency through multi-task parallel design while ensuring the accuracy and compatibility of the results of the two tasks. Specifically, the multi-scale environmental geometric feature tensor generated earlier is first input into the multi-task processing module, starting two parallel processing branches. The first branch uses a lightweight classification algorithm to classify the feature tensor pixel by pixel, dividing it into three semantic regions: safe plane, step edge, and dangerous pit, generating a semantic segmentation map. The second branch first receives the feature tensor, the joint angle sequence collected in real time by the exoskeleton, and the user's historical gait cycle data. Then, it performs temporal modeling through a temporal memory inference structure, such as an LSTM module, to mine gait temporal patterns and iteratively infer the three-dimensional position points of the feet at multiple future moments, generating a trajectory prediction sequence. The two branches execute synchronously without interfering with each other, and finally output the semantic segmentation map and trajectory prediction sequence simultaneously for use in subsequent steps.

[0127] As an example, this example continues the indoor walking scenario of the exoskeleton wearer mentioned earlier. In this scenario, there is a flat road and a small step about 10cm high underfoot, with no dangerous pits. The specific process is as follows:

[0128] First, the environmental geometric feature tensor containing features of flat roads and small steps is input into the multi-task processing module, and two parallel branches are started.

[0129] Next, the semantic segmentation task of the first processing branch is executed: a pixel-level classification algorithm is used to determine the attributes of each pixel in the feature tensor. For example, pixels in flat areas are classified as safe planes because their planar features are significant, while their edge features and height variation features are not significant; pixels at the edges of small steps are classified as step edges because their edge features and height variation features are significant, while their planar features are not significant; and no dangerous pits are marked for areas without depressions in the scene. After classification, the three types of regions are labeled with different gray values ​​to generate a semantic segmentation map, where white marks safe planes and gray marks step edges, visually presenting the distribution and attributes of footholds.

[0130] Simultaneously, the second processing branch performs the foot trajectory prediction task: first, it receives the environmental geometric feature tensor, which includes the height difference information between flat roads and steps, the joint angle sequence collected in real time by the exoskeleton, such as the current ankle joint angle of 30° and the knee joint angle of 90°, reflecting that the user is in the state of lifting the foot, and the user's historical gait cycle data, such as the user's past complete gait cycle of 1.2 seconds and the foot lifting height of about 15cm.

[0131] Subsequently, temporal modeling was performed using a temporal memory deduction structure: combining the current joint angle (foot lifting state), historical gait cycle (1.2 seconds / cycle), and the height information of the steps in the environment, the user's gait patterns were mined, and it was found that when the user encountered similar small steps in the past, he would raise his feet to more than 15cm, cross the edge of the step, and then land on the flat area above the step.

[0132] Then, the inference process is executed iteratively: first, it is inferred that after 100ms, the three-dimensional position of the foot is (x1, y1, 0.15m), which is in the lifting and rising phase; then, based on this position, combined with the changes in joint angles at subsequent moments, such as the gradual decrease of the ankle joint angle and environmental features, it is inferred that after 200ms, the foot position is (x2, y2, 0.12m), which is in the lifting and falling phase; the inference is continuously iterated until eight consecutive three-dimensional foot position points within the next second are generated, forming a trajectory prediction sequence. This sequence clearly presents the complete path of the user's foot from the current position, across the edge of the step, and finally landing on the flat road above the step.

[0133] The two branches execute synchronously, with each taking less than 50ms to ensure no significant delay. They ultimately output a semantic segmentation map annotating the safety plane and step edges, and a trajectory prediction sequence representing the future movement path of the foot. The above example is merely one illustration of this application. In practical applications, the choice of classification algorithm, the type of temporal deduction structure, and the duration of trajectory prediction can be flexibly adjusted according to the exoskeleton's usage scenario and the user's gait habits; this application does not impose any limitations on these aspects.

[0134] In another specific implementation, the first processing branch can adopt a CNN classification model, which uses semantic segmentation samples in the exoskeleton scene to label depth map samples with safe planes, step edges, and dangerous pits. This needs to be trained in advance to improve classification accuracy. The temporal memory inference structure of the second processing branch can be fine-tuned by combining the user's personalized gait data to make the trajectory prediction more in line with the gait habits of different users, thereby further improving the prediction accuracy.

[0135] This application, through dual-branch parallel multi-task processing in this step, accurately obtains the safety attributes of the landing area and the future movement path of the foot while ensuring processing efficiency. It not only clarifies the types of landing areas in the environment but also predicts the user's foot movement trend, providing comprehensive and reliable core data support for subsequent determination of landing intentions. This effectively adapts to the real-time perception and intelligent control needs of exoskeleton wearers when walking.

[0136] S106. Based on the matching relationship between the semantic segmentation map obtained from semantic segmentation and the trajectory prediction sequence obtained from foot trajectory prediction, determine the user's foot placement intention and generate the corresponding intention command.

[0137] The matching relationship refers to the correspondence between all predicted location points in the trajectory prediction sequence and the corresponding semantic categories in the semantic segmentation map. Its core function is to determine whether the user's predicted landing path points to a safe and placeable area. Landing intention is the core purpose of the user's foot movements while walking, including exploratory and confirmatory intentions. Confirmatory intentions indicate that the user clearly intends to land in a safe area, with coherent movements and a clear goal. Exploratory intentions indicate that the user has not yet determined a landing area and is in a tentative state, with slow and uncertain movements. Intention commands are signals generated based on the landing intention and used to control the exoskeleton device, adapting to the coordinated movements of the exoskeleton and the user's gait.

[0138] S106 specifically includes:

[0139] S1061. Map the trajectory prediction sequence onto the semantic segmentation map to obtain semantic category information corresponding to the predicted location point.

[0140] Mapping refers to converting the three-dimensional spatial coordinates of each predicted foot location in the trajectory prediction sequence into two-dimensional pixel coordinates in the semantic segmentation map, ensuring a precise correspondence between each predicted location and its corresponding pixel in the semantic segmentation map. Semantic category information refers to the semantic region type corresponding to each predicted location, used to determine whether the predicted location is suitable for landing, and is the basis for establishing the matching relationship between the two.

[0141] S1062. Obtain the first discrimination criterion based on the semantic category corresponding to the endpoint position point in the trajectory prediction sequence.

[0142] The endpoint location refers to the last predicted foot location in the trajectory prediction sequence, which is the user's most likely landing target. The primary criterion is one of the core reference information used to determine the landing intention, which is essentially the safety attribute of the user's predicted landing target area. The safer the target area, the more likely the user has a clear landing intention.

[0143] S1063. Analyze the velocity change characteristics of the trajectory prediction sequence near the end point to obtain a second discrimination criterion.

[0144] The "near-the-end segment" refers to a series of consecutive points of a predetermined length near the end point in the trajectory prediction sequence, such as the last 2-3 points, corresponding to the stage where the user's foot is about to land. Velocity change characteristics represent information about the foot's movement state, including the velocity descent gradient and velocity fluctuation value. The velocity descent gradient refers to the rate at which the foot's speed decreases as it approaches landing; a larger gradient indicates a more decisive foot landing action. The velocity fluctuation value refers to the stability of the foot's speed within the near-end segment; smaller fluctuations indicate a smoother foot movement. The second criterion is to judge the user's determination in landing based on the foot's movement state; the more decisive and stable the action, the clearer the user's intention to land.

[0145] S1064. Based on the first and second discrimination criteria, determine whether the user's landing intention is exploratory or confirmatory according to preset rules.

[0146] The preset rule means that if the semantic category corresponding to the first discrimination criterion is the safe plane semantic region, and the speed descent gradient in the second discrimination criterion is greater than the first threshold and the speed fluctuation value is less than the second threshold, then the user's landing intention is judged as a confirmatory intention; otherwise, it is judged as an exploratory intention.

[0147] S1065. Based on the determined landing intention, generate intention commands for controlling the exoskeleton device.

[0148] Among them, the intention command is the control signal adapted to the exoskeleton device, and the landing intention... Figure 1One-to-one correspondence: Confirmatory intentions correspond to instructions that control the exoskeleton to coordinate with the user's gait, adjusting joint angles and force to assist the user in landing smoothly. Exploratory intentions correspond to instructions that control the exoskeleton to slow down its movements, maintain its current posture, and wait for further confirmation from the user's actions to avoid safety hazards caused by misoperations and ensure that the exoskeleton's movements are synchronized with the user's intentions.

[0149] In one specific implementation, this step generally follows a process of trajectory determination, semantic mapping, dual-criteria extraction, rule-based judgment, and command generation. The core is to accurately determine the landing intention by combining environmental information and motion state, ensuring that the generated control commands adapt to the user's real needs, and that the processing is efficient, latency-free, and meets the real-time control requirements of the exoskeleton. Specifically, firstly, a coordinate matching algorithm maps all position points of the trajectory prediction sequence to a semantic segmentation map to obtain the semantic category of each position point; secondly, the semantic category of the trajectory endpoint is extracted as the first discrimination criterion; next, a gradient calculation method is used to analyze the velocity descent gradient and velocity fluctuation value near the endpoint as the second discrimination criterion; then, the landing intention is comprehensively determined by comparing the two discrimination criteria with preset rules; finally, based on the discrimination results, corresponding exoskeleton control commands are generated, completing the entire landing intention determination and command generation process.

[0150] As an example, this example continues the indoor walking scenario of the exoskeleton wearer mentioned earlier. In this scenario, there is a flat road and a small step about 10cm high, with no dangerous potholes. The trajectory prediction sequence consists of eight consecutive three-dimensional foot positions within the next second, with each point spaced 0.125s apart. The endpoint is located on a safe plane above the step. The preset first threshold is 0.3m / s², and the second threshold is 0.07m / s. Both thresholds are pre-calibrated based on this indoor scenario and the user's gait data. The specific process is as follows: Figure 2 As shown:

[0151] First, in step S1061, a coordinate matching algorithm is used to convert the three-dimensional spatial coordinates of the eight foot prediction positions in the trajectory prediction sequence into two-dimensional pixel coordinates in the semantic segmentation map, so that each prediction position point corresponds precisely to the corresponding pixel point in the semantic segmentation map, and finally the semantic category information of each position point is obtained: the first five position points correspond to the flat road (safe plane), the sixth and seventh position points correspond to the edge of the steps, and the eighth position point (end point) corresponds to the safe plane above the steps.

[0152] Secondly, the semantic category corresponding to the endpoint location point (the 8th one) in the trajectory prediction sequence is extracted through step S1062. This location point is located on the safety plane above the steps, so the first discrimination criterion obtained is "safe plane".

[0153] Next, step S1063 sets the segment near the endpoint as the last three position points (the 6th, 7th, and 8th) of the trajectory prediction sequence, corresponding to the stage when the user's foot is about to land. The gradient calculation method is used to analyze the velocity change characteristics. Assume the foot velocities at these three position points are: 0.18 m / s at point 6, 0.11 m / s at point 7, and 0.05 m / s at point 8 (the endpoint). The process of analyzing the velocity change characteristics specifically includes:

[0154] First, the velocity descent gradient is calculated. The last two segments near the end of the segment (points 7 and 8) are selected for calculation. The velocity at point 7 is 0.11 m / s, and the velocity at point 8 is 0.05 m / s. The calculated velocity descent gradient is: This value is greater than the first threshold of 0.3 m / s², indicating that the foot landing action was decisive.

[0155] Secondly, the speed fluctuation value is calculated using the standard deviation of the speed at these three points. The smaller the standard deviation, the more stable the speed. The average speed at points 6, 7, and 8 is calculated to be 0.113 m / s. Substituting this into the standard deviation formula, the standard deviation is calculated to be 0.057 m / s. This value is less than the second threshold of 0.07 m / s, indicating stable foot movement. In summary, the second criterion is a rapid decrease in speed and small fluctuations.

[0156] Then, by comparing the preset rules in step S1064, the first criterion is the safety plane, and the second criterion is that the speed descent gradient is greater than the first threshold and the speed fluctuation value is less than the second threshold. Therefore, the user's intention to land is judged as a confirmed intention, indicating that the user clearly wants to land on the safety plane above the steps, and the action is decisive and the goal is clear.

[0157] Finally, in step S1065, based on the confirmed footing intention, the corresponding exoskeleton control command is generated. The command content is: adjust the knee and ankle joint angles, slow down the foot descent speed, cooperate with the user to land smoothly on the safe plane above the step, provide appropriate force support, and ensure that the exoskeleton movement is synchronized with the user's footing intention.

[0158] The above example is only one example of this application. In practical applications, the specific values ​​of the preset length of the near-end segment, the first threshold, and the second threshold can be flexibly adjusted according to the exoskeleton usage scenario and the user's gait habits. This application does not limit this.

[0159] In another specific implementation, the threshold of the preset rule can be personalized by using the user's historical gait data. For example, for elderly users with a slow gait, the first threshold can be lowered and the second threshold can be raised to match their slow walking rhythm. At the same time, the process of mapping the trajectory prediction sequence to the semantic segmentation map can use a more accurate coordinate calibration algorithm to reduce mapping errors, further improve the discrimination accuracy, and better adapt to the personalized needs of different users.

[0160] This application achieves accurate identification of landing intentions and generation of corresponding commands. By combining environmental safety information and user movement status, it clearly distinguishes between confirmatory and exploratory intentions. The generated exoskeleton control commands are tailored to the user's real needs, effectively improving the coordination between the exoskeleton and the user and walking safety. It adapts to the personalized gait of different users and ensures that the exoskeleton movements are synchronized with the user's landing intentions.

[0161] Figure 3 This is a schematic diagram illustrating a specific implementation of a personalized gait intention recognition system based on image recognition, as provided in this application. (Refer to...) Figure 3 The system may include:

[0162] The acquisition module 31 is used to acquire infrared image pairs from the left and right perspectives and distorted infrared images with coded patterns through a binocular infrared camera integrated into the exoskeleton device.

[0163] The first generation module 32 is used to calculate the phase from the distorted infrared image and generate a first depth map, and at the same time calculate the parallax from the infrared image pair and generate a second depth map.

[0164] The fusion module 33 is used to perform pixel-level confidence-weighted fusion of the first depth map and the second depth map to output a three-dimensional depth map of the user's environment.

[0165] Analysis module 34 is used to perform multi-scale geometric analysis on the three-dimensional depth map, capture environmental geometric features, and generate a multi-scale environmental geometric feature tensor, wherein the environmental geometric features include the edge, plane and height variation features of the terrain structure.

[0166] Processing module 35 is used to input the environmental geometric feature tensor into the multi-task processing process and perform semantic segmentation of the user's foot landing area and foot trajectory prediction based on the user's real-time posture and historical gait in parallel.

[0167] The second generation module 36 is used to determine the user's foot placement intention based on the matching relationship between the semantic segmentation map obtained by semantic segmentation and the trajectory prediction sequence obtained by foot trajectory prediction, and to generate a corresponding intention instruction, wherein the foot placement intention includes exploratory intention or confirmatory intention.

[0168] The personalized gait intention recognition system based on image recognition in this application is used to implement the aforementioned personalized gait intention recognition method based on image recognition. Therefore, the specific implementation of the personalized gait intention recognition system based on image recognition can be found in the embodiment section of the personalized gait intention recognition method based on image recognition above. The specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0169] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described image recognition-based personalized gait intention recognition methods.

[0170] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described image recognition-based personalized gait intention recognition methods.

[0171] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0172] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the personalized gait intention recognition method based on image recognition.

[0173] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0174] The above provides a detailed description of a personalized gait intent recognition method and system based on image recognition provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A personalized gait intent recognition method based on image recognition, characterized in that, include: The device acquires infrared image pairs from the left and right perspectives and distorted infrared images with coded patterns using a binocular infrared camera integrated into the exoskeleton device. Phase is calculated from the distorted infrared image to generate a first depth map, and parallax is calculated from the infrared image pair to generate a second depth map. The first depth map and the second depth map are fused together with pixel-level confidence weighting to output a 3D depth map of the environment under the user's feet; Multi-scale geometric analysis is performed on the three-dimensional depth map to capture environmental geometric features, thereby generating a multi-scale environmental geometric feature tensor. The environmental geometric features include the edge, planar and height variation features of the terrain structure. The environmental geometric feature tensor is input into a multi-task processing process to perform semantic segmentation of the user's footing area and foot trajectory prediction based on the user's real-time posture and historical gait in parallel. Based on the matching relationship between the semantic segmentation map obtained from semantic segmentation and the trajectory prediction sequence obtained from foot trajectory prediction, the user's foot placement intention is determined and a corresponding intention instruction is generated. The foot placement intention includes exploratory intention or confirmatory intention.

2. The method according to claim 1, characterized in that, The multi-task processing process includes a first processing branch and a second processing branch that are executed in parallel. The first processing branch is used to perform pixel-by-pixel classification of the multi-scale environmental geometric feature tensor, divide each pixel position into different semantic regions, and generate a semantic segmentation map of the user's landing area. The semantic regions include safety planes, step edges, and dangerous pits. The second processing branch is used to combine multi-scale environmental geometric feature tensors, joint angle sequences collected in real time by the exoskeleton device, and historical gait cycle data, and through a deductive structure with temporal memory function, to predict the continuous position points of the user's feet in three-dimensional space over a period of time in the future, and generate a trajectory prediction sequence.

3. The method according to claim 1, characterized in that, The process of determining the user's foot placement intention based on the matching relationship between the semantic segmentation map obtained from semantic segmentation and the trajectory prediction sequence obtained from foot trajectory prediction, and generating corresponding intention instructions, includes: The trajectory prediction sequence is mapped onto the semantic segmentation map to obtain semantic category information corresponding to the predicted location points; The first discrimination criterion is obtained based on the semantic category corresponding to the endpoint location point in the trajectory prediction sequence; The velocity change characteristics of the trajectory prediction sequence in the segment near the endpoint are analyzed to obtain a second discrimination criterion. The velocity change characteristics include the velocity descent gradient and velocity fluctuation value. The segment near the endpoint refers to the trajectory part of a preset length before the endpoint position point. Based on the first and second criteria, the user's landing intention is determined to be either exploratory or confirmatory according to preset rules; Based on the determined landing intention, intention commands for controlling the exoskeleton device are generated.

4. The method according to claim 1, characterized in that, The process involves performing multi-scale geometric analysis on the 3D depth map to capture environmental geometric features, thereby generating a multi-scale environmental geometric feature tensor. These environmental geometric features include the edges, planar features, and height variation features of the terrain structure, including: The 3D depth map is scanned pixel by pixel using three different window sizes: large, medium, and small. In a small window, the degree of difference in depth values ​​at each point within the window is calculated as an edge feature. Within a medium-sized window, the plane equation of the depth values ​​within the window is fitted to extract planar features; In a large window, the overall height relationship between the window center point and the depth values ​​of the surrounding area is calculated as a height variation feature. The edge features, planar features, and height variation features obtained at the same pixel location under three different window sizes are merged into a multi-dimensional feature vector. Arrange the multi-dimensional feature vectors of all pixel locations according to the image coordinates to form a multi-scale environmental geometric feature tensor.

5. The method according to claim 3, characterized in that, The preset rule means that when the semantic category corresponding to the first discrimination criterion is the safe plane semantic region, and the speed descent gradient in the second discrimination criterion is greater than the first threshold and the speed fluctuation value is less than the second threshold, then the user's landing intention is judged as a confirmatory intention. Otherwise, it is judged as an exploratory intent.

6. The method according to claim 2, characterized in that, The method of predicting the continuous position points of the user's feet in three-dimensional space over a future period through a deductive structure with temporal memory function, and generating a trajectory prediction sequence, includes: It receives multi-scale environmental geometric feature tensors, user joint angle sequences, and historical gait cycle data; In the extrapolation structure, based on the temporal memory function, the received data is temporally modeled to infer the foot movement state at future moments using data from the current and historical moments; The inference is performed iteratively to continuously generate predicted positions of the foot in three-dimensional space at multiple future moments, forming a trajectory prediction sequence.

7. The method according to claim 1, characterized in that, The step of calculating the phase from the distorted infrared image and generating a first depth map, and simultaneously calculating the disparity from the infrared image pair and generating a second depth map, includes: The coded pattern at each pixel in the distorted infrared image is periodically analyzed, and the phase value corresponding to each pixel is obtained by calculating the brightness change sequence of the coded pattern point by point. The phase value of each pixel is mapped to the corresponding distance value to form the first depth map; For the left-view infrared image and the right-view infrared image in the infrared image pair, the corresponding pixel region in the right-view infrared image is found by pixel-by-pixel region matching, and the horizontal offset between the two is calculated as the disparity value. Based on the fixed spacing and imaging focal length of the binocular infrared camera, the disparity value of each pixel is converted into a distance value to form a second depth map.

8. A personalized gait intent recognition system based on image recognition, characterized in that, include: The acquisition module is used to acquire infrared image pairs from the left and right perspectives and distorted infrared images with coded patterns through a binocular infrared camera integrated into the exoskeleton device. The first generation module is used to calculate the phase from the distorted infrared image and generate a first depth map, and at the same time calculate the disparity from the infrared image pair and generate a second depth map. The fusion module is used to perform pixel-level confidence-weighted fusion of the first depth map and the second depth map to output a three-dimensional depth map of the user's environment. The analysis module is used to perform multi-scale geometric analysis on the three-dimensional depth map, capture environmental geometric features, and generate a multi-scale environmental geometric feature tensor, wherein the environmental geometric features include the edge, planar and height variation features of the terrain structure. The processing module is used to input the environmental geometric feature tensor into the multi-task processing process and perform semantic segmentation of the user's foot landing area and foot trajectory prediction based on the user's real-time posture and historical gait in parallel. The second generation module is used to determine the user's foot placement intention based on the matching relationship between the semantic segmentation map obtained by semantic segmentation and the trajectory prediction sequence obtained by foot trajectory prediction, and to generate a corresponding intention instruction, wherein the foot placement intention includes exploratory intention or confirmatory intention.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the image recognition-based personalized gait intention recognition method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the image recognition-based personalized gait intention recognition method as described in any one of claims 1 to 7.

Images