Robot-assisted wound treatment method and system based on multimodal image analysis

Abstract

The present invention provides a robot-assisted wound treatment method and system based on multimodal image analysis, which relates to the field of robotics technology. The method comprises the following steps: collecting multimodal images of a wound to be treated, determining image registration results, obtaining wound area images through a multi-scale fusion algorithm and wound area segmentation, performing wound tissue type classification, and obtaining wound tissue type classification results; performing surround shooting of the wound area to obtain multi-perspective images of the wound, using a first-frame depth map as a reference, registering subsequent depth maps frame by frame and accumulating point cloud data to determine a three-dimensional point cloud of the wound, combining a surface evolution algorithm to generate a three-dimensional reconstruction result of the wound, and determining wound parameter measurement results; using a robot path planning algorithm based on the wound tissue type classification results and the wound parameter measurement results to generate an optimal motion trajectory of the robot, and adjusting the robot's motion parameters in real time through a robot control algorithm with vision and force feedback to complete wound-assisted treatment.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a robot-assisted wound treatment method and system based on multimodal image analysis. Background Art

[0002] Wound treatment is a critical and challenging task in the medical field. Traditional wound treatment methods rely primarily on the experience and judgment of medical staff, which can be subjective and inefficient. With the advancement of medical technology, robotic-assisted wound treatment has gained increasing attention. Applying robotic technology to wound treatment can improve treatment efficiency, reduce the workload of medical staff, and achieve standardized and precise treatment processes.

[0003] Single-modality images are often used, but they struggle to fully reflect the multiple attributes of a wound, limiting accurate assessment of its condition. Traditional robotic-assisted wound treatment systems often rely on pre-defined empirical rules based on specific wound characteristics. However, this approach lacks adaptability to dynamic changes in wounds and struggles to cope with complex and changing clinical scenarios. Existing robotic motion planning methods typically generate the robot's motion trajectory based on a single objective, while the wound treatment process involves multiple objectives. Single-objective optimization struggles to balance multiple treatment objectives. Due to the limitations of existing technologies, the effectiveness and adaptability of robotic-assisted wound treatment struggle to meet clinical needs.

[0004] In summary, there is an urgent need for a robot-assisted wound treatment method that can comprehensively utilize multimodal image information, dynamically adapt to changes in wound status, balance multiple treatment goals, and achieve autonomous learning and optimization to improve the effectiveness and efficiency of wound treatment. The present invention can solve the problems in the existing technology. Summary of the Invention

[0005] The embodiments of the present invention provide a robot-assisted wound treatment method and system based on multimodal image analysis, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] Provides a robot-assisted wound treatment method based on multimodal image analysis, including:

[0008] Acquiring multimodal images of a wound to be treated, performing geometric transformation and spatial alignment on the multimodal images to determine an image registration result, reconstructing the image registration result using a multi-scale fusion algorithm to determine a fused wound image, performing wound region segmentation on the fused wound image using a convolutional neural network-based method to obtain a segmented wound region image, and classifying the segmented wound region image into wound tissue types using a transfer learning-based method to obtain a wound tissue type classification result;

[0009] Surround the wound area and capture multi-view images of the wound. Based on the multi-view images of the wound, the first frame depth map corresponding to the first frame image is used as a reference. The subsequent depth maps are aligned frame by frame and the point cloud data is accumulated to determine the 3D point cloud of the wound. The 3D reconstruction of the wound is generated through point cloud smoothing combined with a surface evolution algorithm. Based on the 3D reconstruction of the wound, the wound parameter measurement results are determined.

[0010] Based on the wound tissue type classification results and wound parameter measurement results, a robot path planning algorithm is used to construct a reward goal for wound treatment and combine it with the penalty constraints of robot motion to generate the optimal robot motion trajectory. Through the robot control algorithm with vision and force feedback, the robot's motion parameters are adjusted in real time to complete wound-assisted treatment.

[0011] In an optional embodiment,

[0012] Acquiring multimodal images of a wound to be treated, performing geometric transformation and spatial alignment on the multimodal images, determining image registration results, and reconstructing the image registration results using a multi-scale fusion algorithm to determine a fused wound image includes:

[0013] Based on a pre-trained local feature descriptor network, the method detects and extracts local stable feature points from multimodal images. Using one modality image as the target image, the method determines the target feature point set, while using the other modality image as the source image, the method determines the source feature point set. Using an attention mechanism, the method searches for candidate matching points in the target feature point set that are most similar to each local stable feature point in the source feature point set. Using the Ransac algorithm to eliminate false matches, the method establishes sparse feature correspondences between local stable feature points in images of different modalities.

[0014] Based on the sparse feature correspondence, the dense spatial mapping from the source image to the target image is determined through affine transformation, perspective transformation and thin plate spline interpolation. The mapping training is performed by combining the cycle consistency loss function and the gradient correlation loss function to obtain the multimodal image registration result.

[0015] Each modality image in the multimodal image registration result is subjected to wavelet transform respectively, and a wavelet coefficient pyramid is obtained by layer-by-layer decomposition. The fusion weight is calculated based on the variance characteristics of the wavelet coefficients, and the wavelet coefficients are fused based on the fusion weight. The fused wound image is obtained by inverse wavelet transform.

[0016] In an optional embodiment,

[0017] Performing wound region segmentation on the fused wound image using a convolutional neural network-based method to obtain a segmented wound region image, and performing wound tissue type classification on the segmented wound region image using a transfer learning-based method to obtain wound tissue type classification results including:

[0018] Taking the fused wound image as input, an encoder-decoder architecture is used to construct a wound area segmentation neural network. The encoder is built on EfficientNet-B7, extracts shallow features, and generates a multi-scale semantic feature map through dilated convolution and pyramid pooling. The decoder fuses the shallow features in the encoder through progressive upsampling and skip connections, gradually restoring the spatial resolution of the multi-scale semantic feature map. A resolution restoration module is set at the end of the decoder. Through multi-scale convolution and attention mechanism, the granularity of the multi-scale semantic feature map with restored spatial resolution is repaired to generate a wound segmentation probability map.

[0019] Based on binary cross entropy loss, Dice similarity loss, and contrast loss, a loss function of the wound area segmentation neural network is constructed. Through iterative training, the distance between the wound area and the background area in the feature space is maximized to obtain the optimal wound segmentation probability map and determine the wound area segmentation result;

[0020] Based on the wound area segmentation results, ResNeXt-101 pre-trained on ImageNet is used as a feature extractor. Through adversarial autoencoding, the distribution of feature space is adjusted. The adversarial game between the generator and discriminator networks is used to learn the distribution of real wound tissue features and determine the low-dimensional feature representation of the wound.

[0021] Multiple local patches are sampled within the wound area, features are extracted from multiple scales and directions, and combined with the low-dimensional feature representation of the wound to obtain a multi-view representation of the wound tissue. The multi-view representation is then fused through a multi-head attention pooling mechanism to obtain globally consistent tissue type features.

[0022] Based on metric learning, a tissue type discrimination model is constructed. By iteratively adjusting the structure of the feature embedding space, combining pairwise loss and triplet loss, the intra-class consistency and inter-class difference are increased to perform tissue type classification. According to the center similarity between the sample to be classified and each tissue type, the nearest neighbor classification is determined.

[0023] A tissue type discrimination model was constructed based on metric learning. By optimizing the structure of the feature embedding space, samples of the same category were clustered together and samples of different categories were kept apart. Intra-class consistency and inter-class differences were enhanced through pairwise loss and triplet loss. The nearest neighbor classification was performed according to the similarity between the samples to be classified and the centers of each category to obtain the wound tissue type classification results.

[0024] In an optional embodiment,

[0025] Surround the wound area and obtain multi-view images of the wound. Based on the multi-view images of the wound, the first frame depth map corresponding to the first frame image is used as a reference. The subsequent depth maps are aligned frame by frame and the point cloud data is accumulated to determine the three-dimensional point cloud of the wound. Through point cloud smoothing and combined with the surface evolution algorithm, the three-dimensional reconstruction results of the wound are generated, including:

[0026] A regular dodecahedron is arranged, and a high-resolution main camera with a resolution greater than a preset resolution threshold and multiple low-resolution auxiliary cameras with a resolution less than a preset resolution threshold are placed on each face of the regular dodecahedron to form a multi-camera array. The wound area is surrounded by the multi-camera array to obtain a compound eye multi-view image.

[0027] Based on the multi-scale stereo matching algorithm, a matching scale is selected through a multi-scale phase pyramid. At each matching scale, a disparity map is obtained through local phase consistency constraints and global energy optimization. The disparity map is converted into a corresponding basic depth map by combining the camera focal length and camera baseline distance. A Markov random field model of the basic depth map is constructed, and an energy loss function is determined. The minimum value of the energy loss function is iteratively solved to obtain a depth estimation result, and the optimal depth map is determined to form a multi-scale depth map.

[0028] Based on the Harris-Laplace operator, local extreme points in the compound-eye multi-view image are detected in the scale space to determine the feature points. Based on the scale and position of the feature points, the descriptors corresponding to the feature points are calculated to generate feature vectors. Based on the feature vectors corresponding to two feature points, the feature similarity is calculated. A weighted graph is constructed with the feature points as nodes and the feature similarity as the weight of the edges between the nodes. A graph-based ant colony algorithm is used to perform a search process on the weighted graph by ants, and iterative exploration is performed to find the global optimal feature match and determine the feature matching result.

[0029] Based on the multi-scale depth map and the feature matching results, multi-scale point cloud registration is performed. By minimizing the distance error between points in the point cloud, the rigid body transformation matrix is ​​adjusted. In combination with the local deformation non-rigid constraint, the affine transformation matrix is ​​estimated to compensate for the non-rigid deformation. At the same time, the weight coefficient is adjusted between different scales to balance the rigid deformation and non-rigid deformation. Based on the preset skeletal motion model, the wound surface is divided into multiple local rigid body areas. By constructing the skeletal structure and calculating the joint parameters, the point cloud motion deformation state is determined to obtain the point cloud registration result.

[0030] Based on the point cloud registration results, an implicit surface evolution method is adopted. By defining the initial level set function, the point cloud is embedded in the three-dimensional space, and the surface is approximated to the point cloud by minimizing the energy functional. Combined with the curvature regularization term, the smoothness and continuity of the surface are controlled. The level set function is updated by numerical solution, and the zero level set is extracted as the current surface. The shape prior based on tensor integral is introduced. By obtaining the properties and structure of the wound surface, the local tangent space is constructed, and the tensor integral is defined. By minimizing the tensor integral, the optimal evolution direction of the surface is determined, and finally the three-dimensional reconstruction result of the wound is obtained.

[0031] In an optional embodiment,

[0032] The energy loss function is formulated as follows:

[0033]

[0034] Among them, E represents the energy loss function, p represents the pixel index, q represents another pixel index, N p represents the neighborhood of pixel p, w pq represents the similarity weight between pixel p and pixel q, ρ(·) represents the robust function for handling outliers, and I p Represents the image intensity at pixel p, I q represents the image intensity at pixel q, d p Indicates the depth value of pixel p, d q represents the depth value of pixel q, λ represents the balance coefficient, and α represents the truncation threshold.

[0035] In an optional embodiment,

[0036] Based on the wound tissue type classification results and wound parameter measurement results, a robot path planning algorithm is used to construct a reward goal for wound treatment and combine it with the penalty constraints of robot motion to generate the optimal robot motion trajectory, including:

[0037] A mapping relationship between wound status and healing stages is established based on a machine learning algorithm to construct a wound status assessment model, wherein the wound status includes wound tissue type distribution, area, and color, and the healing stages include inflammation, proliferation, and remodeling;

[0038] Based on the output results of the wound status assessment model, the weight of the reward target is adjusted in real time. When the wound status changes, the corresponding weight coefficient in the reward target is updated to determine the dynamic reward weight, including: when the healing stage is in the inflammatory stage, the weight coefficient corresponding to tissue clearance is increased, and the weight coefficient corresponding to tissue protection is decreased; when the healing stage is in the proliferative stage, the weight coefficient corresponding to tissue protection is increased, and the weight coefficient corresponding to tissue migration is decreased;

[0039] Based on the wound treatment trajectory data collected in advance through expert teaching, the corresponding wound state features and treatment motion features are extracted to construct state-action pairs;

[0040] Based on the maximum entropy inverse reinforcement learning algorithm, the parameters of the reward function are determined by maximizing the likelihood probability of the wound treatment trajectory data to obtain the initial reward function. The gradient descent algorithm is used to minimize the difference between the wound treatment trajectory data and the generated trajectory obtained by the maximum entropy inverse reinforcement learning algorithm. The parameters of the reward function are adjusted to obtain the implicit reward function.

[0041] Different reward goals are used as independent objective functions, combined with dynamic reward weights and implicit reward functions, and the robot motion constraints are used as constraints to construct a multi-objective optimization model. Based on the multi-objective optimization model, a multi-objective evolutionary algorithm is used for optimization and solution to obtain an optimal solution set. Each optimal solution in the Pareto optimal solution set corresponds to a robot motion trajectory.

[0042] In combination with the current treatment needs and the doctor's wishes, a robot motion trajectory is selected from the Pareto optimal solution set to determine the optimal robot motion trajectory.

[0043] In an optional embodiment,

[0044] The multi-objective evolutionary algorithm includes:

[0045] Based on preset historical data and prior knowledge, a knowledge-guided initialization strategy is adopted to determine the initial population;

[0046] Taking the initial population as the current population, for each individual in the current population, calculate the standard fitness value under each independent objective function corresponding to different reward targets;

[0047] A grid-based adaptive crowding measurement mechanism dynamically adjusts the grid size according to the distribution characteristics of the current population in the target space and calculates the individual crowding by combining the grid density and the Euclidean distance between individuals.

[0048] Based on the standard fitness value, combined with the dominance relationship of each individual in the current population and the individual crowding, the standard fitness value is dynamically adjusted, and the individual fitness value is determined through weight aggregation. Based on the individual fitness value, the individuals in the current population are sorted to determine the optimal individual, and the parent individual is determined according to the preset preferred number;

[0049] Based on simulating binary crossover and polynomial mutation, a crossover mutation operator is determined, and the crossover mutation operator parameters are adjusted based on the number of iterations. The adjusted crossover mutation operator is used to perform a crossover mutation operation on the parent individuals to generate offspring individuals. Based on the parent individuals, random individuals are determined through uniform sampling and Gaussian perturbation according to a preset generation ratio. The optimal individuals, offspring individuals, and random individuals are merged to generate an extended population.

[0050] Repeat the iteration until the preset number of iterations is reached to obtain the optimal extended population. Based on the individual fitness value of each individual in the optimal extended population, the optimal solution set is obtained according to the preset preferred number.

[0051] According to a second aspect of the embodiments of the present invention,

[0052] A robot-assisted wound treatment system based on multimodal image analysis is provided, comprising:

[0053] A first unit is configured to acquire multimodal images of a wound to be treated, perform geometric transformation and spatial alignment on the multimodal images, determine an image registration result, reconstruct the image registration result using a multi-scale fusion algorithm to determine a fused wound image, perform wound region segmentation on the fused wound image using a convolutional neural network-based method to obtain a segmented wound region image, and perform wound tissue type classification on the segmented wound region image using a transfer learning-based method to obtain a wound tissue type classification result;

[0054] The second unit is used to perform surround shooting of the wound area to obtain multi-view images of the wound. Based on the multi-view images of the wound, the first frame depth map corresponding to the first frame image is used as a reference to register subsequent depth maps frame by frame and accumulate point cloud data to determine a three-dimensional point cloud of the wound. Through point cloud smoothing and combined with a surface evolution algorithm, a three-dimensional reconstruction result of the wound is generated. Based on the three-dimensional reconstruction result of the wound, the measurement results of the wound parameters are determined;

[0055] The third unit is used to use a robot path planning algorithm based on the wound tissue type classification results and wound parameter measurement results. By constructing a reward goal for wound treatment and combining it with the penalty constraints of the robot motion, it generates the optimal motion trajectory of the robot. Through the robot control algorithm of vision and force feedback, it adjusts the robot's motion parameters in real time to complete wound-assisted treatment.

[0056] According to a third aspect of the embodiments of the present invention,

[0057] An electronic device is provided, comprising:

[0058] processor;

[0059] a memory for storing processor-executable instructions;

[0060] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0061] According to a fourth aspect of the embodiments of the present invention,

[0062] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0063] In an embodiment of the present invention, by fusing visible light images, infrared images and three-dimensional structured light images, more comprehensive and accurate wound information can be obtained; information from different modalities complements each other, providing more comprehensive wound status information, which helps to comprehensively assess the severity and healing status of the wound; through geometric transformation and spatial alignment, images of different modalities are ensured to be precisely aligned in the same spatial coordinate system, thereby improving the accuracy of subsequent analysis; using a multi-scale fusion algorithm, useful information in multi-modal images can be effectively fused, improving the quality and information content of the fused image; providing high-precision three-dimensional wound reconstruction and accurate wound parameter measurement results, significantly improving the scientificity and effectiveness of wound diagnosis and treatment. Through sophisticated point cloud processing and surface reconstruction, noise and errors are reduced, ensuring the continuity and smoothness of the reconstruction results, thereby providing comprehensive and detailed wound information and helping medical personnel make more accurate diagnosis and treatment decisions; by constructing reward goals, the system can prioritize the path that is most beneficial to wound treatment and improve treatment effectiveness; combined with penalty constraints on robot motion, the robot is prevented from causing secondary damage to the wound during treatment, ensuring safety; through real-time monitoring of the wound treatment process through visual feedback, deviations are promptly discovered and corrected to ensure the accuracy of the treatment process; the force feedback system is used to sense the pressure applied by the robot, preventing the impact of excessive pressure on the wound and providing gentler treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 Schematic diagram of a process of a robot-assisted wound treatment method based on multimodal image analysis according to an embodiment of the present invention;

[0065] Figure 2 Schematic diagram of the structure of a robot-assisted wound treatment system based on multimodal image analysis according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0067] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0068] Figure 1 FIG. 1 is a flow chart of a robot-assisted wound treatment method based on multimodal image analysis according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0069] S101. Acquire multimodal images of the wound to be treated, perform geometric transformation and spatial alignment on the multimodal images, determine image registration results, reconstruct the image registration results through a multi-scale fusion algorithm to determine a fused wound image, perform wound area segmentation on the fused wound image through a method based on a convolutional neural network to obtain a segmented wound area image, and perform wound tissue type classification on the segmented wound area image through a method based on transfer learning to obtain a wound tissue type classification result.

[0070] Multimodal images are collected, including visible light images, infrared images and three-dimensional structured light images. Visible light images are collected by a visible light camera to show the color and texture characteristics of the wound; infrared images are collected by an infrared thermal imager to reflect the temperature distribution of the wound area; three-dimensional structured light images are collected by a structured light scanner to obtain the three-dimensional morphology information of the wound.

[0071] Perform geometric transformation and spatial alignment, and perform necessary preprocessing on visible light images, infrared images, and three-dimensional structured light images, such as noise removal and contrast enhancement. Use geometric transformation technology to transform images of different modalities into a common coordinate system, involving operations such as translation, rotation, and scaling. Use image registration algorithms, such as feature point matching-based methods, to spatially align multimodal images to ensure that they are precisely aligned in the same spatial coordinate system.

[0072] Mutual information is used to evaluate the quality of the image registration results to ensure that the alignment accuracy meets the requirements. Based on the evaluation results, it may be necessary to further optimize the parameters of the registration algorithm to improve the alignment accuracy.

[0073] A multi-scale fusion algorithm is applied to reconstruct the registered multimodal images, fusing information from different modalities, such as the color and texture of visible light, the temperature distribution of infrared, and the morphology of three-dimensional structured light, into one image to form a fused wound image with rich information.

[0074] Determine the convolutional neural network model, train and infer the fused wound image, and segment the wound area; label the training data to clarify the wound area, use the labeled data to train the convolutional neural network, optimize the model parameters, use the validation set to evaluate the model performance, adjust the model structure or parameters, and obtain the segmented wound area image.

[0075] Using a pre-trained deep learning model and the transfer learning method, the segmented wound area images are subjected to feature extraction and classification. The pre-trained model is used to extract high-level features of the wound images. The classifier is trained on a small amount of labeled wound tissue type data, and the model is adjusted to suit the specific task to obtain classification results of wound tissue types, such as necrotic tissue, granulation tissue, epithelial tissue, etc.

[0076] In this embodiment, by fusing visible light images, infrared images and three-dimensional structured light images, more comprehensive and accurate wound information can be obtained; information from different modalities complement each other, providing more comprehensive wound status information, which helps to comprehensively assess the severity and healing status of the wound; through geometric transformation and spatial alignment, images of different modalities are precisely aligned in the same spatial coordinate system, thereby improving the accuracy of subsequent analysis; using a multi-scale fusion algorithm, useful information in multimodal images can be effectively fused, improving the quality and information content of the fused image; wound area segmentation based on convolutional neural networks can automatically and accurately segment the wound area, reducing the workload and errors of manual labeling; improving the degree of automation and efficiency of wound area segmentation, saving medical resources and time; through transfer learning, pre-trained models are used to achieve efficient wound tissue type classification on limited labeled data, thereby improving the accuracy and robustness of classification; different tissue types in the wound, such as necrotic tissue, granulation tissue, epithelial tissue, etc., can be finely distinguished, providing more detailed information for the formulation of treatment plans.

[0077] In an optional embodiment, acquiring multimodal images of a wound to be treated, performing geometric transformation and spatial alignment on the multimodal images, determining image registration results, reconstructing the image registration results using a multi-scale fusion algorithm, and determining a fused wound image includes:

[0078] Based on a pre-trained local feature descriptor network, the method detects and extracts local stable feature points from multimodal images. Using one modality image as the target image, the method determines the target feature point set, while using the other modality image as the source image, the method determines the source feature point set. Using an attention mechanism, the method searches for candidate matching points in the target feature point set that are most similar to each local stable feature point in the source feature point set. Using the Ransac algorithm to eliminate false matches, the method establishes sparse feature correspondences between local stable feature points in images of different modalities.

[0079] Based on the sparse feature correspondence, the dense spatial mapping from the source image to the target image is determined through affine transformation, perspective transformation and thin plate spline interpolation. The mapping training is performed by combining the cycle consistency loss function and the gradient correlation loss function to obtain the multimodal image registration result.

[0080] Each modality image in the multimodal image registration result is subjected to wavelet transform respectively, and a wavelet coefficient pyramid is obtained by layer-by-layer decomposition. The fusion weight is calculated based on the variance characteristics of the wavelet coefficients, and the wavelet coefficients are fused based on the fusion weight. The fused wound image is obtained by inverse wavelet transform.

[0081] Select a pre-trained local feature descriptor network, preferably SIFT, and input the multimodal image into the selected feature descriptor network. The network extracts local stable feature points of the image through a series of convolutions, pooling and nonlinear transformations. For each feature point, the network generates a high-dimensional feature descriptor vector to represent the local image feature of the point.

[0082] An image of one modality is selected as the target image and the other as the source image. In the target image, a set of local stable feature points is detected and extracted by the feature descriptor network to form a target feature point set. In the source image, another set of local stable feature points is detected and extracted by the feature descriptor network to form a source feature point set.

[0083] For each feature point in the source feature point set, the most similar candidate matching point is searched in the target feature point set. The attention mechanism is used to measure the similarity between feature points. The attention weights of each point in the source feature point and target feature point set are calculated, and the target feature point with the highest attention weight is selected as the candidate matching point of the current source feature point.

[0084] The Ransac algorithm is used to perform robustness estimation on candidate matching point pairs. A minimum number of matching point pairs are randomly sampled and the transformation model between them is calculated, such as affine transformation or homography transformation. According to the transformation model, all candidate matching point pairs are classified, and the point pairs that meet the model are marked as inliers, and those that do not meet the model are marked as outliers. The random sampling and classification process is repeated multiple times, and the transformation model with the largest number of inliers is selected as the optimal model. Based on the optimal model, the incorrect matches marked as outliers are eliminated to obtain the refined feature point correspondence.

[0085] The refined feature point correspondence is used as the sparse feature correspondence between images of different modalities. The coordinate position and correspondence of each pair of matching feature points are recorded. Based on the sparse feature correspondence, the dense spatial mapping from the source image to the target image is estimated. Using affine transformation, perspective transformation or thin plate spline interpolation methods, each pixel in the source image is mapped to the corresponding position in the target image. The parameters of the mapping function are optimized by minimizing the mapping error and regularization term.

[0086] A cycle consistency loss function is used to encourage the consistency of forward mapping and reverse mapping, and a gradient correlation loss function is used to promote the correlation between the gradient of the mapped image and the gradient of the target image. Through the backpropagation algorithm, the parameters of the mapping function are updated to minimize the overall loss function. The optimization process is iterated until the preset number of iterations or convergence conditions are reached.

[0087] Using the trained optimal mapping function, each pixel in the source image is mapped to the corresponding position in the target image to generate a registered source image so that it is spatially aligned with the target image.

[0088] Wavelet transform is performed on each modality image after registration, and the image is decomposed layer by layer using wavelet basis functions to obtain a wavelet coefficient pyramid. At each decomposition level, wavelet coefficients representing different frequency and directional characteristics of the image are obtained.

[0089] The variance characteristics of the wavelet coefficients of each modality image are analyzed, and the fusion weights of different modality images at each decomposition level are calculated according to the variance of the wavelet coefficients. Wavelet coefficients with larger variances are assigned higher weights, indicating that they contain more significant feature information.

[0090] According to the calculated fusion weights, the wavelet coefficients of different modal images are weightedly fused. At each decomposition level, the wavelet coefficients at the corresponding position are multiplied by their corresponding fusion weights and then added together to obtain a fused wavelet coefficient pyramid, which represents the characteristics of the fused image in different frequencies and directions.

[0091] The fused wavelet coefficient pyramid is subjected to inverse wavelet transform, and the wavelet coefficients are gradually merged through layer-by-layer reconstruction to finally obtain a fused wound image. The fused image integrates the features and detail information of images of different modalities, providing a more comprehensive and information-rich wound representation.

[0092] In this embodiment, stable local feature points are extracted to improve the accuracy of feature matching between images of different modalities; the Ransac algorithm is used to eliminate erroneous matches, optimize the correspondence between feature points, and improve the robustness and reliability of the registration results; the registered image is subjected to wavelet transform to decompose the multi-scale information of the image, and the fusion weight is calculated based on the variance characteristics of the wavelet coefficients, so that coefficients with larger variance account for a larger proportion in the fusion, thereby enhancing the expressiveness of the fused image; a pre-trained local feature descriptor network is used to automatically extract image features, reduce manual intervention, and improve processing efficiency; the use of the pre-trained feature descriptor network can achieve good feature extraction and matching on limited data, thereby improving the generalization ability of the model; and the transfer learning method can effectively utilize the advantages of the pre-trained model to improve the accuracy of wound image classification and segmentation.

[0093] In an optional embodiment, wound area segmentation is performed on the fused wound image using a convolutional neural network-based method to obtain a segmented wound area image, and wound tissue type classification is performed on the segmented wound area image using a transfer learning-based method. The wound tissue type classification result obtained includes:

[0094] Taking the fused wound image as input, an encoder-decoder architecture is used to construct a wound area segmentation neural network. The encoder is built on EfficientNet-B7, extracts shallow features, and generates a multi-scale semantic feature map through dilated convolution and pyramid pooling. The decoder fuses the shallow features in the encoder through progressive upsampling and skip connections, gradually restoring the spatial resolution of the multi-scale semantic feature map. A resolution restoration module is set at the end of the decoder. Through multi-scale convolution and attention mechanism, the granularity of the multi-scale semantic feature map with restored spatial resolution is repaired to generate a wound segmentation probability map.

[0095] Based on binary cross entropy loss, Dice similarity loss, and contrast loss, a loss function of the wound area segmentation neural network is constructed. Through iterative training, the distance between the wound area and the background area in the feature space is maximized to obtain the optimal wound segmentation probability map and determine the wound area segmentation result;

[0096] Based on the wound area segmentation results, ResNeXt-101 pre-trained on ImageNet is used as a feature extractor. Through adversarial autoencoding, the distribution of feature space is adjusted. The adversarial game between the generator and discriminator networks is used to learn the distribution of real wound tissue features and determine the low-dimensional feature representation of the wound.

[0097] Multiple local patches are sampled within the wound area, features are extracted from multiple scales and directions, and combined with the low-dimensional feature representation of the wound to obtain a multi-view representation of the wound tissue. The multi-view representation is then fused through a multi-head attention pooling mechanism to obtain globally consistent tissue type features.

[0098] Based on metric learning, a tissue type discrimination model is constructed. By iteratively adjusting the structure of the feature embedding space, combining pairwise loss and triplet loss, the intra-class consistency and inter-class difference are increased to perform tissue type classification. According to the center similarity between the sample to be classified and each tissue type, the nearest neighbor classification is determined.

[0099] A tissue type discrimination model was constructed based on metric learning. By optimizing the structure of the feature embedding space, samples of the same category were clustered together and samples of different categories were kept apart. Intra-class consistency and inter-class differences were enhanced through pairwise loss and triplet loss. The nearest neighbor classification was performed according to the similarity between the samples to be classified and the centers of each category to obtain the wound tissue type classification results.

[0100] Taking the fused wound image as input, a wound area segmentation neural network based on the encoder-decoder architecture is constructed. The encoder part adopts the pre-trained EfficientNet-B7 network to extract the shallow features of the input image through a series of convolution and downsampling operations. In the feature extraction process of the encoder, the dilated convolution and pyramid pooling techniques are introduced to generate a feature map with multi-scale semantic information. The decoder part fuses the shallow features in the encoder with the features of the decoder through progressive upsampling and skip connections, and gradually restores the spatial resolution of the feature map. At the end of the decoder, a resolution restoration module is set. Through multi-scale convolution and attention mechanism, the multi-scale semantic feature map with restored spatial resolution is fine-grainedly repaired. Through the resolution restoration module, a pixel-level wound segmentation probability map is generated, indicating the probability of each pixel belonging to the wound area.

[0101] A loss function for the wound area segmentation neural network was constructed, including binary cross entropy loss, Dice similarity loss, and contrast loss. Binary cross entropy loss measures the pixel-level difference between the segmentation result and the true label, encouraging the network to correctly predict the category of each pixel. Dice similarity loss calculates the overlap between the segmentation result and the true label, promoting the network to generate segmentation results that highly overlap with the true wound area. Contrast loss enhances the network's ability to distinguish wound areas by maximizing the distance between the wound area and the background area in the feature space. Through the iterative training process, the network parameters are continuously adjusted to minimize the overall loss function and obtain the optimal wound segmentation probability map.

[0102] Based on the wound area segmentation results, the pre-trained ResNeXt-101 network is used as a feature extractor to extract the high-dimensional features of the wound area. Through the adversarial autoencoding technology, the distribution of the feature space is adjusted, and the low-dimensional representation of the real wound tissue features is learned. The generator and discriminator networks are introduced. Through the adversarial game process, the generator attempts to generate features similar to the real wound features, and the discriminator attempts to distinguish between the generated features and the real features. Through the continuous game between the generator and the discriminator, the generator learns to capture the essence of the real wound tissue features and obtains a compact and expressive low-dimensional feature representation.

[0103] Multiple local blocks are sampled within the wound area, and local features are extracted from different scales and directions. The features of the local blocks are combined with the low-dimensional feature representation of the wound to obtain a multi-view representation of the wound tissue, capturing tissue features at different granularities and angles. A multi-head attention pooling mechanism is adopted to calculate the correlation and importance between features of different views through the self-attention mechanism. According to the attention weight, the multi-view features are weightedly fused to obtain a globally consistent and information-rich tissue type feature representation.

[0104] A tissue type discrimination model is constructed based on the principle of metric learning. A feature embedding space is learned so that samples of the same category are clustered in the embedding space, and samples of different categories are far away from each other in the embedding space. Through pairwise loss and triplet loss, the structure of the feature embedding space is optimized to enhance intra-class consistency and inter-class difference. Pairwise loss encourages samples of the same category to be close in the embedding space and samples of different categories to be far away in the embedding space. Triplet loss minimizes the distance between anchor samples and positive samples and maximizes the distance between anchor samples and negative samples by selecting anchor samples, positive samples and negative samples. Through an iterative optimization process, the feature embedding space is continuously adjusted so that samples of different categories are well separated and clustered in the embedding space. For the wound tissue sample to be classified, it is mapped to the learned feature embedding space, and the similarity with the center of each tissue type is calculated. According to the similarity score, the tissue type most similar to the sample to be classified is selected as the final classification result.

[0105] In this embodiment, the pre-trained EfficientNet-B7 is used as the encoder, which can take advantage of its rich features learned on large-scale data sets. The dilated convolution and pyramid pooling techniques are introduced in the encoder to capture multi-scale semantic information and enhance the expressiveness of the feature map. The decoder part improves the accuracy of wound area segmentation through skip connections and progressive upsampling; a resolution restoration module is set at the end of the decoder to perform fine-grained repair and generate a more accurate segmentation probability map; by introducing multi-scale semantic information and fine-grained repair mechanism, the segmentation ability of the model in complex scenes is enhanced, and the accuracy of the segmentation results is improved. Detail restoration and accuracy; construct a comprehensive loss function including binary cross entropy loss, Dice similarity loss and contrast loss to measure the quality of segmentation results from multiple angles and improve the performance of the segmentation model; introduce an adversarial game mechanism between the generator and the discriminator to capture the essence of real wound tissue characteristics and improve the quality and generalization ability of feature representation; calculate the correlation and importance between features of different views through a multi-head attention pooling mechanism, perform weighted fusion, and improve the global consistency and information richness of feature representation; construct a tissue type discrimination model based on metric learning to enhance intra-class consistency and inter-class differences and improve classification accuracy.

[0106] S102. Surround the wound area with images to obtain multi-view images of the wound. Based on the multi-view images of the wound, using the first frame depth map corresponding to the first frame as a reference, register the subsequent depth maps frame by frame and accumulate point cloud data to determine a three-dimensional point cloud of the wound. Smoothing the point cloud and combining it with a surface evolution algorithm generates a three-dimensional reconstruction of the wound. Based on the three-dimensional reconstruction of the wound, determine the wound parameter measurement results.

[0107] Use a camera or other imaging device to capture surround images of the wound area, obtain multi-perspective images of the wound from different angles, and form a set of continuous wound image sequences.

[0108] The depth map corresponding to the first frame is selected as a reference, and the depth maps of subsequent frames are registered with the reference depth map. The registration process aligns the depth maps of subsequent frames with the reference depth map by estimating the geometric transformation relationship between adjacent frames. The registered depth map is converted into 3D point cloud data and accumulated into the wound point cloud model.

[0109] The accumulated wound point cloud is smoothed to remove noise and outliers, improving its quality. Surface evolution algorithms, such as the moving least squares method or Poisson surface reconstruction algorithm, are used to convert the point cloud data into a continuous, smooth 3D surface model. Through an iterative optimization process, a high-quality 3D reconstruction of the wound is generated.

[0110] Based on the 3D reconstruction results of the wound, various wound parameters are extracted and calculated, such as wound area, volume, depth, length, etc. By analyzing the geometric features of the 3D surface model, such as curvature and normal vector, the morphological parameters of the wound can be obtained.

[0111] This embodiment provides high-precision 3D wound reconstruction and accurate wound parameter measurement, significantly improving the scientificity and effectiveness of wound diagnosis and treatment. Through sophisticated point cloud processing and surface reconstruction, noise and errors are reduced, ensuring the continuity and smoothness of the reconstruction results. This provides comprehensive and detailed wound information, helping medical personnel make more accurate diagnostic and treatment decisions.

[0112] In an optional embodiment, a wound area is photographed in a surround manner to obtain a multi-view image of the wound. Based on the multi-view image of the wound, the first frame depth map corresponding to the first frame image is used as a reference, and subsequent depth maps are registered frame by frame and point cloud data is accumulated to determine a three-dimensional point cloud of the wound. The three-dimensional reconstruction result of the wound is generated by smoothing the point cloud and combining it with a surface evolution algorithm, including:

[0113] A regular dodecahedron is arranged, and a high-resolution main camera with a resolution greater than a preset resolution threshold and multiple low-resolution auxiliary cameras with a resolution less than a preset resolution threshold are placed on each face of the regular dodecahedron to form a multi-camera array. The wound area is surrounded by the multi-camera array to obtain a compound eye multi-view image.

[0114] Based on the multi-scale stereo matching algorithm, a matching scale is selected through a multi-scale phase pyramid. At each matching scale, a disparity map is obtained through local phase consistency constraints and global energy optimization. The disparity map is converted into a corresponding basic depth map by combining the camera focal length and camera baseline distance. A Markov random field model of the basic depth map is constructed, and an energy loss function is determined. The minimum value of the energy loss function is iteratively solved to obtain a depth estimation result, and the optimal depth map is determined to form a multi-scale depth map.

[0115] Based on the Harris-Laplace operator, local extreme points in the compound-eye multi-view image are detected in the scale space to determine the feature points. Based on the scale and position of the feature points, the descriptors corresponding to the feature points are calculated to generate feature vectors. Based on the feature vectors corresponding to two feature points, the feature similarity is calculated. A weighted graph is constructed with the feature points as nodes and the feature similarity as the weight of the edges between the nodes. A graph-based ant colony algorithm is used to perform a search process on the weighted graph by ants, and iterative exploration is performed to find the global optimal feature match and determine the feature matching result.

[0116] Based on the multi-scale depth map and the feature matching results, multi-scale point cloud registration is performed. By minimizing the distance error between points in the point cloud, the rigid body transformation matrix is ​​adjusted. In combination with the local deformation non-rigid constraint, the affine transformation matrix is ​​estimated to compensate for the non-rigid deformation. At the same time, the weight coefficient is adjusted between different scales to balance the rigid deformation and non-rigid deformation. Based on the preset skeletal motion model, the wound surface is divided into multiple local rigid body areas. By constructing the skeletal structure and calculating the joint parameters, the point cloud motion deformation state is determined to obtain the point cloud registration result.

[0117] Based on the point cloud registration results, an implicit surface evolution method is adopted. By defining the initial level set function, the point cloud is embedded in the three-dimensional space, and the surface is approximated to the point cloud by minimizing the energy functional. Combined with the curvature regularization term, the smoothness and continuity of the surface are controlled. The level set function is updated by numerical solution, and the zero level set is extracted as the current surface. The shape prior based on tensor integral is introduced. By obtaining the properties and structure of the wound surface, the local tangent space is constructed, and the tensor integral is defined. By minimizing the tensor integral, the optimal evolution direction of the surface is determined, and finally the three-dimensional reconstruction result of the wound is obtained.

[0118] A regular dodecahedron is arranged, and cameras are placed on each face of the dodecahedron. A high-resolution main camera and multiple low-resolution auxiliary cameras are placed on each face to form a multi-camera array. The resolution of the high-resolution main camera is greater than a preset resolution threshold, and the resolution of the low-resolution auxiliary cameras is less than the preset resolution threshold. The wound area is surrounded by the multi-camera array to obtain wound images from different angles. The images obtained are composed of compound-eye multi-perspective images to provide all-round information about the wound.

[0119] A multi-scale stereo matching algorithm is adopted. By constructing a multi-scale phase pyramid and selecting the appropriate matching scale, the disparity map is calculated at each matching scale using local phase consistency constraints and global energy optimization. The disparity map is converted into the corresponding basic depth map by combining the focal length and baseline distance information of the camera. A Markov random field model of the basic depth map is constructed, and an energy loss function is defined. The minimum value of the energy loss function is solved through iterative optimization to obtain the depth estimation result. The depth estimation result with the minimum energy loss is selected as the optimal depth map. The optimal depth maps at different scales are combined to form a multi-scale depth map.

[0120] The Harris-Laplace operator is used to detect local extreme points of compound-eye multi-view images in scale space, determine feature points, calculate the descriptor corresponding to each feature point based on the scale and position information of the feature points, generate feature vectors, and calculate the similarity between the feature vectors corresponding to two feature points. This is used as the basis for feature matching, and a weighted graph is constructed with feature points as nodes and feature similarity as the weight of the edges between nodes. A graph-based ant colony algorithm is used to iteratively explore the optimal feature matching by simulating the search process of ants on the weighted graph. The globally optimal feature matching result is found through the ants' pheromone update and path selection.

[0121] Multi-scale point cloud registration is performed using multi-scale depth maps and feature matching results. By minimizing the distance error between the points in the point cloud, the rigid body transformation matrix is ​​estimated and adjusted to achieve rigid registration of the point cloud. Local deformation non-rigid constraints are introduced. The non-rigid deformation of the point cloud is compensated by estimating the affine transformation matrix. The weight coefficient is adjusted between different scales to balance the effects of rigid and non-rigid deformations. The preset skeletal motion model is used to divide the wound surface into multiple local rigid body areas. The motion deformation state of the point cloud is determined by constructing the skeletal structure and calculating the joint parameters. The rigid deformation, non-rigid deformation and skeletal motion are comprehensively considered to obtain the final point cloud registration result.

[0122] An implicit surface evolution method is used to perform three-dimensional reconstruction of the point cloud registration results. The initial level set function is defined, and the point cloud is embedded in three-dimensional space. The surface is approximated to the point cloud data by minimizing the energy functional. A curvature regularization term is introduced to control the smoothness and continuity of the surface. The level set function is iteratively updated through a numerical solution method, and the zero level set is extracted as the current surface. A shape prior based on tensor integral is introduced to obtain the properties and structural information of the wound surface. The local tangent space is constructed, and the tensor integral is defined. The optimal evolution direction of the surface is determined by minimizing the tensor integral. After multiple iterative evolutions, the final three-dimensional reconstruction of the wound is obtained.

[0123] In this embodiment, by forming a multi-camera array, wound images can be acquired from different angles to obtain compound-eye multi-perspective images, thereby providing all-round information of the wound; the disparity map is accurately calculated and converted into a depth map, and a Markov random field model of the basic depth map is constructed. The optimal depth map is obtained through energy loss function optimization, and finally a multi-scale depth map is formed to achieve high-precision depth estimation; the ant colony algorithm is used to iteratively explore the weighted graph to find the globally optimal feature matching result to ensure the accuracy of feature matching; the implicit surface evolution method is used to perform three-dimensional reconstruction of the point cloud registration results, and the level set function is iteratively updated through curvature regularization and shape prior based on tensor integral to finally obtain a smooth, continuous and accurate three-dimensional reconstruction result of the wound.

[0124] In an optional embodiment, the energy loss function is formulated as follows:

[0125]

[0126] Among them, E represents the energy loss function, p represents the pixel index, q represents another pixel index, N p represents the neighborhood of pixel p, w pq represents the similarity weight between pixel p and pixel q, ρ(·) represents the robust function for handling outliers, and I p Represents the image intensity at pixel p, I q represents the image intensity at pixel q, d p Indicates the depth value of pixel p, d q represents the depth value of pixel q, λ represents the balance coefficient, and α represents the truncation threshold.

[0127] The formula consists of two parts. The first part calculates the difference in image intensity between a pixel and its neighbors. This difference is weighted by similarity weights and processed with a robust function to reduce the influence of outliers on the result. The second part calculates the difference in depth value between a pixel and its neighbors. This difference is processed using similarity weights and a truncation threshold to minimize the impact of large depth differences on the energy function. A balance coefficient is used to adjust the weight of this component in the total energy.

[0128] According to the formula, by using robust functions and truncation thresholds, the negative impact of outliers and excessive depth differences on energy optimization is reduced, making the algorithm more stable and robust when dealing with noise and outliers; by introducing weights of image intensity differences and depth value differences, the energy loss function can more accurately reflect the true situation of the image, thereby improving the accuracy of the depth estimation results; the balance coefficient enables the energy function to reasonably weigh image intensity and depth information during the optimization process, avoiding bias towards one side, and ensuring the rationality and accuracy of the final depth map; by flexibly adjusting parameters and, the energy loss function can adapt to different image scenarios and application requirements, improving the applicability and breadth of the algorithm.

[0129] S103. Based on the wound tissue type classification results and wound parameter measurement results, a robot path planning algorithm is used to generate the optimal robot motion trajectory by constructing a reward goal for wound treatment and combining it with the penalty constraints of robot motion. The robot control algorithm based on vision and force feedback is used to adjust the robot's motion parameters in real time to complete wound-assisted treatment.

[0130] Based on the type and parameters of the wound, treatment reward objectives are determined. These objectives may include different treatment operations such as debridement, disinfection, and application of medication. A robot path planning algorithm is applied, taking into account the treatment objectives and penalty constraints of the robot's motion (such as avoiding collisions and saving energy), to generate the robot's optimal motion trajectory. The robot's optimal motion trajectory should be able to cover the entire area of ​​the wound and meet the predetermined treatment requirements. The robot control algorithm is implemented, combining vision and force feedback mechanisms to monitor the robot's position and operation in real time. Based on the feedback information, the robot's motion parameters are dynamically adjusted to ensure its motion trajectory is accurate and fine-tuned as needed. The robot performs auxiliary treatment operations according to the planned trajectory, ensuring comprehensive and precise treatment of the wound. During the treatment process, vision and force feedback are continuously used to promptly correct any deviations to achieve the best treatment effect.

[0131] In this embodiment, by constructing a reward goal, the system can give priority to the path that is most beneficial to wound treatment and improve the treatment effect; combined with the penalty constraint of the robot movement, the robot is prevented from causing secondary damage to the wound during the treatment process, ensuring safety; the wound treatment process is monitored in real time through visual feedback, deviations are discovered and corrected in time, and the accuracy of the treatment process is ensured; the force feedback system is used to sense the pressure applied by the robot, prevent the impact of excessive pressure on the wound, and provide gentler treatment; the automated robot control algorithm can autonomously perform complex treatment tasks, reducing errors and instability in human operations; the robot's motion parameters are adjusted in real time to make the treatment process more flexible and efficient, shortening the treatment time.

[0132] In an optional embodiment, based on the wound tissue type classification results and wound parameter measurement results, a robot path planning algorithm is used to construct a reward goal for wound treatment and combine it with the penalty constraints of the robot motion to generate the optimal robot motion trajectory, including:

[0133] A mapping relationship between wound status and healing stages is established based on a machine learning algorithm to construct a wound status assessment model, wherein the wound status includes wound tissue type distribution, area, and color, and the healing stages include inflammation, proliferation, and remodeling;

[0134] Based on the output results of the wound status assessment model, the weight of the reward target is adjusted in real time. When the wound status changes, the corresponding weight coefficient in the reward target is updated to determine the dynamic reward weight, including: when the healing stage is in the inflammatory stage, the weight coefficient corresponding to tissue clearance is increased, and the weight coefficient corresponding to tissue protection is decreased; when the healing stage is in the proliferative stage, the weight coefficient corresponding to tissue protection is increased, and the weight coefficient corresponding to tissue migration is decreased;

[0135] Based on the wound treatment trajectory data collected in advance through expert teaching, the corresponding wound state features and treatment motion features are extracted to construct state-action pairs;

[0136] Based on the maximum entropy inverse reinforcement learning algorithm, the parameters of the reward function are determined by maximizing the likelihood probability of the wound treatment trajectory data to obtain the initial reward function. The gradient descent algorithm is used to minimize the difference between the wound treatment trajectory data and the generated trajectory obtained by the maximum entropy inverse reinforcement learning algorithm. The parameters of the reward function are adjusted to obtain the implicit reward function.

[0137] Different reward goals are used as independent objective functions, combined with dynamic reward weights and implicit reward functions, and the robot motion constraints are used as constraints to construct a multi-objective optimization model. Based on the multi-objective optimization model, a multi-objective evolutionary algorithm is used for optimization and solution to obtain an optimal solution set. Each optimal solution in the Pareto optimal solution set corresponds to a robot motion trajectory.

[0138] In combination with the current treatment needs and the doctor's wishes, a robot motion trajectory is selected from the Pareto optimal solution set to determine the optimal robot motion trajectory.

[0139] Collect wound images and corresponding wound status information, including wound tissue type distribution, area and color, collect annotation information of wound healing stages, including inflammatory stage, proliferation stage and remodeling stage, extract wound image features such as texture, shape, color, etc. as model input, use machine learning algorithms such as support vector machine and random forest to establish a mapping relationship between wound status and healing stage, optimize model parameters through training data set, and evaluate model performance on validation set to obtain a trained wound status assessment model for real-time assessment of wound status and healing stage.

[0140] Based on the output results of the wound status assessment model, the wound healing stage is judged in real time. When the wound status changes, the corresponding weight coefficient in the reward target is dynamically adjusted. When the healing stage is in the inflammatory stage, the weight coefficient corresponding to tissue clearance is increased, and the weight coefficient corresponding to tissue protection is reduced. When the healing stage is in the proliferative stage, the weight coefficient corresponding to tissue protection is increased, and the weight coefficient corresponding to tissue migration is reduced. Based on the adjusted weight coefficient, the dynamic reward weight is determined for subsequent motion trajectory optimization.

[0141] Through expert teaching, trajectory data from the wound treatment process is collected in advance, preprocessed, and the corresponding wound state features and treatment motion features are extracted. The extracted wound state features and treatment motion features are paired to construct state-action pairs. The state-action pairs represent the treatment motion operations performed by the expert under specific wound states. The constructed state-action pairs are used as input to the inverse reinforcement learning algorithm to learn the implicit reward function.

[0142] The maximum entropy inverse reinforcement learning algorithm is used to maximize the likelihood probability of wound treatment trajectory data to determine the initial parameters of the reward function. The gradient descent algorithm is used to minimize the difference between the wound treatment trajectory data and the trajectory generated by the maximum entropy inverse reinforcement learning algorithm. Through iterative optimization, the parameters of the reward function are adjusted to make the generated trajectory as similar as possible to the trajectory taught by the expert. The optimized implicit reward function is obtained and used to evaluate the quality of the robot's motion trajectory.

[0143] Different reward goals, such as tissue clearance, tissue protection, and tissue migration, are used as independent objective functions. Combined with dynamic reward weights and implicit reward functions, a multi-objective optimization model is constructed. The constraints of robot motion, such as joint angle limits and speed limits, are used as constraints of the optimization model. The objective function and constraints together constitute a multi-objective optimization problem, which is used to solve the optimal robot motion trajectory.

[0144] A multi-objective evolutionary algorithm is used to solve the constructed multi-objective optimization model. A set of candidate solutions is generated through population initialization. Each candidate solution represents a possible robot motion trajectory. The individuals in the population are evaluated, and the fitness value of each individual on different objective functions is calculated. Through operations such as selection, crossover, and mutation, the population is continuously iterated and optimized to continuously improve the quality of the solution. After multiple iterations, a Pareto optimal solution set is obtained, where each optimal solution corresponds to a robot motion trajectory. The Pareto optimal solution set represents a set of optimal trajectories that balance different objectives.

[0145] According to the current treatment needs and the doctor's wishes, a robot motion trajectory is selected from the Pareto optimal solution set. The doctor can comprehensively consider the importance of different goals based on clinical experience and treatment goals and choose the trajectory that best suits the current situation. The selected robot motion trajectory is used as the optimal trajectory to guide the robot to perform wound treatment tasks.

[0146] In this embodiment, the wound status assessment model established by the machine learning algorithm can judge the wound healing stage in real time and provide an accurate basis for treatment; according to the output results of the assessment model, the weight coefficient of the treatment target is adjusted in real time to ensure that the most appropriate treatment strategy is adopted in different healing stages; the treatment trajectory data is collected through expert teaching, and the implicit reward function is learned by the inverse reinforcement learning algorithm, so that the robot can intelligently simulate the expert's treatment operation and improve the level of automation; the maximum entropy inverse reinforcement learning algorithm is used to obtain the implicit reward function through iterative optimization, so that the generated trajectory is as similar as possible to the expert's trajectory, ensuring the accuracy and effectiveness of the treatment; a multi-objective optimization model combining dynamic reward weights and implicit reward functions is constructed, and comprehensive consideration is given to the implicit reward function. Taking into account different goals such as tissue clearance, tissue protection, and tissue migration, the optimal robot motion trajectory is generated; a multi-objective evolutionary algorithm is used to iteratively optimize through operations such as population initialization, selection, crossover, and mutation to obtain a Pareto optimal solution set, ensuring that the optimal trajectory is found in the trade-off between different goals; a set of optimal trajectory options that weigh different goals is provided to doctors, who can choose the trajectory that best suits the current situation based on clinical experience and treatment goals to achieve personalized treatment; automated path planning and real-time feedback control reduce errors in human operation and improve the stability and consistency of treatment; through inverse reinforcement learning and expert teaching, the expert's treatment experience and knowledge are integrated into the robot system, reducing dependence on the operator's skills.

[0147] In an optional embodiment, the multi-objective evolutionary algorithm includes:

[0148] Based on preset historical data and prior knowledge, a knowledge-guided initialization strategy is adopted to determine the initial population;

[0149] Taking the initial population as the current population, for each individual in the current population, calculate the standard fitness value under each independent objective function corresponding to different reward targets;

[0150] A grid-based adaptive crowding measurement mechanism dynamically adjusts the grid size according to the distribution characteristics of the current population in the target space and calculates the individual crowding by combining the grid density and the Euclidean distance between individuals.

[0151] Based on the standard fitness value, combined with the dominance relationship of each individual in the current population and the individual crowding, the standard fitness value is dynamically adjusted, and the individual fitness value is determined through weight aggregation. Based on the individual fitness value, the individuals in the current population are sorted to determine the optimal individual, and the parent individual is determined according to the preset preferred number;

[0152] Based on simulating binary crossover and polynomial mutation, a crossover mutation operator is determined, and the crossover mutation operator parameters are adjusted based on the number of iterations. The adjusted crossover mutation operator is used to perform a crossover mutation operation on the parent individuals to generate offspring individuals. Based on the parent individuals, random individuals are determined through uniform sampling and Gaussian perturbation according to a preset generation ratio. The optimal individuals, offspring individuals, and random individuals are merged to generate an extended population.

[0153] Repeat the iteration until the preset number of iterations is reached to obtain the optimal extended population. Based on the individual fitness value of each individual in the optimal extended population, the optimal solution set is obtained according to the preset preferred number.

[0154] Collect and organize historical data and prior knowledge related to the optimization problem. Based on the historical data, extract valuable information, such as the characteristics of high-quality solutions and the value ranges of important variables. Use the prior knowledge to set the generation rules of the initial solution, such as the constraint relationship between variables, heuristic strategies, etc. Combine historical data and prior knowledge to generate a set of high-quality initial solutions as the initial population. The individuals in the initial population should be diverse, covering different areas of the solution space, and provide a good starting point for subsequent evolution.

[0155] Taking the initial population as the current population, each individual in the population is evaluated. According to the different reward objectives of the optimization problem, the fitness value of each individual under each independent objective function is calculated. The fitness value measures the performance of the individual on a single objective. The higher the fitness value, the better the individual performs on the objective. For each objective function, the fitness values ​​of all individuals in the population are calculated to obtain the standard fitness value.

[0156] Analyze the distribution characteristics of the current population in the target space, such as the density and clustering of individuals. Based on the population distribution characteristics, dynamically adjust the grid size of the target space so that the grid can adapt to the distribution of the population. Divide the target space into multiple grids, calculate the number of individuals in each grid, and obtain the grid density. For each individual, calculate the Euclidean distance between it and other individuals. Combined with the grid density, calculate the individual's crowding degree. The crowding degree reflects the degree of crowding around the individual. The higher the crowding degree, the more individuals there are in the area where the individual is located.

[0157] Based on the standard fitness value, the dominance relationship between individuals is taken into account and the fitness value is adjusted. If individual A dominates individual B, the fitness value of individual A should be better than that of individual B. Combined with the crowding degree of the individuals, the adjusted fitness value is further revised to maintain the diversity of the population. The adjusted fitness values ​​of multiple targets are weighted and aggregated to obtain the comprehensive fitness value of the individual. The individuals in the current population are sorted according to the comprehensive fitness value. The higher the fitness value, the higher the ranking. A certain number of individuals with the highest ranking are selected as parents for subsequent crossover and mutation operations.

[0158] Simulated binary crossover and polynomial mutation are used as crossover and mutation operators. According to the current number of iterations, the parameters of the crossover and mutation operators, such as crossover probability and mutation probability, are dynamically adjusted. The parent individuals are crossovered in a certain pairing manner to generate new offspring individuals. The parent individuals are mutated in a certain mutation probability to introduce new genetic information. The crossover and mutation operations are designed to explore new solutions and inherit the excellent characteristics of the parent individuals.

[0159] In addition to generating offspring individuals through crossover mutation, a certain proportion of random individuals also needs to be introduced. Random individuals are generated through uniform sampling and Gaussian perturbation to maintain the diversity of the population. Uniform sampling randomly generates individuals in the solution space. Gaussian perturbation adds random noise based on the parent individuals. According to the preset generation ratio, the number of random individuals is determined, and the optimal individuals, offspring individuals and random individuals are merged to form an extended population as the next generation population.

[0160] The population is iterated repeatedly, and the extended population is iteratively optimized until the preset number of iterations is reached. In each iteration, the fitness values ​​of the individuals in the extended population are evaluated, and they are sorted and selected. After multiple iterations, the overall quality of the population continues to improve and gradually converges to the optimal solution. When the preset number of iterations is reached, the optimal extended population is obtained. According to the comprehensive fitness values ​​of the individuals in the optimal extended population, the best performing individuals are selected according to the preset preferred number. The selected individuals constitute the optimal solution set, which serves as the Pareto optimal solution to the multi-objective optimization problem.

[0161] In this embodiment, historical data and prior knowledge are used to generate a high-quality initial population, which improves the starting point quality of the optimization process; the initial population is diverse and covers different areas of the solution space, providing a good foundation for subsequent evolution; the grid size of the target space is dynamically adjusted according to the population distribution characteristics to improve the accuracy of fitness calculation; by calculating the crowding degree of individuals, it is avoided that individuals in the population are too concentrated in a certain area, and the diversity of the population is maintained; the fitness values ​​of multiple objectives are comprehensively considered, and the comprehensive fitness value of individuals is obtained through weighted aggregation, which improves the effect of multi-objective optimization; after iterative optimization, the Pareto optimal solution set is finally obtained, which provides multiple optimal solutions that balance different objectives; these crossover mutation operators are used to dynamically adjust the parameters. The algorithm generates random individuals through uniform sampling and Gaussian perturbation to further maintain the diversity of the population and avoid the local optimal trap. It dynamically adjusts the parameters of crossover mutation according to the number of iterations to adapt to different stages of the optimization process and improve the optimization efficiency. It calculates the standard fitness value according to different objective functions to ensure the accurate performance evaluation of each individual. It adjusts the fitness value based on the dominance relationship between individuals to ensure the dominant position of high-quality individuals in the population. It ensures that the selected parent individuals have high quality through comprehensive fitness value sorting and selection, and improves the excellent characteristics of subsequent offspring individuals. The final Pareto optimal solution set provides a variety of optimal balance schemes between different objectives, providing more options for decision-making.

[0162] Figure 2 FIG is a schematic structural diagram of a robot-assisted wound treatment system based on multimodal image analysis according to an embodiment of the present invention. Figure 2 As shown, the system includes:

[0163] A first unit is configured to acquire multimodal images of a wound to be treated, perform geometric transformation and spatial alignment on the multimodal images, determine an image registration result, reconstruct the image registration result using a multi-scale fusion algorithm to determine a fused wound image, perform wound region segmentation on the fused wound image using a convolutional neural network-based method to obtain a segmented wound region image, and perform wound tissue type classification on the segmented wound region image using a transfer learning-based method to obtain a wound tissue type classification result;

[0164] The second unit is used to perform surround shooting of the wound area to obtain multi-view images of the wound. Based on the multi-view images of the wound, the first frame depth map corresponding to the first frame image is used as a reference to register subsequent depth maps frame by frame and accumulate point cloud data to determine a three-dimensional point cloud of the wound. Through point cloud smoothing and combined with a surface evolution algorithm, a three-dimensional reconstruction result of the wound is generated. Based on the three-dimensional reconstruction result of the wound, the measurement results of the wound parameters are determined;

[0165] The third unit is used to use a robot path planning algorithm based on the wound tissue type classification results and wound parameter measurement results. By constructing a reward goal for wound treatment and combining it with the penalty constraints of the robot motion, it generates the optimal motion trajectory of the robot. Through the robot control algorithm of vision and force feedback, it adjusts the robot's motion parameters in real time to complete wound-assisted treatment.

[0166] According to a third aspect of the embodiments of the present invention,

[0167] An electronic device is provided, comprising:

[0168] processor;

[0169] a memory for storing processor-executable instructions;

[0170] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0171] According to a fourth aspect of the embodiments of the present invention,

[0172] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0173] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot-assisted wound treatment method based on multimodal image analysis, characterized in that: include: Acquiring multimodal images of a wound to be treated, performing geometric transformation and spatial alignment on the multimodal images to determine an image registration result, reconstructing the image registration result using a multi-scale fusion algorithm to determine a fused wound image, performing wound region segmentation on the fused wound image using a convolutional neural network-based method to obtain a segmented wound region image, and classifying the segmented wound region image into wound tissue types using a transfer learning-based method to obtain a wound tissue type classification result; Surround the wound area and capture multi-view images of the wound. Based on the multi-view images of the wound, the first frame depth map corresponding to the first frame image is used as a reference. The subsequent depth maps are aligned frame by frame and the point cloud data is accumulated to determine the 3D point cloud of the wound. The 3D reconstruction of the wound is generated through point cloud smoothing combined with a surface evolution algorithm. Based on the 3D reconstruction of the wound, the wound parameter measurement results are determined. Based on the wound tissue type classification results and wound parameter measurement results, a robot path planning algorithm is used to generate the optimal robot motion trajectory by constructing a reward goal for wound treatment and combining it with the penalty constraints of robot motion, including: A mapping relationship between wound status and healing stages is established based on a machine learning algorithm to construct a wound status assessment model, wherein the wound status includes wound tissue type distribution, area, and color, and the healing stages include inflammation, proliferation, and remodeling; Based on the output results of the wound status assessment model, the weight of the reward target is adjusted in real time. When the wound status changes, the corresponding weight coefficient in the reward target is updated to determine the dynamic reward weight, including: when the healing stage is in the inflammatory stage, the weight coefficient corresponding to tissue clearance is increased, and the weight coefficient corresponding to tissue protection is decreased; when the healing stage is in the proliferative stage, the weight coefficient corresponding to tissue protection is increased, and the weight coefficient corresponding to tissue migration is decreased; Based on the wound treatment trajectory data collected in advance through expert teaching, the corresponding wound state features and treatment motion features are extracted to construct state-action pairs; Based on the maximum entropy inverse reinforcement learning algorithm, the parameters of the reward function are determined by maximizing the likelihood probability of the wound treatment trajectory data to obtain the initial reward function. The gradient descent algorithm is used to minimize the difference between the wound treatment trajectory data and the generated trajectory obtained by the maximum entropy inverse reinforcement learning algorithm. The parameters of the reward function are adjusted to obtain the implicit reward function. Different reward goals are used as independent objective functions, combined with dynamic reward weights and implicit reward functions, and the robot motion constraints are used as constraints to construct a multi-objective optimization model. Based on the multi-objective optimization model, a multi-objective evolutionary algorithm is used for optimization and solution to obtain an optimal solution set. Each optimal solution in the Pareto optimal solution set corresponds to a robot motion trajectory. In combination with current treatment needs and doctor's wishes, a robot motion trajectory is selected from the Pareto optimal solution set to determine the optimal robot motion trajectory; Through the robot control algorithm of vision and force feedback, the robot's motion parameters are adjusted in real time to complete wound-assisted treatment.

2. The method according to claim 1, characterized in that Acquiring multimodal images of a wound to be treated, performing geometric transformation and spatial alignment on the multimodal images, determining image registration results, and reconstructing the image registration results using a multi-scale fusion algorithm to determine a fused wound image includes: Based on a pre-trained local feature descriptor network, the method detects and extracts local stable feature points from multimodal images. Using one modality image as the target image, the method determines the target feature point set, while using the other modality image as the source image, the method determines the source feature point set. Using an attention mechanism, the method searches for candidate matching points in the target feature point set that are most similar to each local stable feature point in the source feature point set. Using the Ransac algorithm to eliminate false matches, the method establishes sparse feature correspondences between local stable feature points in images of different modalities. Based on the sparse feature correspondence, the dense spatial mapping from the source image to the target image is determined through affine transformation, perspective transformation and thin plate spline interpolation. The mapping training is performed by combining the cycle consistency loss function and the gradient correlation loss function to obtain the multimodal image registration result. Each modality image in the multimodal image registration result is subjected to wavelet transform respectively, and a wavelet coefficient pyramid is obtained by layer-by-layer decomposition. The fusion weight is calculated based on the variance characteristics of the wavelet coefficients, and the wavelet coefficients are fused based on the fusion weight. The fused wound image is obtained by inverse wavelet transform.

3. The method according to claim 1, characterized in that Performing wound region segmentation on the fused wound image using a convolutional neural network-based method to obtain a segmented wound region image, and performing wound tissue type classification on the segmented wound region image using a transfer learning-based method to obtain wound tissue type classification results including: Taking the fused wound image as input, an encoder-decoder architecture is used to construct a wound area segmentation neural network. The encoder is built on EfficientNet-B7, extracts shallow features, and generates a multi-scale semantic feature map through dilated convolution and pyramid pooling. The decoder fuses the shallow features in the encoder through progressive upsampling and skip connections, gradually restoring the spatial resolution of the multi-scale semantic feature map. A resolution restoration module is set at the end of the decoder. Through multi-scale convolution and attention mechanism, the granularity of the multi-scale semantic feature map with restored spatial resolution is repaired to generate a wound segmentation probability map. Based on binary cross entropy loss, Dice similarity loss, and contrast loss, a loss function of the wound area segmentation neural network is constructed. Through iterative training, the distance between the wound area and the background area in the feature space is maximized to obtain the optimal wound segmentation probability map and determine the wound area segmentation result; Based on the wound area segmentation results, ResNeXt-101 pre-trained on ImageNet is used as a feature extractor. Through adversarial autoencoding, the distribution of feature space is adjusted. The adversarial game between the generator and discriminator networks is used to learn the distribution of real wound tissue features and determine the low-dimensional feature representation of the wound. Multiple local patches are sampled within the wound area, features are extracted from multiple scales and directions, and combined with the low-dimensional feature representation of the wound to obtain a multi-view representation of the wound tissue. The multi-view representation is then fused through a multi-head attention pooling mechanism to obtain globally consistent tissue type features. Based on metric learning, a tissue type discrimination model is constructed. By iteratively adjusting the structure of the feature embedding space, combining pairwise loss and triplet loss, the intra-class consistency and inter-class difference are increased to perform tissue type classification. According to the center similarity between the sample to be classified and each tissue type, the nearest neighbor classification is determined. A tissue type discrimination model was constructed based on metric learning. By optimizing the structure of the feature embedding space, samples of the same category were clustered together and samples of different categories were kept apart. Intra-class consistency and inter-class differences were enhanced through pairwise loss and triplet loss. The nearest neighbor classification was performed according to the similarity between the samples to be classified and the centers of each category to obtain the wound tissue type classification results.

4. The method according to claim 1, wherein Surround the wound area and obtain multi-view images of the wound. Based on the multi-view images of the wound, the first frame depth map corresponding to the first frame image is used as a reference. The subsequent depth maps are aligned frame by frame and the point cloud data is accumulated to determine the three-dimensional point cloud of the wound. Through point cloud smoothing and combined with the surface evolution algorithm, the three-dimensional reconstruction results of the wound are generated, including: Setting an arrangement of a regular dodecahedron, placing a high-resolution main camera with a resolution greater than a preset resolution threshold and multiple low-resolution auxiliary cameras with a resolution less than the preset resolution threshold on each face of the regular dodecahedron to form a multi-camera array, and using the multi-camera array to capture a surround image of the wound area to obtain a compound-eye multi-view image; Based on the multi-scale stereo matching algorithm, a matching scale is selected through a multi-scale phase pyramid. At each matching scale, a disparity map is obtained through local phase consistency constraints and global energy optimization. The disparity map is converted into a corresponding basic depth map by combining the camera focal length and camera baseline distance. A Markov random field model of the basic depth map is constructed, and an energy loss function is determined. The minimum value of the energy loss function is iteratively solved to obtain a depth estimation result, and the optimal depth map is determined to form a multi-scale depth map. Based on the Harris-Laplace operator, local extreme points in the compound-eye multi-view image are detected in scale space to determine feature points. Based on the scale and position of the feature points, descriptors corresponding to the feature points are calculated to generate feature vectors. Feature similarity is calculated based on the feature vectors corresponding to two feature points. A weighted graph is constructed using the feature points as nodes and the feature similarities as weights of edges between nodes. A graph-based ant colony algorithm is used to perform a search process on the weighted graph by ants, iteratively exploring to find a global optimal feature match and determine a feature matching result. Based on the multi-scale depth map and the feature matching results, multi-scale point cloud registration is performed. By minimizing the distance error between points in the point cloud, the rigid body transformation matrix is ​​adjusted. In combination with the local deformation non-rigid constraint, the affine transformation matrix is ​​estimated to compensate for the non-rigid deformation. At the same time, the weight coefficient is adjusted between different scales to balance the rigid deformation and non-rigid deformation. Based on the preset skeletal motion model, the wound surface is divided into multiple local rigid body areas. By constructing the skeletal structure and calculating the joint parameters, the point cloud motion deformation state is determined to obtain the point cloud registration result. Based on the point cloud registration results, an implicit surface evolution method is adopted. By defining the initial level set function, the point cloud is embedded in the three-dimensional space, and the surface is approximated to the point cloud by minimizing the energy functional. Combined with the curvature regularization term, the smoothness and continuity of the surface are controlled. The level set function is updated by numerical solution, and the zero level set is extracted as the current surface. The shape prior based on tensor integral is introduced. By obtaining the properties and structure of the wound surface, the local tangent space is constructed, and the tensor integral is defined. By minimizing the tensor integral, the optimal evolution direction of the surface is determined, and finally the three-dimensional reconstruction result of the wound is obtained.

5. The method according to claim 4, characterized in that The energy loss function is formulated as follows: Among them, E represents the energy loss function, p represents the pixel index, q represents another pixel index, N p represents the neighborhood of pixel p, w pq represents the similarity weight between pixel p and pixel q, ρ(·) represents the robust function for handling outliers, and I p Represents the image intensity at pixel p, I q represents the image intensity at pixel q, d p Indicates the depth value of pixel p, d q represents the depth value of pixel q, λ represents the balance coefficient, and α represents the truncation threshold.

6. The method according to claim 1, characterized in that The multi-objective evolutionary algorithm includes: Based on preset historical data and prior knowledge, a knowledge-guided initialization strategy is adopted to determine the initial population; Taking the initial population as the current population, for each individual in the current population, calculate the standard fitness value under each independent objective function corresponding to different reward targets; A grid-based adaptive crowding measurement mechanism dynamically adjusts the grid size according to the distribution characteristics of the current population in the target space and calculates the individual crowding by combining the grid density and the Euclidean distance between individuals. Based on the standard fitness value, combined with the dominance relationship of each individual in the current population and the individual crowding, the standard fitness value is dynamically adjusted, and the individual fitness value is determined through weight aggregation. Based on the individual fitness value, the individuals in the current population are sorted to determine the optimal individual, and the parent individual is determined according to the preset preferred number; Based on simulating binary crossover and polynomial mutation, a crossover mutation operator is determined, and the crossover mutation operator parameters are adjusted based on the number of iterations. The adjusted crossover mutation operator is used to perform a crossover mutation operation on the parent individuals to generate offspring individuals. Based on the parent individuals, random individuals are determined through uniform sampling and Gaussian perturbation according to a preset generation ratio. The optimal individuals, offspring individuals, and random individuals are merged to generate an extended population. Repeat the iteration until the preset number of iterations is reached to obtain the optimal extended population. Based on the individual fitness value of each individual in the optimal extended population, the optimal solution set is obtained according to the preset preferred number.

7. A robot-assisted wound treatment system based on multimodal image analysis, for implementing the method according to any one of claims 1 to 6, characterized in that: include: A first unit is configured to acquire multimodal images of a wound to be treated, perform geometric transformation and spatial alignment on the multimodal images, determine an image registration result, reconstruct the image registration result using a multi-scale fusion algorithm to determine a fused wound image, perform wound region segmentation on the fused wound image using a convolutional neural network-based method to obtain a segmented wound region image, and perform wound tissue type classification on the segmented wound region image using a transfer learning-based method to obtain a wound tissue type classification result; The second unit is used to perform surround shooting of the wound area to obtain multi-view images of the wound. Based on the multi-view images of the wound, the first frame depth map corresponding to the first frame image is used as a reference to register subsequent depth maps frame by frame and accumulate point cloud data to determine a three-dimensional point cloud of the wound. Through point cloud smoothing and combined with a surface evolution algorithm, a three-dimensional reconstruction result of the wound is generated. Based on the three-dimensional reconstruction result of the wound, the measurement results of the wound parameters are determined; The third unit is used to use a robot path planning algorithm based on the wound tissue type classification results and wound parameter measurement results. By constructing a reward goal for wound treatment and combining it with the penalty constraints of the robot motion, it generates the optimal motion trajectory of the robot. Through the robot control algorithm of vision and force feedback, it adjusts the robot's motion parameters in real time to complete wound-assisted treatment.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images