Robot-assisted wound treatment method and system based on multi-modal image analysis

Abstract

The invention provides a multi-modal image analysis-based robot-assisted wound treatment method and system, and relates to the technical field of robots, and the method comprises the steps: collecting a multi-modal image of a to-be-treated wound, determining an image registration result, obtaining a wound region image through a multi-scale fusion algorithm and wound region segmentation, and carrying out the type classification of wound tissues. A wound tissue type classification result is obtained; performing surrounding shooting on a wound area to obtain a wound multi-view image, registering subsequent depth maps frame by frame by taking a first frame of depth map as a reference, accumulating point cloud data, determining a wound three-dimensional point cloud, generating a wound three-dimensional reconstruction result by combining a curved surface evolution algorithm, and determining a wound parameter measurement result; based on the wound tissue type classification result and the wound parameter measurement result, a robot path planning algorithm is adopted, the optimal movement track of the robot is generated, the movement parameters of the robot are adjusted in real time through a robot control algorithm of vision and force feedback, and wound adjuvant therapy is completed.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a robot-assisted wound treatment method and system based on multimodal image analysis. Background Art

[0002] Wound treatment is an important and challenging task in the medical field. Traditional wound treatment methods mainly rely on the experience and judgment of medical staff, which are highly subjective and inefficient. With the development of medical technology, robot-assisted treatment technology has gradually attracted attention. Applying robot technology to the field of wound treatment can improve treatment efficiency, reduce the workload of medical staff, and achieve standardized and precise treatment processes.

[0003] Single-modality images are usually used, but single-modality images are difficult to fully reflect the multiple attributes of the wound, limiting the accurate assessment of the wound status; traditional robot-assisted wound treatment systems usually rely on pre-set empirical rules based on the specific characteristics of the wound, but this method lacks the ability to adapt to the dynamic changes of the wound and is difficult to cope with complex and changing clinical scenarios; existing robot motion planning methods are usually based on a single goal to generate the robot's motion trajectory, while the wound treatment process involves multiple goals, and single-objective optimization is difficult to balance multiple treatment goals. Due to the limitations of existing technologies, the effect and adaptability of robot-assisted wound treatment are difficult to meet clinical needs.

[0004] In summary, there is an urgent need for a robot-assisted wound treatment method that can comprehensively utilize multimodal image information, dynamically adapt to changes in wound status, balance multiple treatment goals, and achieve autonomous learning and optimization to improve the effectiveness and efficiency of wound treatment. The present invention can solve the problems in the prior art. Summary of the invention

[0005] The embodiments of the present invention provide a robot-assisted wound treatment method and system based on multimodal image analysis, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] Provides a robot-assisted wound treatment method based on multimodal image analysis, including:

[0008] Acquire multimodal images of the wound to be treated, perform geometric transformation and spatial alignment on the multimodal images, determine image registration results, reconstruct the image registration results by a multi-scale fusion algorithm, determine a fused wound image, perform wound region segmentation on the fused wound image by a method based on a convolutional neural network to obtain a segmented wound region image, perform wound tissue type classification on the segmented wound region image by a method based on transfer learning to obtain a wound tissue type classification result;

[0009] Surrounding shooting of the wound area is performed to obtain a multi-view image of the wound. Based on the multi-view image of the wound, the first frame depth map corresponding to the first frame image is used as a reference, and subsequent depth maps are aligned frame by frame and point cloud data are accumulated to determine the three-dimensional point cloud of the wound. Through point cloud smoothing and combined with a surface evolution algorithm, a three-dimensional reconstruction result of the wound is generated. Based on the three-dimensional reconstruction result of the wound, the measurement result of the wound parameters is determined;

[0010] Based on the wound tissue type classification results and wound parameter measurement results, a robot path planning algorithm is used to generate the optimal robot motion trajectory by constructing a reward goal for wound treatment and combining it with the penalty constraints of robot motion. The robot control algorithm based on vision and force feedback is used to adjust the robot's motion parameters in real time to complete wound-assisted treatment.

[0011] In an optional embodiment,

[0012] Acquiring a multimodal image of a wound to be treated, performing geometric transformation and spatial alignment on the multimodal image, determining an image registration result, reconstructing the image registration result by a multiscale fusion algorithm, and determining a fused wound image includes:

[0013] Based on the pre-trained local feature descriptor network, the local stable feature points of multimodal images are detected and extracted. One modality image is used as the target image to determine the target feature point set, and the other modality image is used as the source image to determine the source feature point set. Through the attention mechanism, the candidate matching points that are most similar to each local stable feature point in the source feature point set are searched in the target feature point set. The Ransac algorithm is used to eliminate false matches and establish the sparse feature correspondence relationship between the local stable feature points of different modality images.

[0014] Based on the sparse feature correspondence, the dense spatial mapping from the source image to the target image is determined through affine transformation, perspective transformation and thin plate spline interpolation. The mapping training is performed by combining the cycle consistency loss function and the gradient correlation loss function to obtain the multimodal image registration result.

[0015] Each modality image in the multimodal image registration result is subjected to wavelet transform respectively, and a wavelet coefficient pyramid is obtained by layer-by-layer decomposition. The fusion weight is calculated based on the variance characteristics of the wavelet coefficients, and the wavelet coefficients are fused based on the fusion weight. The fused wound image is obtained by inverse wavelet transform.

[0016] In an optional embodiment,

[0017] The fused wound image is segmented into wound regions by a convolutional neural network-based method to obtain a segmented wound region image, and the segmented wound region image is classified into wound tissue types by a transfer learning-based method to obtain wound tissue type classification results including:

[0018] Taking the fused wound image as input, an encoder-decoder architecture is adopted to construct a neural network for wound area segmentation, wherein the encoder is constructed based on EfficientNet-B7, extracts shallow features, and generates a multi-scale semantic feature map through dilated convolution and pyramid pooling. The decoder fuses the shallow features in the encoder through progressive upsampling and skip connections, gradually restores the spatial resolution of the multi-scale semantic feature map, and sets a resolution restoration module at the end of the decoder. Through multi-scale convolution and attention mechanism, the multi-scale semantic feature map with restored spatial resolution is granularly repaired to generate a wound segmentation probability map.

[0019] Based on binary cross entropy loss, Dice similarity loss and contrast loss, a loss function of the wound area segmentation neural network is constructed, and through iterative training, the distance between the wound area and the background area in the feature space is maximized to obtain an optimal wound segmentation probability map and determine the wound area segmentation result;

[0020] Based on the wound area segmentation results, ResNeXt-101 pre-trained on ImageNet is used as a feature extractor, and the distribution of feature space is adjusted through adversarial autoencoding. The adversarial game between the generator and the discriminator network is used to learn the distribution of real wound tissue features and determine the low-dimensional feature representation of the wound;

[0021] Sampling multiple local blocks in the wound area, extracting features from multiple scales and multiple directions, combining with the low-dimensional feature representation of the wound to obtain a multi-view representation of the wound tissue, and fusing the multi-view representation through a multi-head attention pooling mechanism to obtain globally consistent tissue type features;

[0022] The tissue type discrimination model is constructed based on metric learning. The structure of the feature embedding space is iteratively adjusted, and the pairwise loss and triplet loss are combined to increase the intra-class consistency and inter-class difference, and the tissue type is divided. The nearest neighbor classification is determined according to the center similarity between the sample to be classified and each tissue type.

[0023] A tissue type discrimination model was constructed based on metric learning. By optimizing the structure of the feature embedding space, samples of the same category were clustered and samples of different categories were kept apart. The intra-class consistency and inter-class difference were enhanced through pairwise loss and triplet loss. The nearest neighbor classification was performed according to the similarity between the samples to be classified and the centers of each category to obtain the wound tissue type classification results.

[0024] In an optional embodiment,

[0025] Surround the wound area and obtain a multi-view image of the wound. Based on the multi-view image of the wound, the first frame depth map corresponding to the first frame image is used as a reference, and subsequent depth maps are registered frame by frame and point cloud data is accumulated to determine the three-dimensional point cloud of the wound. Through point cloud smoothing and combined with the surface evolution algorithm, the three-dimensional reconstruction results of the wound are generated, including:

[0026] A regular dodecahedron is arranged, and a high-resolution main camera with a resolution greater than a preset resolution threshold and multiple low-resolution auxiliary cameras with a resolution less than a preset resolution threshold are placed on each face of the regular dodecahedron to form a multi-camera array. The wound area is surrounded by the multi-camera array to obtain a compound eye multi-view image.

[0027] Based on the multi-scale stereo matching algorithm, a matching scale is selected through a multi-scale phase pyramid. At each matching scale, a disparity map is obtained through local phase consistency constraints and global energy optimization. The disparity map is converted into a corresponding basic depth map in combination with the camera focal length and the camera baseline distance. A Markov random field model of the basic depth map is constructed, an energy loss function is determined, and the minimum value of the energy loss function is iteratively solved to obtain a depth estimation result, determine the optimal depth map, and form a multi-scale depth map.

[0028] Based on the Harris-Laplace operator, the local extreme points in the compound-eye multi-view image are detected in the scale space to determine the feature points. Based on the scale and position of the feature points, the descriptors corresponding to the feature points are calculated to generate feature vectors. Based on the feature vectors corresponding to two feature points, the feature similarity is calculated. A weighted graph is constructed with the feature points as nodes and the feature similarity as the weights of the edges between the nodes. A graph-based ant colony algorithm is used to perform a search process on the weighted graph by ants, and iterative exploration is performed to find the global optimal feature match and determine the feature matching result.

[0029] Based on the multi-scale depth map and the feature matching result, multi-scale point cloud registration is performed, by minimizing the distance error between points in the point cloud, adjusting the rigid body transformation matrix, combining the local deformation non-rigid constraint, and estimating the affine transformation matrix to compensate for the non-rigid deformation. At the same time, the weight coefficient is adjusted between different scales to balance the rigid deformation and the non-rigid deformation. Based on the preset bone motion model, the wound surface is divided into multiple local rigid body areas, and the point cloud motion deformation state is determined by constructing the bone structure and calculating the joint parameters to obtain the point cloud registration result;

[0030] Based on the point cloud registration result, an implicit surface evolution method is adopted. The point cloud is embedded in three-dimensional space by defining an initial level set function. The surface is approximated to the point cloud by minimizing the energy functional. Combined with the curvature regularization term, the smoothness and continuity of the surface are controlled. The level set function is updated by numerical solution, and the zero level set is extracted as the current surface. The shape prior based on tensor integral is introduced. The local tangent space is constructed by obtaining the properties and structure of the wound surface, and the tensor integral is defined. The optimal evolution direction of the surface is determined by minimizing the tensor integral, and finally the three-dimensional reconstruction result of the wound is obtained.

[0031] In an optional embodiment,

[0032] The energy loss function has the following formula:

[0033]

[0034] Among them, E represents the energy loss function, p represents the pixel index, q represents another pixel index, and N p represents the neighborhood of pixel p, w pq represents the similarity weight between pixel p and pixel q, ρ(·) represents the robust function for handling outliers, and I p Represents the image intensity at pixel p, I q represents the image intensity at pixel q, d p Indicates the depth value of pixel p, d q represents the depth value of pixel q, λ represents the balance coefficient, and α represents the truncation threshold.

[0035] In an optional embodiment,

[0036] Based on the wound tissue type classification results and wound parameter measurement results, the robot path planning algorithm is used to construct the reward goal of wound treatment and combine the penalty constraints of robot motion to generate the optimal robot motion trajectory including:

[0037] A mapping relationship between wound status and healing stage is established based on a machine learning algorithm to construct a wound status assessment model, wherein the wound status includes wound tissue type distribution, area and color, and the healing stage includes inflammation stage, proliferation stage and remodeling stage;

[0038] Based on the output results of the wound status assessment model, the weight of the reward target is adjusted in real time. When the wound status changes, the corresponding weight coefficient in the reward target is updated to determine the dynamic reward weight, including: when the healing stage is in the inflammatory stage, the weight coefficient corresponding to tissue clearance is increased, and the weight coefficient corresponding to tissue protection is reduced; when the healing stage is in the proliferative stage, the weight coefficient corresponding to tissue protection is increased, and the weight coefficient corresponding to tissue migration is reduced;

[0039] Based on the wound treatment trajectory data collected in advance through expert teaching, the corresponding wound state features and treatment motion features are extracted to construct state-action pairs;

[0040] Based on the maximum entropy inverse reinforcement learning algorithm, the parameters of the reward function are determined by maximizing the likelihood probability of the wound treatment trajectory data to obtain the initial reward function. The difference between the wound treatment trajectory data and the generated trajectory obtained by the maximum entropy inverse reinforcement learning algorithm is minimized through the gradient descent algorithm, and the parameters of the reward function are adjusted to obtain the implicit reward function.

[0041] Taking different reward targets as independent objective functions, combining dynamic reward weights and implicit reward functions, taking robot motion constraints as constraint conditions, building a multi-objective optimization model, and using a multi-objective evolutionary algorithm to optimize and solve based on the multi-objective optimization model to obtain an optimal solution set, each optimal solution in the Pareto optimal solution set corresponds to a robot motion trajectory;

[0042] In combination with current treatment needs and doctor's wishes, a robot motion trajectory is selected from the Pareto optimal solution set to determine the optimal robot motion trajectory.

[0043] In an optional embodiment,

[0044] The multi-objective evolutionary algorithm comprises:

[0045] Based on preset historical data and prior knowledge, a knowledge-guided initialization strategy is adopted to determine the initial population;

[0046] Take the initial population as the current population, and for each individual in the current population, calculate the standard fitness value under each independent objective function corresponding to different reward targets;

[0047] The grid-based adaptive crowding measurement mechanism dynamically adjusts the grid size according to the distribution characteristics of the current population in the target space, and calculates the individual crowding by combining the grid density and the Euclidean distance between individuals.

[0048] Based on the standard fitness value, combined with the dominance relationship of each individual in the current population and the individual crowding, the standard fitness value is dynamically adjusted, and the individual fitness value is determined through weight aggregation. Based on the individual fitness value, the individuals in the current population are sorted to determine the optimal individual, and the parent individual is determined according to the preset preferred number;

[0049] Based on simulating binary crossover and polynomial mutation, a crossover mutation operator is determined, and the crossover mutation operator parameters are adjusted based on the number of iterations. The crossover mutation operator is used to perform a crossover mutation operation on the parent individuals to generate offspring individuals. Based on the parent individuals, random individuals are determined according to a preset generation ratio through uniform sampling and Gaussian perturbation. The optimal individuals, offspring individuals and random individuals are merged to generate an extended population.

[0050] Repeat the iteration until the preset number of iterations is reached to obtain the optimal extended population. Based on the individual fitness value of each individual in the optimal extended population, the optimal solution set is obtained according to the preset preferred number.

[0051] According to a second aspect of the embodiments of the present invention,

[0052] A robot-assisted wound treatment system based on multimodal image analysis is provided, comprising:

[0053] The first unit is used to collect multimodal images of a wound to be treated, perform geometric transformation and spatial alignment on the multimodal images, determine an image registration result, reconstruct the image registration result by a multi-scale fusion algorithm, determine a fused wound image, perform wound region segmentation on the fused wound image by a method based on a convolutional neural network to obtain a segmented wound region image, and perform wound tissue type classification on the segmented wound region image by a method based on transfer learning to obtain a wound tissue type classification result;

[0054] The second unit is used to perform surround shooting of the wound area to obtain a multi-view image of the wound, based on the multi-view image of the wound, taking the first frame depth map corresponding to the first frame image as a reference, aligning subsequent depth maps frame by frame and accumulating point cloud data, determining a three-dimensional point cloud of the wound, generating a three-dimensional reconstruction result of the wound through point cloud smoothing combined with a surface evolution algorithm, and determining a measurement result of wound parameters based on the three-dimensional reconstruction result of the wound;

[0055] The third unit is used to generate the optimal motion trajectory of the robot by using a robot path planning algorithm based on the wound tissue type classification results and wound parameter measurement results, by constructing a reward goal for wound treatment and combining it with the penalty constraints of the robot motion. The robot control algorithm based on vision and force feedback can adjust the robot's motion parameters in real time to complete wound-assisted treatment.

[0056] According to a third aspect of the embodiments of the present invention,

[0057] An electronic device is provided, comprising:

[0058] processor;

[0059] a memory for storing processor-executable instructions;

[0060] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0061] A fourth aspect of the embodiments of the present invention is:

[0062] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0063] In an embodiment of the present invention, by fusing visible light images, infrared images and three-dimensional structured light images, more comprehensive and accurate wound information can be obtained; information from different modalities complement each other, providing more comprehensive wound status information, which is helpful to comprehensively evaluate the severity and healing status of the wound; through geometric transformation and spatial alignment, it is ensured that images of different modalities are accurately aligned in the same spatial coordinate system, thereby improving the accuracy of subsequent analysis; using a multi-scale fusion algorithm, useful information in multi-modal images can be effectively fused, improving the quality and information content of the fused image; providing high-precision three-dimensional wound reconstruction and accurate wound parameter measurement results, significantly improving the scientificity and effectiveness of wound diagnosis and treatment. Through sophisticated point cloud processing and surface reconstruction, noise and errors are reduced, ensuring the continuity and smoothness of the reconstruction results, thereby providing comprehensive and detailed wound information to help medical personnel make more accurate diagnosis and treatment decisions; by building reward goals, the system can prioritize the path that is most beneficial to wound treatment and improve the treatment effect; combined with the penalty constraints of the robot's motion, the robot is prevented from causing secondary damage to the wound during the treatment process to ensure safety; the wound treatment process is monitored in real time through visual feedback, deviations are discovered and corrected in a timely manner, and the accuracy of the treatment process is ensured; the force feedback system is used to sense the pressure applied by the robot to prevent excessive pressure from affecting the wound and provide gentler treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is a schematic diagram of a process of a robot-assisted wound treatment method based on multimodal image analysis according to an embodiment of the present invention;

[0065] Figure 2 Schematic diagram of the structure of a robot-assisted wound treatment system based on multimodal image analysis according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0068] Figure 1 FIG. 1 is a flow chart of a robot-assisted wound treatment method based on multimodal image analysis according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0069] S101. Acquire multimodal images of the wound to be treated, perform geometric transformation and spatial alignment on the multimodal images, determine image registration results, reconstruct the image registration results through a multi-scale fusion algorithm, determine a fused wound image, perform wound area segmentation on the fused wound image through a method based on a convolutional neural network to obtain a segmented wound area image, perform wound tissue type classification on the segmented wound area image through a method based on transfer learning to obtain a wound tissue type classification result.

[0070] Multimodal images are collected, including visible light images, infrared images and three-dimensional structured light images. Visible light images are collected by visible light cameras to show the color and texture characteristics of the wound; infrared images are collected by infrared thermal imagers to reflect the temperature distribution of the wound area; three-dimensional structured light images are collected by structured light scanners to obtain three-dimensional morphological information of the wound.

[0071] Perform geometric transformation and spatial alignment, and perform necessary preprocessing on visible light images, infrared images, and three-dimensional structured light images, such as noise removal and contrast enhancement. Use geometric transformation technology to transform images of different modalities into a common coordinate system, involving operations such as translation, rotation, and scaling. Use image registration algorithms, such as feature point matching-based methods, to spatially align multimodal images to ensure that they are precisely aligned in the same spatial coordinate system.

[0072] Mutual information is used to evaluate the quality of the image registration results to ensure that the alignment accuracy meets the requirements. Based on the evaluation results, it may be necessary to further optimize the parameters of the registration algorithm to improve the alignment accuracy.

[0073] A multi-scale fusion algorithm is used to reconstruct the registered multimodal images, integrating information from different modalities, such as the color and texture of visible light, temperature distribution of infrared, and morphology of three-dimensional structured light, into one image to form a fused wound image with rich information.

[0074] Determine the convolutional neural network model, train and infer the fused wound image, and segment the wound area; label the training data, clarify the wound area, use the labeled data to train the convolutional neural network, optimize the model parameters, use the validation set to evaluate the model performance, adjust the model structure or parameters, and obtain the segmented wound area image.

[0075] Using a pre-trained deep learning model and the method of transfer learning, the segmented wound area images are subjected to feature extraction and classification. The pre-trained model is used to extract high-level features of the wound images. The classifier is trained on a small amount of labeled wound tissue type data, and the model is adjusted to adapt to specific tasks to obtain classification results of wound tissue types, such as necrotic tissue, granulation tissue, epithelial tissue, etc.

[0076] In this embodiment, by fusing visible light images, infrared images and three-dimensional structured light images, more comprehensive and accurate wound information can be obtained; the information of different modalities complement each other, providing more comprehensive wound status information, which is helpful to comprehensively evaluate the severity and healing status of the wound; through geometric transformation and spatial alignment, it is ensured that images of different modalities are accurately aligned in the same spatial coordinate system, thereby improving the accuracy of subsequent analysis; using a multi-scale fusion algorithm, the useful information in multi-modal images can be effectively fused to improve the quality and information content of the fused image; wound area segmentation based on convolutional neural networks can automatically and accurately segment the wound area, reducing the workload and errors of manual labeling; improving the degree of automation and efficiency of wound area segmentation, saving medical resources and time; through transfer learning, using pre-trained models to achieve efficient wound tissue type classification on limited labeled data, improving the accuracy and robustness of classification; different tissue types in the wound, such as necrotic tissue, granulation tissue, epithelial tissue, etc., can be finely distinguished, providing more detailed information for the formulation of treatment plans.

[0077] In an optional embodiment, acquiring a multimodal image of a wound to be treated, performing geometric transformation and spatial alignment on the multimodal image, determining an image registration result, reconstructing the image registration result by a multiscale fusion algorithm, and determining a fused wound image includes:

[0078] Based on the pre-trained local feature descriptor network, the local stable feature points of multimodal images are detected and extracted. One modality image is used as the target image to determine the target feature point set, and the other modality image is used as the source image to determine the source feature point set. Through the attention mechanism, the candidate matching points that are most similar to each local stable feature point in the source feature point set are searched in the target feature point set. The Ransac algorithm is used to eliminate false matches and establish the sparse feature correspondence relationship between the local stable feature points of different modality images.

[0079] Based on the sparse feature correspondence, the dense spatial mapping from the source image to the target image is determined through affine transformation, perspective transformation and thin plate spline interpolation. The mapping training is performed by combining the cycle consistency loss function and the gradient correlation loss function to obtain the multimodal image registration result.

[0080] Each modality image in the multimodal image registration result is subjected to wavelet transform respectively, and a wavelet coefficient pyramid is obtained by layer-by-layer decomposition. The fusion weight is calculated based on the variance characteristics of the wavelet coefficients, and the wavelet coefficients are fused based on the fusion weight. The fused wound image is obtained by inverse wavelet transform.

[0081] Select a pre-trained local feature descriptor network, preferably SIFT, and input the multimodal image into the selected feature descriptor network. The network extracts local stable feature points of the image through a series of convolutions, pooling and nonlinear transformations. For each feature point, the network generates a high-dimensional feature descriptor vector to represent the local image feature of the point.

[0082] An image of one modality is selected as the target image and the other as the source image. In the target image, a set of local stable feature points is detected and extracted by the feature descriptor network to form a target feature point set. In the source image, another set of local stable feature points is detected and extracted by the feature descriptor network to form a source feature point set.

[0083] For each feature point in the source feature point set, search for the most similar candidate matching point in the target feature point set, use the attention mechanism to measure the similarity between feature points, calculate the attention weights of each point in the source feature point and target feature point set, and select the target feature point with the highest attention weight as the candidate matching point for the current source feature point.

[0084] Use the Ransac algorithm to perform robustness estimation on candidate matching point pairs, randomly sample a set of minimum number of matching point pairs, calculate the transformation model between them, such as affine transformation or homography transformation, and classify all candidate matching point pairs according to the transformation model. Mark the point pairs that meet the model as inliers and those that do not as outliers. Repeat the random sampling and classification process multiple times, and select the transformation model with the largest number of inliers as the optimal model. According to the optimal model, eliminate the erroneous matches marked as outliers to obtain the refined feature point correspondence.

[0085] The refined feature point correspondence is used as the sparse feature correspondence between images of different modalities. The coordinate position and correspondence of each pair of matching feature points are recorded. Based on the sparse feature correspondence, the dense spatial mapping from the source image to the target image is estimated. Each pixel in the source image is mapped to the corresponding position in the target image using affine transformation, perspective transformation or thin plate spline interpolation methods. The parameters of the mapping function are optimized by minimizing the mapping error and regularization term.

[0086] The cycle consistency loss function is used to encourage the consistency of forward mapping and reverse mapping. The gradient correlation loss function is used to promote the correlation between the gradient of the mapped image and the gradient of the target image. The parameters of the mapping function are updated through the back-propagation algorithm to minimize the overall loss function. The optimization process is iterated until the preset number of iterations or convergence conditions are reached.

[0087] Using the optimal mapping function obtained through training, each pixel in the source image is mapped to the corresponding position in the target image to generate a registered source image so that it is spatially aligned with the target image.

[0088] Wavelet transform is performed on each modality image after registration, and the image is decomposed layer by layer using wavelet basis functions to obtain a wavelet coefficient pyramid. At each decomposition level, wavelet coefficients representing different frequency and directional characteristics of the image are obtained.

[0089] The variance characteristics of the wavelet coefficients of each modality image are analyzed, and the fusion weights of different modality images at each decomposition level are calculated according to the variance of the wavelet coefficients. Wavelet coefficients with larger variances are assigned higher weights, indicating that they contain more significant feature information.

[0090] According to the calculated fusion weights, the wavelet coefficients of images of different modalities are weightedly fused. At each decomposition level, the wavelet coefficients at the corresponding position are multiplied by their corresponding fusion weights and then added together to obtain a fused wavelet coefficient pyramid, which represents the characteristics of the fused image in different frequencies and directions.

[0091] The fused wavelet coefficient pyramid is subjected to inverse wavelet transform, and the wavelet coefficients are gradually merged through layer-by-layer reconstruction to finally obtain a fused wound image. The fused image integrates the features and detail information of images of different modalities, providing a more comprehensive and informative wound representation.

[0092] In this embodiment, stable local feature points are extracted to improve the accuracy of feature matching between images of different modalities; the Ransac algorithm is used to eliminate erroneous matches, optimize the correspondence between feature points, and improve the robustness and reliability of the registration results; the registered image is subjected to wavelet transform to decompose the multi-scale information of the image, and the fusion weight is calculated based on the variance characteristics of the wavelet coefficients, so that coefficients with larger variance account for a larger proportion in the fusion, thereby enhancing the expressiveness of the fused image; a pre-trained local feature descriptor network is used to automatically extract image features, reduce manual intervention, and improve processing efficiency; the pre-trained feature descriptor network can be used to achieve good feature extraction and matching on limited data, thereby improving the generalization ability of the model; and the transfer learning method can effectively utilize the advantages of the pre-trained model to improve the accuracy of wound image classification and segmentation.

[0093] In an optional embodiment, the fused wound image is segmented into wound regions by a convolutional neural network-based method to obtain a segmented wound region image, and the segmented wound region image is classified into wound tissue types by a transfer learning-based method to obtain a wound tissue type classification result including:

[0094] Taking the fused wound image as input, an encoder-decoder architecture is adopted to construct a neural network for wound area segmentation, wherein the encoder is constructed based on EfficientNet-B7, extracts shallow features, and generates a multi-scale semantic feature map through dilated convolution and pyramid pooling. The decoder fuses the shallow features in the encoder through progressive upsampling and skip connections, gradually restores the spatial resolution of the multi-scale semantic feature map, and sets a resolution restoration module at the end of the decoder. Through multi-scale convolution and attention mechanism, the multi-scale semantic feature map with restored spatial resolution is granularly repaired to generate a wound segmentation probability map.

[0095] Based on binary cross entropy loss, Dice similarity loss and contrast loss, a loss function of the wound area segmentation neural network is constructed, and through iterative training, the distance between the wound area and the background area in the feature space is maximized to obtain an optimal wound segmentation probability map and determine the wound area segmentation result;

[0096] Based on the wound area segmentation results, ResNeXt-101 pre-trained on ImageNet is used as a feature extractor, and the distribution of feature space is adjusted through adversarial autoencoding. The adversarial game between the generator and the discriminator network is used to learn the distribution of real wound tissue features and determine the low-dimensional feature representation of the wound;

[0097] Sampling multiple local blocks in the wound area, extracting features from multiple scales and multiple directions, combining with the low-dimensional feature representation of the wound to obtain a multi-view representation of the wound tissue, and fusing the multi-view representation through a multi-head attention pooling mechanism to obtain globally consistent tissue type features;

[0098] The tissue type discrimination model is constructed based on metric learning. The structure of the feature embedding space is iteratively adjusted, and the pairwise loss and triplet loss are combined to increase the intra-class consistency and inter-class difference, and the tissue type is divided. The nearest neighbor classification is determined according to the center similarity between the sample to be classified and each tissue type.

[0099] A tissue type discrimination model was constructed based on metric learning. By optimizing the structure of the feature embedding space, samples of the same category were clustered and samples of different categories were kept apart. The intra-class consistency and inter-class difference were enhanced through pairwise loss and triplet loss. The nearest neighbor classification was performed according to the similarity between the samples to be classified and the centers of each category to obtain the wound tissue type classification results.

[0100] Taking the fused wound image as input, a wound area segmentation neural network based on the encoder-decoder architecture is constructed. The encoder part adopts the pre-trained EfficientNet-B7 network to extract the shallow features of the input image through a series of convolution and downsampling operations. In the feature extraction process of the encoder, the dilated convolution and pyramid pooling techniques are introduced to generate a feature map with multi-scale semantic information. The decoder part fuses the shallow features in the encoder with the features of the decoder through progressive upsampling and skip connections, and gradually restores the spatial resolution of the feature map. At the end of the decoder, a resolution restoration module is set. Through multi-scale convolution and attention mechanism, the multi-scale semantic feature map with restored spatial resolution is fine-grainedly repaired. Through the resolution restoration module, a pixel-level wound segmentation probability map is generated, indicating the probability of each pixel belonging to the wound area.

[0101] The loss function of the wound area segmentation neural network was constructed, including binary cross entropy loss, Dice similarity loss and contrast loss. The binary cross entropy loss measures the pixel-level difference between the segmentation result and the true label, encouraging the network to correctly predict the category of each pixel. The Dice similarity loss calculates the overlap between the segmentation result and the true label, promoting the network to generate segmentation results that are highly consistent with the true wound area. The contrast loss maximizes the distance between the wound area and the background area in the feature space, thereby enhancing the network's ability to distinguish the wound area. Through the iterative training process, the network parameters are continuously adjusted to minimize the overall loss function and obtain the optimal wound segmentation probability map.

[0102] Based on the wound area segmentation results, the pre-trained ResNeXt-101 network is used as a feature extractor to extract the high-dimensional features of the wound area. The adversarial autoencoding technology is used to adjust the distribution of the feature space, and the low-dimensional representation of the real wound tissue features is learned. The generator and discriminator network are introduced. Through the adversarial game process, the generator tries to generate features similar to the real wound features, and the discriminator tries to distinguish between the generated features and the real features. Through the continuous game between the generator and the discriminator, the generator learns to capture the essence of the real wound tissue features and obtains a compact and expressive low-dimensional feature representation.

[0103] Multiple local blocks are sampled in the wound area, local features are extracted from different scales and directions, and the features of the local blocks are combined with the low-dimensional feature representation of the wound to obtain a multi-view representation of the wound tissue, capturing tissue features at different granularities and angles. A multi-head attention pooling mechanism is used to calculate the correlation and importance between features of different views through the self-attention mechanism. According to the attention weights, the multi-view features are weightedly fused to obtain a globally consistent and information-rich tissue type feature representation.

[0104] A tissue type discrimination model is constructed based on the metric learning principle. A feature embedding space is learned so that samples of the same category are clustered in the embedding space, and samples of different categories are far away from each other in the embedding space. The structure of the feature embedding space is optimized through pairwise loss and triplet loss to enhance intra-class consistency and inter-class difference. Pairwise loss encourages samples of the same category to be close in the embedding space, and samples of different categories to be far away in the embedding space. Triplet loss minimizes the distance between anchor samples and positive samples and maximizes the distance between anchor samples and negative samples by selecting anchor samples, positive samples and negative samples. Through the iterative optimization process, the feature embedding space is continuously adjusted so that samples of different categories are well separated and clustered in the embedding space. For the wound tissue samples to be classified, they are mapped to the learned feature embedding space, and the similarity with the center of each tissue type is calculated. According to the similarity score, the tissue type most similar to the sample to be classified is selected as the final classification result.

[0105] In this embodiment, the pre-trained EfficientNet-B7 is used as the encoder, which can utilize its rich features learned on large-scale data sets, introduce dilated convolution and pyramid pooling technology in the encoder, and can capture multi-scale semantic information and enhance the expressiveness of feature maps. The decoder part improves the accuracy of wound area segmentation through skip connections and progressive upsampling; a resolution restoration module is set at the end of the decoder to perform fine-grained repair and generate a more accurate segmentation probability map; by introducing multi-scale semantic information and fine-grained repair mechanism, the segmentation ability of the model in complex scenes is enhanced, and the accuracy of segmentation results is improved. Detail restoration and accuracy; construct a comprehensive loss function including binary cross entropy loss, Dice similarity loss and contrast loss to measure the quality of segmentation results from multiple angles and improve the performance of the segmentation model; introduce the adversarial game mechanism between the generator and the discriminator to capture the essence of real wound tissue characteristics and improve the quality and generalization ability of feature representation; calculate the correlation and importance between features of different views through the multi-head attention pooling mechanism, perform weighted fusion, and improve the global consistency and information richness of feature representation; construct a tissue type discrimination model based on metric learning to enhance intra-class consistency and inter-class differences and improve classification accuracy.

[0106] S102. Perform surround shooting of the wound area to obtain a multi-view image of the wound, based on the multi-view image of the wound, take the first frame depth map corresponding to the first frame image as a reference, align the subsequent depth maps frame by frame and accumulate point cloud data to determine the three-dimensional point cloud of the wound, generate a three-dimensional reconstruction result of the wound by smoothing the point cloud and combining it with a surface evolution algorithm, and determine the wound parameter measurement result based on the three-dimensional reconstruction result of the wound;

[0107] Use a camera or other imaging device to capture surround images of the wound area, obtain multi-view images of the wound from different angles, and form a set of continuous wound image sequences.

[0108] The depth map corresponding to the first frame is selected as the reference, and the depth maps of subsequent frames are registered with the reference depth map. The registration process aligns the depth maps of subsequent frames with the reference depth map by estimating the geometric transformation relationship between adjacent frames. The registered depth map is converted into 3D point cloud data and accumulated into the wound point cloud model.

[0109] The accumulated wound point cloud is smoothed to remove noise and outliers to improve the quality of the point cloud. A surface evolution algorithm, such as the moving least squares method or the Poisson surface reconstruction algorithm, is used to convert the point cloud data into a continuous and smooth three-dimensional surface model. Through an iterative optimization process, a high-quality three-dimensional reconstruction result of the wound is generated.

[0110] Based on the wound 3D reconstruction results, various wound parameters are extracted and calculated, such as wound area, volume, depth, length, etc. The morphological parameters of the wound can be obtained by analyzing the geometric features of the 3D surface model, such as curvature, normal vector, etc.

[0111] In this embodiment, high-precision three-dimensional wound reconstruction and accurate wound parameter measurement results are provided, which significantly improves the scientificity and effectiveness of wound diagnosis and treatment. Through fine point cloud processing and surface reconstruction, noise and errors are reduced, the continuity and smoothness of the reconstruction results are ensured, and comprehensive and detailed wound information is provided to help medical personnel make more accurate diagnosis and treatment decisions.

[0112] In an optional embodiment, a wound area is photographed in a surround manner to obtain a multi-view image of the wound. Based on the multi-view image of the wound, a first frame depth map corresponding to the first frame image is used as a reference, subsequent depth maps are registered frame by frame and point cloud data are accumulated to determine a three-dimensional point cloud of the wound. The three-dimensional reconstruction result of the wound is generated by smoothing the point cloud and combining the surface evolution algorithm, including:

[0113] A regular dodecahedron is arranged, and a high-resolution main camera with a resolution greater than a preset resolution threshold and multiple low-resolution auxiliary cameras with a resolution less than a preset resolution threshold are placed on each face of the regular dodecahedron to form a multi-camera array. The wound area is surrounded by the multi-camera array to obtain a compound eye multi-view image.

[0114] Based on the multi-scale stereo matching algorithm, a matching scale is selected through a multi-scale phase pyramid. At each matching scale, a disparity map is obtained through local phase consistency constraints and global energy optimization. The disparity map is converted into a corresponding basic depth map in combination with the camera focal length and the camera baseline distance. A Markov random field model of the basic depth map is constructed, an energy loss function is determined, and the minimum value of the energy loss function is iteratively solved to obtain a depth estimation result, determine the optimal depth map, and form a multi-scale depth map.

[0115] Based on the Harris-Laplace operator, the local extreme points in the compound-eye multi-view image are detected in the scale space to determine the feature points. Based on the scale and position of the feature points, the descriptors corresponding to the feature points are calculated to generate feature vectors. Based on the feature vectors corresponding to two feature points, the feature similarity is calculated. A weighted graph is constructed with the feature points as nodes and the feature similarity as the weights of the edges between the nodes. A graph-based ant colony algorithm is used to perform a search process on the weighted graph by ants, and iterative exploration is performed to find the global optimal feature match and determine the feature matching result.

[0116] Based on the multi-scale depth map and the feature matching result, multi-scale point cloud registration is performed, by minimizing the distance error between points in the point cloud, adjusting the rigid body transformation matrix, combining the local deformation non-rigid constraint, and estimating the affine transformation matrix to compensate for the non-rigid deformation. At the same time, the weight coefficient is adjusted between different scales to balance the rigid deformation and the non-rigid deformation. Based on the preset bone motion model, the wound surface is divided into multiple local rigid body areas, and the point cloud motion deformation state is determined by constructing the bone structure and calculating the joint parameters to obtain the point cloud registration result;

[0117] Based on the point cloud registration result, an implicit surface evolution method is adopted. The point cloud is embedded in three-dimensional space by defining an initial level set function. The surface is approximated to the point cloud by minimizing the energy functional. Combined with the curvature regularization term, the smoothness and continuity of the surface are controlled. The level set function is updated by numerical solution, and the zero level set is extracted as the current surface. The shape prior based on tensor integral is introduced. The local tangent space is constructed by obtaining the properties and structure of the wound surface, and the tensor integral is defined. The optimal evolution direction of the surface is determined by minimizing the tensor integral, and finally the three-dimensional reconstruction result of the wound is obtained.

[0118] A regular dodecahedron is arranged, and a camera is placed on each face of the regular dodecahedron. A high-resolution main camera and multiple low-resolution auxiliary cameras are placed on each face to form a multi-camera array. The resolution of the high-resolution main camera is greater than a preset resolution threshold, and the resolution of the low-resolution auxiliary cameras is less than the preset resolution threshold. The wound area is surrounded by the multi-camera array to obtain wound images from different angles. The images obtained form a compound-eye multi-perspective image to provide all-round information about the wound.

[0119] A multi-scale stereo matching algorithm is adopted. By constructing a multi-scale phase pyramid, the appropriate matching scale is selected. At each matching scale, the disparity map is calculated by using local phase consistency constraints and global energy optimization. The disparity map is converted into the corresponding basic depth map by combining the focal length and baseline distance information of the camera. A Markov random field model of the basic depth map is constructed, and an energy loss function is defined. The minimum value of the energy loss function is solved through iterative optimization to obtain the depth estimation result. The depth estimation result with the minimum energy loss is selected as the optimal depth map. The optimal depth maps at different scales are combined to form a multi-scale depth map.

[0120] The Harris-Laplace operator is used to detect local extreme points of compound-eye multi-view images in scale space, determine feature points, calculate the descriptor corresponding to each feature point according to the scale and position information of the feature points, generate feature vectors, and calculate the similarity between the feature vectors corresponding to two feature points. This is used as the basis for feature matching, and a weighted graph is constructed. The feature points are used as nodes, and the feature similarity is used as the weight of the edges between nodes. The graph-based ant colony algorithm is used to iteratively explore the optimal feature matching by simulating the search process of ants on the weighted graph. The globally optimal feature matching result is found through the ant pheromone update and path selection.

[0121] Multi-scale point cloud registration is performed using multi-scale depth maps and feature matching results. The rigid body transformation matrix is ​​estimated and adjusted by minimizing the distance error between points in the point cloud to achieve rigid registration of the point cloud. Local deformation non-rigid constraints are introduced to compensate for the non-rigid deformation of the point cloud by estimating the affine transformation matrix. The weight coefficient is adjusted between different scales to balance the effects of rigid and non-rigid deformations. The wound surface is divided into multiple local rigid areas using the preset skeletal motion model. The motion deformation state of the point cloud is determined by constructing the skeletal structure and calculating the joint parameters. The rigid deformation, non-rigid deformation and skeletal motion are comprehensively considered to obtain the final point cloud registration result.

[0122] The implicit surface evolution method is used to reconstruct the point cloud registration results in three-dimensional form. The initial level set function is defined, and the point cloud is embedded in three-dimensional space. The surface is approximated to the point cloud data by minimizing the energy functional. The curvature regularization term is introduced to control the smoothness and continuity of the surface. The level set function is iteratively updated by a numerical solution method, and the zero level set is extracted as the current surface. The shape prior based on tensor integral is introduced to obtain the properties and structural information of the wound surface, and the local tangent space is constructed. The tensor integral is defined, and the optimal evolution direction of the surface is determined by minimizing the tensor integral. After multiple iterative evolutions, the final three-dimensional reconstruction result of the wound is obtained.

[0123] In this embodiment, by forming a multi-camera array, wound images can be acquired from different angles to obtain compound-eye multi-view images, thereby providing all-round information of the wound; the disparity map is accurately calculated and converted into a depth map, and a Markov random field model of the basic depth map is constructed. The optimal depth map is obtained through energy loss function optimization, and finally a multi-scale depth map is formed to achieve high-precision depth estimation; the ant colony algorithm is used to iteratively explore the weighted graph to find the globally optimal feature matching result to ensure the accuracy of feature matching; the implicit surface evolution method is used to perform three-dimensional reconstruction of the point cloud registration results, and the level set function is iteratively updated through curvature regularization and shape prior based on tensor integral, and finally a smooth, continuous and accurate three-dimensional reconstruction result of the wound is obtained.

[0124] In an optional embodiment, the energy loss function is as follows:

[0125]

[0126] Among them, E represents the energy loss function, p represents the pixel index, q represents another pixel index, and N p represents the neighborhood of pixel p, w pq represents the similarity weight between pixel p and pixel q, ρ(·) represents the robust function for handling outliers, and I p Represents the image intensity at pixel p, I q represents the image intensity at pixel q, d p Indicates the depth value of pixel p, d q represents the depth value of pixel q, λ represents the balance coefficient, and α represents the truncation threshold.

[0127] The formula consists of two parts. The first part calculates the difference in image intensity between a pixel and its neighbors. The difference is weighted by similarity weights and processed with robust functions to reduce the impact of outliers on the results. The second part calculates the difference in depth value between a pixel and its neighbors. The difference is processed by similarity weights and truncation thresholds so that the energy function is not affected too much when the depth value difference is too large. The balance coefficient is used to adjust the weight of this part in the total energy.

[0128] According to the formula, by using robust functions and truncation thresholds, the negative impact of outliers and excessive depth differences on energy optimization is reduced, making the algorithm more stable and robust when dealing with noise and outliers; by introducing weights of image intensity differences and depth value differences, the energy loss function can more accurately reflect the true situation of the image, thereby improving the accuracy of the depth estimation results; the balance coefficient enables the energy function to reasonably weigh image intensity and depth information during the optimization process, avoiding bias towards one side, and ensuring the rationality and accuracy of the final depth map; by flexibly adjusting parameters and, the energy loss function can adapt to different image scenarios and application requirements, thereby improving the applicability and breadth of the algorithm.

[0129] S103. Based on the wound tissue type classification results and wound parameter measurement results, a robot path planning algorithm is used to generate the optimal robot motion trajectory by constructing a reward goal for wound treatment and combining the penalty constraints of robot motion. The robot control algorithm based on vision and force feedback is used to adjust the robot's motion parameters in real time to complete wound-assisted treatment.

[0130] According to the type and parameters of the wound, the reward goals of the treatment are determined. These goals may include different treatment operations such as debridement, disinfection, and application of medicine; the robot path planning algorithm is applied, considering the treatment goals and the penalty constraints of the robot movement (such as avoiding collisions, saving energy, etc.), to generate the optimal motion trajectory of the robot. The optimal motion trajectory of the robot should be able to cover the entire area of ​​the wound and meet the predetermined treatment requirements. The robot control algorithm is implemented, combined with the visual and force feedback mechanism, and the position and operation of the robot are monitored in real time. According to the feedback information, the robot's motion parameters are dynamically adjusted to ensure that its motion trajectory is accurate and fine-tuned as needed. The robot performs auxiliary treatment operations according to the planned trajectory to ensure comprehensive and accurate treatment of the wound. During the treatment process, visual and force feedback are continuously used to correct any deviations in a timely manner to achieve the best treatment effect.

[0131] In this embodiment, by constructing a reward goal, the system can give priority to the path that is most beneficial to wound treatment and improve the treatment effect; combine the penalty constraints of the robot's movement to avoid the robot causing secondary damage to the wound during the treatment process and ensure safety; monitor the wound treatment process in real time through visual feedback, promptly discover and correct deviations, and ensure the accuracy of the treatment process; use the force feedback system to sense the pressure applied by the robot to prevent excessive pressure from affecting the wound and provide gentler treatment; the automated robot control algorithm can autonomously perform complex treatment tasks and reduce errors and instability in human operations; adjust the robot's motion parameters in real time to make the treatment process more flexible and efficient and shorten the treatment time.

[0132] In an optional embodiment, based on the wound tissue type classification results and the wound parameter measurement results, a robot path planning algorithm is used to construct a reward target for wound treatment and combine the penalty constraints of the robot motion to generate the optimal robot motion trajectory, including:

[0133] A mapping relationship between wound status and healing stage is established based on a machine learning algorithm to construct a wound status assessment model, wherein the wound status includes wound tissue type distribution, area and color, and the healing stage includes inflammation stage, proliferation stage and remodeling stage;

[0134] Based on the output results of the wound status assessment model, the weight of the reward target is adjusted in real time. When the wound status changes, the corresponding weight coefficient in the reward target is updated to determine the dynamic reward weight, including: when the healing stage is in the inflammatory stage, the weight coefficient corresponding to tissue clearance is increased, and the weight coefficient corresponding to tissue protection is reduced; when the healing stage is in the proliferative stage, the weight coefficient corresponding to tissue protection is increased, and the weight coefficient corresponding to tissue migration is reduced;

[0135] Based on the wound treatment trajectory data collected in advance through expert teaching, the corresponding wound state features and treatment motion features are extracted to construct state-action pairs;

[0136] Based on the maximum entropy inverse reinforcement learning algorithm, the parameters of the reward function are determined by maximizing the likelihood probability of the wound treatment trajectory data to obtain the initial reward function. The difference between the wound treatment trajectory data and the generated trajectory obtained by the maximum entropy inverse reinforcement learning algorithm is minimized through the gradient descent algorithm, and the parameters of the reward function are adjusted to obtain the implicit reward function.

[0137] Taking different reward targets as independent objective functions, combining dynamic reward weights and implicit reward functions, taking robot motion constraints as constraint conditions, building a multi-objective optimization model, and using a multi-objective evolutionary algorithm to optimize and solve based on the multi-objective optimization model to obtain an optimal solution set, each optimal solution in the Pareto optimal solution set corresponds to a robot motion trajectory;

[0138] In combination with current treatment needs and doctor's wishes, a robot motion trajectory is selected from the Pareto optimal solution set to determine the optimal robot motion trajectory.

[0139] Collect wound images and corresponding wound status information, including wound tissue type distribution, area and color. Collect annotation information of wound healing stages, including inflammatory stage, proliferation stage and remodeling stage. Extract features of wound images, such as texture, shape, color, etc., as input of the model. Use machine learning algorithms, such as support vector machine and random forest, to establish a mapping relationship between wound status and healing stage. Optimize model parameters through training data set, and evaluate model performance on validation set to obtain a trained wound status assessment model for real-time assessment of wound status and healing stage.

[0140] According to the output results of the wound status assessment model, the healing stage of the wound is judged in real time. When the wound status changes, the corresponding weight coefficient in the reward target is dynamically adjusted. When the healing stage is in the inflammatory stage, the weight coefficient corresponding to tissue clearance is increased, and the weight coefficient corresponding to tissue protection is reduced. When the healing stage is in the proliferative stage, the weight coefficient corresponding to tissue protection is increased, and the weight coefficient corresponding to tissue migration is reduced. According to the adjusted weight coefficient, the dynamic reward weight is determined for subsequent motion trajectory optimization.

[0141] Through expert teaching, trajectory data during wound treatment is collected in advance, and the collected trajectory data is preprocessed to extract corresponding wound state features and treatment motion features. The extracted wound state features and treatment motion features are paired to construct state-action pairs. The state-action pairs represent the treatment motion operations performed by experts under specific wound states. The constructed state-action pairs are used as the input of the inverse reinforcement learning algorithm to learn the implicit reward function.

[0142] The maximum entropy inverse reinforcement learning algorithm is used to maximize the likelihood probability of wound treatment trajectory data to determine the initial parameters of the reward function. The gradient descent algorithm is used to minimize the difference between the wound treatment trajectory data and the trajectory generated by the maximum entropy inverse reinforcement learning algorithm. Through iterative optimization, the parameters of the reward function are adjusted to make the generated trajectory as similar as possible to the trajectory taught by the expert. The optimized implicit reward function is obtained to evaluate the quality of the robot's motion trajectory.

[0143] Different reward goals, such as tissue clearance, tissue protection, and tissue migration, are taken as independent objective functions. A multi-objective optimization model is constructed by combining dynamic reward weights and implicit reward functions. The constraints of robot motion, such as joint angle limits and speed limits, are taken as constraints of the optimization model. The objective function and constraints together constitute a multi-objective optimization problem, which is used to solve the optimal robot motion trajectory.

[0144] A multi-objective evolutionary algorithm is used to solve the constructed multi-objective optimization model. A set of candidate solutions is generated through population initialization. Each candidate solution represents a possible robot motion trajectory. The individuals in the population are evaluated, and the fitness value of each individual on different objective functions is calculated. The population is continuously iterated and optimized through operations such as selection, crossover, and mutation, so that the quality of the solution is continuously improved. After multiple iterations, the Pareto optimal solution set is obtained, where each optimal solution corresponds to a robot motion trajectory. The Pareto optimal solution set represents a set of optimal trajectories that weigh different objectives.

[0145] According to the current treatment needs and the doctor's wishes, a robot motion trajectory is selected from the Pareto optimal solution set. The doctor can choose the trajectory that best suits the current situation based on clinical experience and treatment goals, taking into account the importance of different goals. The selected robot motion trajectory is used as the optimal trajectory to guide the robot to perform wound treatment tasks.

[0146] In this embodiment, the wound status assessment model established by the machine learning algorithm can judge the wound healing stage in real time and provide an accurate basis for treatment; according to the output results of the assessment model, the weight coefficient of the treatment target is adjusted in real time to ensure that the most appropriate treatment strategy is adopted in different healing stages; the treatment trajectory data is collected through expert teaching, and the implicit reward function is learned by the inverse reinforcement learning algorithm, so that the robot can intelligently simulate the expert's treatment operation and improve the level of automation; the maximum entropy inverse reinforcement learning algorithm is used to obtain the implicit reward function through iterative optimization, so that the generated trajectory is as similar as possible to the expert's trajectory, ensuring the accuracy and effectiveness of the treatment; a multi-objective optimization model combining dynamic reward weights and implicit reward functions is constructed, and a comprehensive consideration is given to the treatment trajectory. Considering different goals such as tissue removal, tissue protection, and tissue migration, the optimal robot motion trajectory is generated; a multi-objective evolutionary algorithm is used to iteratively optimize through operations such as population initialization, selection, crossover, and mutation to obtain the Pareto optimal solution set, ensuring that the optimal trajectory is found in the trade-off between different goals; a set of optimal trajectory options that weigh different goals are provided to doctors. Doctors can choose the trajectory that best suits the current situation based on clinical experience and treatment goals to achieve personalized treatment; automated path planning and real-time feedback control reduce errors in human operations and improve the stability and consistency of treatment; through inverse reinforcement learning and expert teaching, the expert's treatment experience and knowledge are integrated into the robot system, reducing dependence on operator skills.

[0147] In an optional embodiment, the multi-objective evolutionary algorithm includes:

[0148] Based on preset historical data and prior knowledge, a knowledge-guided initialization strategy is adopted to determine the initial population;

[0149] Take the initial population as the current population, and for each individual in the current population, calculate the standard fitness value under each independent objective function corresponding to different reward targets;

[0150] The grid-based adaptive crowding measurement mechanism dynamically adjusts the grid size according to the distribution characteristics of the current population in the target space, and calculates the individual crowding by combining the grid density and the Euclidean distance between individuals.

[0151] Based on the standard fitness value, combined with the dominance relationship of each individual in the current population and the individual crowding, the standard fitness value is dynamically adjusted, and the individual fitness value is determined through weight aggregation. Based on the individual fitness value, the individuals in the current population are sorted to determine the optimal individual, and the parent individual is determined according to the preset preferred number;

[0152] Based on simulating binary crossover and polynomial mutation, a crossover mutation operator is determined, and the crossover mutation operator parameters are adjusted based on the number of iterations. The crossover mutation operator is used to perform a crossover mutation operation on the parent individuals to generate offspring individuals. Based on the parent individuals, random individuals are determined according to a preset generation ratio through uniform sampling and Gaussian perturbation. The optimal individuals, offspring individuals and random individuals are merged to generate an extended population.

[0153] Repeat the iteration until the preset number of iterations is reached to obtain the optimal extended population. Based on the individual fitness value of each individual in the optimal extended population, the optimal solution set is obtained according to the preset preferred number.

[0154] Collect and organize historical data and prior knowledge related to the optimization problem. Based on historical data, extract valuable information, such as the characteristics of high-quality solutions, the value range of important variables, etc. Use prior knowledge to set the generation rules of the initial solution, such as the constraint relationship between variables, heuristic strategies, etc. Combine historical data and prior knowledge to generate a set of high-quality initial solutions as the initial population. The individuals in the initial population should be diverse, covering different areas of the solution space, and provide a good starting point for subsequent evolution.

[0155] Take the initial population as the current population, evaluate each individual in the population, and calculate the fitness value of each individual under each independent objective function according to the different reward objectives of the optimization problem. The fitness value measures the performance of the individual on a single objective. The higher the fitness value, the better the individual performs on the objective. For each objective function, calculate the fitness values ​​of all individuals in the population to obtain the standard fitness value.

[0156] Analyze the distribution characteristics of the current population in the target space, such as the density and clustering of individuals. Based on the population distribution characteristics, dynamically adjust the grid size of the target space so that the grid can adapt to the distribution of the population. Divide the target space into multiple grids, calculate the number of individuals in each grid, and obtain the grid density. For each individual, calculate the Euclidean distance between it and other individuals. Combined with the grid density, calculate the crowding of the individual. The crowding reflects the degree of crowding around the individual. The higher the crowding, the more individuals there are in the area where the individual is located.

[0157] Based on the standard fitness value, the fitness value is adjusted considering the dominance relationship between individuals. If individual A dominates individual B, the fitness value of individual A should be better than that of individual B. Combined with the crowding degree of the individuals, the adjusted fitness value is further corrected to maintain the diversity of the population. The adjusted fitness values ​​of multiple targets are weighted and aggregated to obtain the comprehensive fitness value of the individual. The individuals in the current population are sorted according to the comprehensive fitness value. The higher the fitness value, the higher the ranking. A certain number of individuals with the highest ranking are selected as parents for subsequent crossover and mutation operations.

[0158] Simulated binary crossover and polynomial mutation are used as crossover mutation operators. According to the current number of iterations, the parameters of the crossover mutation operator, such as crossover probability and mutation probability, are dynamically adjusted. The parent individuals are crossovered in a certain pairing manner to generate new offspring individuals. The parent individuals are mutated according to a certain mutation probability to introduce new genetic information. The crossover mutation operation aims to explore new solutions and inherit the excellent characteristics of the parent individuals.

[0159] In addition to generating offspring individuals through crossover mutation, a certain proportion of random individuals also need to be introduced. Random individuals are generated through uniform sampling and Gaussian perturbation to maintain the diversity of the population. Uniform sampling randomly generates individuals in the solution space. Gaussian perturbation adds random noise based on the parent individuals. According to the preset generation ratio, the number of random individuals is determined, and the optimal individuals, offspring individuals and random individuals are merged to form an extended population as the next generation population.

[0160] Repeat the iterative population and iteratively optimize the extended population until the preset number of iterations is reached. In each iteration, the fitness values ​​of the individuals in the extended population are evaluated, sorted and selected. After multiple iterations, the overall quality of the population continues to improve and gradually converges to the optimal solution. When the preset number of iterations is reached, the optimal extended population is obtained. According to the comprehensive fitness values ​​of the individuals in the optimal extended population, the best performing individuals are selected according to the preset preferred number. The selected individuals constitute the optimal solution set, which is the Pareto optimal solution for the multi-objective optimization problem.

[0161] In this embodiment, historical data and prior knowledge are used to generate a high-quality initial population, thereby improving the starting point quality of the optimization process; the initial population is diverse and covers different areas of the solution space, providing a good foundation for subsequent evolution; the grid size of the target space is dynamically adjusted according to the population distribution characteristics to improve the accuracy of fitness calculation; by calculating the crowding degree of individuals, it is avoided that individuals in the population are too concentrated in a certain area, thereby maintaining the diversity of the population; the fitness values ​​of multiple objectives are comprehensively considered, and the comprehensive fitness value of the individual is obtained by weighted aggregation, thereby improving the effect of multi-objective optimization; after iterative optimization, a Pareto optimal solution set is finally obtained, providing multiple optimal solutions that balance different objectives; these crossover mutation operators are used to dynamically adjust the parameters. number, improve the exploration ability and diversity of the solution; generate random individuals through uniform sampling and Gaussian perturbation to further maintain the diversity of the population and avoid local optimal traps; dynamically adjust the parameters of crossover mutation according to the number of iterations to adapt to different stages of the optimization process and improve the optimization efficiency; calculate the standard fitness value according to different objective functions to ensure accurate performance evaluation of each individual; consider the dominance relationship between individuals and adjust the fitness value to ensure the dominant position of high-quality individuals in the population; through comprehensive fitness value sorting and selection, ensure that the selected parent individuals have high quality and improve the excellent characteristics of subsequent offspring individuals; the final Pareto optimal solution set provides a variety of optimal balance schemes between different objectives, providing more choices for decision-making.

[0162] Figure 2 FIG. 1 is a schematic diagram of the structure of a robot-assisted wound treatment system based on multimodal image analysis according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0163] The first unit is used to collect multimodal images of a wound to be treated, perform geometric transformation and spatial alignment on the multimodal images, determine an image registration result, reconstruct the image registration result by a multi-scale fusion algorithm, determine a fused wound image, perform wound region segmentation on the fused wound image by a method based on a convolutional neural network to obtain a segmented wound region image, and perform wound tissue type classification on the segmented wound region image by a method based on transfer learning to obtain a wound tissue type classification result;

[0164] The second unit is used to perform surround shooting of the wound area to obtain a multi-view image of the wound, based on the multi-view image of the wound, taking the first frame depth map corresponding to the first frame image as a reference, aligning subsequent depth maps frame by frame and accumulating point cloud data, determining a three-dimensional point cloud of the wound, generating a three-dimensional reconstruction result of the wound through point cloud smoothing combined with a surface evolution algorithm, and determining a measurement result of wound parameters based on the three-dimensional reconstruction result of the wound;

[0165] The third unit is used to generate the optimal motion trajectory of the robot by using a robot path planning algorithm based on the wound tissue type classification results and wound parameter measurement results, by constructing a reward goal for wound treatment and combining it with the penalty constraints of the robot motion. The robot control algorithm based on vision and force feedback can adjust the robot's motion parameters in real time to complete wound-assisted treatment.

[0166] According to a third aspect of the embodiments of the present invention,

[0167] An electronic device is provided, comprising:

[0168] processor;

[0169] a memory for storing processor-executable instructions;

[0170] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0171] A fourth aspect of the embodiments of the present invention is:

[0172] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0173] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot-assisted wound treatment method based on multimodal image analysis, characterized in that: include: Acquire multimodal images of the wound to be treated, perform geometric transformation and spatial alignment on the multimodal images, determine image registration results, reconstruct the image registration results by a multi-scale fusion algorithm, determine a fused wound image, perform wound region segmentation on the fused wound image by a method based on a convolutional neural network to obtain a segmented wound region image, perform wound tissue type classification on the segmented wound region image by a method based on transfer learning to obtain a wound tissue type classification result; Surrounding shooting of the wound area is performed to obtain a multi-view image of the wound. Based on the multi-view image of the wound, the first frame depth map corresponding to the first frame image is used as a reference, and subsequent depth maps are aligned frame by frame and point cloud data are accumulated to determine the three-dimensional point cloud of the wound. Through point cloud smoothing and combined with a surface evolution algorithm, a three-dimensional reconstruction result of the wound is generated. Based on the three-dimensional reconstruction result of the wound, the measurement result of the wound parameters is determined; Based on the wound tissue type classification results and wound parameter measurement results, a robot path planning algorithm is used to generate the optimal robot motion trajectory by constructing a reward goal for wound treatment and combining it with the penalty constraints of robot motion. The robot control algorithm based on vision and force feedback is used to adjust the robot's motion parameters in real time to complete wound-assisted treatment.

2. The method according to claim 1, characterized in that Acquiring a multimodal image of a wound to be treated, performing geometric transformation and spatial alignment on the multimodal image, determining an image registration result, reconstructing the image registration result by a multiscale fusion algorithm, and determining a fused wound image includes: Based on the pre-trained local feature descriptor network, the local stable feature points of multimodal images are detected and extracted. One modality image is used as the target image to determine the target feature point set, and the other modality image is used as the source image to determine the source feature point set. Through the attention mechanism, the candidate matching points that are most similar to each local stable feature point in the source feature point set are searched in the target feature point set. The Ransac algorithm is used to eliminate false matches and establish the sparse feature correspondence relationship between the local stable feature points of different modality images. Based on the sparse feature correspondence, the dense spatial mapping from the source image to the target image is determined through affine transformation, perspective transformation and thin plate spline interpolation. The mapping training is performed by combining the cycle consistency loss function and the gradient correlation loss function to obtain the multimodal image registration result. Each modality image in the multimodal image registration result is subjected to wavelet transform respectively, and a wavelet coefficient pyramid is obtained by layer-by-layer decomposition. The fusion weight is calculated based on the variance characteristics of the wavelet coefficients, and the wavelet coefficients are fused based on the fusion weight. The fused wound image is obtained by inverse wavelet transform.

3. The method according to claim 1, characterized in that The fused wound image is segmented into wound regions by a convolutional neural network-based method to obtain a segmented wound region image, and the segmented wound region image is classified into wound tissue types by a transfer learning-based method to obtain wound tissue type classification results including: Taking the fused wound image as input, an encoder-decoder architecture is adopted to construct a neural network for wound area segmentation, wherein the encoder is constructed based on EfficientNet-B7, extracts shallow features, and generates a multi-scale semantic feature map through dilated convolution and pyramid pooling. The decoder fuses the shallow features in the encoder through progressive upsampling and skip connections, gradually restores the spatial resolution of the multi-scale semantic feature map, and sets a resolution restoration module at the end of the decoder. Through multi-scale convolution and attention mechanism, the multi-scale semantic feature map with restored spatial resolution is granularly repaired to generate a wound segmentation probability map. Based on binary cross entropy loss, Dice similarity loss and contrast loss, a loss function of the wound area segmentation neural network is constructed, and through iterative training, the distance between the wound area and the background area in the feature space is maximized to obtain an optimal wound segmentation probability map and determine the wound area segmentation result; Based on the wound area segmentation results, ResNeXt-101 pre-trained on ImageNet is used as a feature extractor, and the distribution of feature space is adjusted through adversarial autoencoding. The adversarial game between the generator and the discriminator network is used to learn the distribution of real wound tissue features and determine the low-dimensional feature representation of the wound; Sampling multiple local blocks in the wound area, extracting features from multiple scales and multiple directions, combining with the low-dimensional feature representation of the wound to obtain a multi-view representation of the wound tissue, and fusing the multi-view representation through a multi-head attention pooling mechanism to obtain globally consistent tissue type features; The tissue type discrimination model is constructed based on metric learning. The structure of the feature embedding space is iteratively adjusted, and the pairwise loss and triplet loss are combined to increase the intra-class consistency and inter-class difference, and the tissue type is divided. The nearest neighbor classification is determined according to the center similarity between the sample to be classified and each tissue type. A tissue type discrimination model was constructed based on metric learning. By optimizing the structure of the feature embedding space, samples of the same category were clustered and samples of different categories were kept apart. The intra-class consistency and inter-class difference were enhanced through pairwise loss and triplet loss. The nearest neighbor classification was performed according to the similarity between the samples to be classified and the centers of each category to obtain the wound tissue type classification results.

4. The method according to claim 1, characterized in that: Surround the wound area and obtain a multi-view image of the wound. Based on the multi-view image of the wound, the first frame depth map corresponding to the first frame image is used as a reference, and subsequent depth maps are registered frame by frame and point cloud data is accumulated to determine the three-dimensional point cloud of the wound. Through point cloud smoothing and combined with the surface evolution algorithm, the three-dimensional reconstruction results of the wound are generated, including: Setting an arrangement of a regular dodecahedron, placing a high-resolution main camera with a resolution greater than a preset resolution threshold and multiple low-resolution auxiliary cameras with a resolution less than the preset resolution threshold on each face of the regular dodecahedron to form a multi-camera array, and performing surround shooting of the wound area through the multi-camera array to obtain a compound eye multi-view image; Based on the multi-scale stereo matching algorithm, a matching scale is selected through a multi-scale phase pyramid. At each matching scale, a disparity map is obtained through local phase consistency constraints and global energy optimization. The disparity map is converted into a corresponding basic depth map in combination with the camera focal length and the camera baseline distance. A Markov random field model of the basic depth map is constructed, an energy loss function is determined, and the minimum value of the energy loss function is iteratively solved to obtain a depth estimation result, determine the optimal depth map, and form a multi-scale depth map. Based on the Harris-Laplace operator, local extreme points in the compound-eye multi-view image are detected in the scale space to determine the feature points. Based on the scale and position of the feature points, the descriptors corresponding to the feature points are calculated to generate feature vectors. Based on the feature vectors corresponding to two feature points, the feature similarity is calculated. The feature points are used as nodes, and the feature similarity is used as the weight of the edge between the nodes to construct a weighted graph. The graph-based ant colony algorithm is used to perform a search process on the weighted graph by ants, iterative exploration is performed, the global optimal feature match is found, and the feature matching result is determined; Based on the multi-scale depth map and the feature matching result, multi-scale point cloud registration is performed, by minimizing the distance error between points in the point cloud, adjusting the rigid body transformation matrix, combining the local deformation non-rigid constraint, and estimating the affine transformation matrix to compensate for the non-rigid deformation. At the same time, the weight coefficient is adjusted between different scales to balance the rigid deformation and the non-rigid deformation. Based on the preset bone motion model, the wound surface is divided into multiple local rigid body areas, and the point cloud motion deformation state is determined by constructing the bone structure and calculating the joint parameters to obtain the point cloud registration result; Based on the point cloud registration result, an implicit surface evolution method is adopted. The point cloud is embedded in three-dimensional space by defining an initial level set function. The surface is approximated to the point cloud by minimizing the energy functional. Combined with the curvature regularization term, the smoothness and continuity of the surface are controlled. The level set function is updated by numerical solution, and the zero level set is extracted as the current surface. The shape prior based on tensor integral is introduced. The local tangent space is constructed by obtaining the properties and structure of the wound surface, and the tensor integral is defined. The optimal evolution direction of the surface is determined by minimizing the tensor integral, and finally the three-dimensional reconstruction result of the wound is obtained.

5. The method according to claim 4, characterized in that The energy loss function is formulated as follows: Among them, E represents the energy loss function, p represents the pixel index, q represents another pixel index, and N p represents the neighborhood of pixel p, w pq represents the similarity weight between pixel p and pixel q, ρ(·) represents the robust function for handling outliers, and I p Represents the image intensity at pixel p, I q represents the image intensity at pixel q, d p Indicates the depth value of pixel p, d q represents the depth value of pixel q, λ represents the balance coefficient, and α represents the truncation threshold.

6. The method according to claim 1, characterized in that Based on the wound tissue type classification results and wound parameter measurement results, the robot path planning algorithm is used to construct the reward goal of wound treatment and combine the penalty constraints of robot motion to generate the optimal robot motion trajectory including: A mapping relationship between wound status and healing stage is established based on a machine learning algorithm to construct a wound status assessment model, wherein the wound status includes wound tissue type distribution, area and color, and the healing stage includes inflammation stage, proliferation stage and remodeling stage; Based on the output results of the wound status assessment model, the weight of the reward target is adjusted in real time. When the wound status changes, the corresponding weight coefficient in the reward target is updated to determine the dynamic reward weight, including: when the healing stage is in the inflammatory stage, the weight coefficient corresponding to tissue clearance is increased, and the weight coefficient corresponding to tissue protection is reduced; when the healing stage is in the proliferative stage, the weight coefficient corresponding to tissue protection is increased, and the weight coefficient corresponding to tissue migration is reduced; Based on the wound treatment trajectory data collected in advance through expert teaching, the corresponding wound state features and treatment motion features are extracted to construct state-action pairs; Based on the maximum entropy inverse reinforcement learning algorithm, the parameters of the reward function are determined by maximizing the likelihood probability of the wound treatment trajectory data to obtain the initial reward function. The difference between the wound treatment trajectory data and the generated trajectory obtained by the maximum entropy inverse reinforcement learning algorithm is minimized through the gradient descent algorithm, and the parameters of the reward function are adjusted to obtain the implicit reward function. Taking different reward targets as independent objective functions, combining dynamic reward weights and implicit reward functions, taking robot motion constraints as constraint conditions, building a multi-objective optimization model, and using a multi-objective evolutionary algorithm to optimize and solve based on the multi-objective optimization model to obtain an optimal solution set, each optimal solution in the Pareto optimal solution set corresponds to a robot motion trajectory; In combination with current treatment needs and doctor's wishes, a robot motion trajectory is selected from the Pareto optimal solution set to determine the optimal robot motion trajectory.

7. The method according to claim 6, characterized in that The multi-objective evolutionary algorithm comprises: Based on preset historical data and prior knowledge, a knowledge-guided initialization strategy is adopted to determine the initial population; Take the initial population as the current population, and for each individual in the current population, calculate the standard fitness value under each independent objective function corresponding to different reward targets; The grid-based adaptive crowding measurement mechanism dynamically adjusts the grid size according to the distribution characteristics of the current population in the target space, and calculates the individual crowding by combining the grid density and the Euclidean distance between individuals. Based on the standard fitness value, combined with the dominance relationship of each individual in the current population and the individual crowding, the standard fitness value is dynamically adjusted, and the individual fitness value is determined through weight aggregation. Based on the individual fitness value, the individuals in the current population are sorted to determine the optimal individual, and the parent individual is determined according to the preset preferred number; Based on simulating binary crossover and polynomial mutation, a crossover mutation operator is determined, and the crossover mutation operator parameters are adjusted based on the number of iterations. The crossover mutation operator is used to perform a crossover mutation operation on the parent individuals to generate offspring individuals. Based on the parent individuals, random individuals are determined according to a preset generation ratio through uniform sampling and Gaussian perturbation. The optimal individuals, offspring individuals and random individuals are merged to generate an extended population. Repeat the iteration until the preset number of iterations is reached to obtain the optimal extended population. Based on the individual fitness value of each individual in the optimal extended population, the optimal solution set is obtained according to the preset preferred number.

8. A robot-assisted wound treatment system based on multimodal image analysis, used to implement the method according to any one of claims 1 to 7, characterized in that: include: The first unit is used to collect multimodal images of a wound to be treated, perform geometric transformation and spatial alignment on the multimodal images, determine an image registration result, reconstruct the image registration result by a multi-scale fusion algorithm, determine a fused wound image, perform wound region segmentation on the fused wound image by a method based on a convolutional neural network to obtain a segmented wound region image, and perform wound tissue type classification on the segmented wound region image by a method based on transfer learning to obtain a wound tissue type classification result; The second unit is used to perform surround shooting of the wound area to obtain a multi-view image of the wound, based on the multi-view image of the wound, taking the first frame depth map corresponding to the first frame image as a reference, aligning subsequent depth maps frame by frame and accumulating point cloud data, determining a three-dimensional point cloud of the wound, generating a three-dimensional reconstruction result of the wound through point cloud smoothing combined with a surface evolution algorithm, and determining a measurement result of wound parameters based on the three-dimensional reconstruction result of the wound; The third unit is used to generate the optimal motion trajectory of the robot by using a robot path planning algorithm based on the wound tissue type classification results and wound parameter measurement results, by constructing a reward goal for wound treatment and combining it with the penalty constraints of the robot motion. The robot control algorithm based on vision and force feedback can adjust the robot's motion parameters in real time to complete wound-assisted treatment.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images