A method and system for 3D reconstruction of skin scars based on multimodal fusion of vision and lidar

Abstract

The present application relates to the technical field of skin information processing, and discloses a method and system for three-dimensional reconstruction of skin scars based on multimodal fusion of vision and laser radar. The method uses vision and laser radar to perform three-dimensional scanning and reconstruction on skin scars respectively, and then fuses the results of the two to obtain the final three-dimensional reconstruction result of the skin scar. During the reconstruction process, images are collected at the optimal viewing angle according to quantitative visual acquisition rules, and factors affecting the image quality by noise interference are taken into account. Scar sub-areas with different structural complexity are scanned with different scanning intensities of the laser radar, which can improve the speed and accuracy of the three-dimensional reconstruction.

Description

Technical Field

[0001] The present application relates to the technical field of skin information processing, and in particular to a method and system for three-dimensional reconstruction of skin scars based on multi-modal fusion of vision and laser radar. Background Art

[0002] 3D modeling of skin scars is of great reference significance for evaluating the condition of scars, monitoring scar changes and guiding surgical plans. However, due to the various types of skin scars, complex tissue structures and significant differences in color changes, traditional single-modality scanning, such as single LiDAR 3D scanning or single vision-based 3D scanning, cannot meet the requirements of high-precision 3D scanning of skin scars.

[0003] Among them, the main shortcomings of three-dimensional reconstruction of skin scars based on LiDAR are: in the skin scar area, the complexity of the scar structure in different sub-areas is different. If high-intensity and dense scanning of LiDAR is used in the entire scar area, the scanning time will be too long; if low-intensity and sparse scanning of LiDAR is used in the entire scar area, the reconstruction accuracy will be insufficient. Therefore, it is necessary to perform sparse scanning in sub-areas with flat structures and dense scanning in sub-areas with complex structures in order to achieve efficient and high-precision three-dimensional reconstruction. However, the existing LiDAR three-dimensional reconstruction technology does not have a quantitative solution to this problem. In addition, LiDAR three-dimensional reconstruction also has significant disadvantages in obtaining color and texture information, which further leads to insufficient accuracy of three-dimensional reconstruction results.

[0004] The main shortcomings of vision-based three-dimensional reconstruction of skin scars are that the existing technology does not collect images at the optimal perspective according to the quantitative collection rules during the process of collecting multi-perspective images, and after completing the image collection, it does not identify and evaluate the perspective of the collected images, resulting in serious perspective duplication and missing perspectives in the collected images. In addition, in bad weather or low light conditions, the image quality is seriously affected, resulting in insufficient accuracy of the three-dimensional reconstruction results.

[0005] It can be seen that the three-dimensional reconstruction method of skin scars in the prior art has the problem of low accuracy. Summary of the invention

[0006] The present application provides a method and system for three-dimensional reconstruction of skin scars based on multi-modal fusion of vision and lidar, so as to solve the problem of low precision of the three-dimensional reconstruction method of skin scars in the prior art.

[0007] In order to achieve the above objectives, the present application is implemented through the following technical solutions:

[0008] In a first aspect, the present application provides a method for three-dimensional reconstruction of skin scars based on multi-modal fusion of vision and laser radar, comprising:

[0009] S1: Collect multi-view datasets of skin scar images to be 3D imaged;

[0010] S2: inputting the multi-view data set into a view perception module to perform shooting view evaluation and image quality evaluation to obtain angle confidence and quality confidence;

[0011] S3: Screening out a multi-view data set that meets the requirements based on the angle confidence and the quality confidence, and inputting the multi-view data set that meets the requirements into a three-dimensional reconstruction module to obtain a visual reconstruction result;

[0012] S4: dividing the scar area in the skin scar image to be three-dimensionally imaged into a plurality of sub-areas, and setting a corresponding scanning intensity for each sub-area, performing sparse scanning using a laser radar according to the set different scanning intensities to obtain point cloud data, and converting the point cloud data into a three-dimensional model;

[0013] S5: Fusing the visual reconstruction result and the three-dimensional model to obtain a final three-dimensional reconstruction result of the skin scar.

[0014] In a second aspect, the present application provides a three-dimensional reconstruction system for skin scars based on multi-modal fusion of vision and lidar, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the computer program.

[0015] Beneficial effects:

[0016] The present application provides a method for 3D reconstruction of skin scars based on multimodal fusion of vision and lidar. After using vision and lidar to perform 3D scanning and reconstruct the skin scar respectively, the results of the two are fused to obtain the final 3D reconstruction result of the skin scar. During the reconstruction process, images are collected at the optimal viewing angle according to the quantitative visual acquisition rules, and the factor that the image quality is affected by noise interference is taken into account. Scar sub-areas with different structural complexity are scanned with different scanning intensities of the lidar, which can improve the speed and accuracy of 3D reconstruction.

[0017] In a further solution, the use of multi-view shooting can improve the integrity and accuracy of 3D reconstruction data, reduce errors caused by factors such as shadows and light, and improve the speed and accuracy of 3D reconstruction.

[0018] In a further solution, the timing information is used as a supplement to improve the accuracy of feature point detection.

[0019] In a further scheme, the complexity of the scar sub-area is taken into consideration, and the lidar can be required to perform quantitative scanning with different scanning emphases on different areas. For areas with low complexity, low-intensity scanning is used to save time; for areas with high complexity, high-intensity scanning is used to help improve the accuracy of three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of a method for three-dimensional reconstruction of skin scars based on multi-modal fusion of vision and laser radar according to a preferred embodiment of the present application;

[0021] Figure 2 A schematic diagram of a skin scar visual three-dimensional scanning method according to a preferred embodiment of the present application;

[0022] Figure 3 A schematic diagram of a multi-view shooting method of a preferred embodiment of the present application;

[0023] Figure 4 A schematic diagram of a three-dimensional space of a preferred embodiment of the present application;

[0024] Figure 5 A schematic diagram of calculating the angle confidence of a preferred embodiment of the present application;

[0025] Figure 6 A schematic diagram of a rough division of scar areas in a preferred embodiment of the present application;

[0026] Figure 7 This is a schematic diagram of comparing the differences of symmetric secondary sub-regions in a preferred embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solution of the present application is described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0028] Unless otherwise defined, the technical terms or scientific terms used in this application should be understood by people with ordinary skills in the field to which this application belongs. The "first", "second" and similar words used in this application do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one" or "one" do not indicate quantity restrictions, but indicate the existence of at least one. Similar words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship also changes accordingly.

[0029] See also Figure 1 This embodiment provides a skin scar 3D reconstruction method based on vision and laser radar multimodal fusion, including:

[0030] S1: Collect multi-view datasets of skin scar images to be 3D imaged;

[0031] S2: Input the multi-view data set into the view perception module to perform shooting view evaluation and image quality evaluation to obtain angle confidence and quality confidence;

[0032] S3: Screen out a multi-view dataset that meets the requirements based on the angle confidence and the quality confidence, and input the multi-view dataset that meets the requirements into a 3D reconstruction module to obtain a visual reconstruction result;

[0033] S4: dividing the scar area in the skin scar image to be three-dimensionally imaged into a plurality of sub-areas, and setting a corresponding scanning intensity for each sub-area, performing sparse scanning using a laser radar according to the set different scanning intensities to obtain point cloud data, and converting the point cloud data into a three-dimensional model;

[0034] S5: The visual reconstruction result and the 3D model are integrated to obtain the final 3D reconstruction result of the skin scar.

[0035] The above-mentioned three-dimensional reconstruction method of skin scars based on multimodal fusion of vision and lidar uses vision and lidar to perform three-dimensional scanning and reconstruct the skin scar respectively, and then fuses the results of the two to obtain the final three-dimensional reconstruction result of the skin scar. During the reconstruction process, images are collected at the optimal viewing angle according to the quantitative visual acquisition law, and the factor that the image quality is affected by noise interference is taken into account. Scar sub-areas with different structural complexity are scanned with different scanning intensities of the lidar, which can improve the speed and accuracy of three-dimensional reconstruction.

[0036] The steps of the above method are described in detail below with a complete embodiment.

[0037] 1. Schematic diagram of the method steps for visual 3D scanning and reconstruction of skin scars Figure 2 As shown, specifically including:

[0038] 1.1. Collect skin scar images and annotate them, including scar type and scar area location; construct a neural network-based skin scar image intelligent recognition model, and use the annotated skin scar image data for end-to-end training to input scar images and output scar type and scar area location.

[0039] 1.2. For the skin scar to be imaged in three dimensions, the skin scar image is firstly photographed by a conventional camera and then input into the trained skin scar image intelligent recognition model for processing to obtain the scar type and scar area location.

[0040] 1.3. Based on the scar type and scar area location output by the skin scar image intelligent recognition model, use a camera to take multi-angle shots. The shooting steps are as follows:

[0041] Step 1: In terms of angle, the vertical axis perpendicular to the center of the scar area is used as the rotation axis, and the distance from the edge of the scar area is Each rotation Shoot at different angles until 360° shooting is completed along the edge of the scar area;

[0042] Step 2: Update the shooting distance , repeat step 1 to continue shooting until ;

[0043] Step 3: Update the shooting angle , repeat steps 1 and 2 until .

[0044] Specifically, the multi-angle shooting method is as follows: Figure 3 As shown, during the shooting process, the shots are taken in sequence according to the preset shooting angle sequence.

[0045] In this embodiment, the use of a multi-view shooting method can improve the integrity and accuracy of the three-dimensional reconstruction data, reduce errors caused by factors such as shadows and light, and improve the speed and accuracy of the three-dimensional reconstruction.

[0046] 1.4. After completing the multi-view data collection, the multi-view data set is sent to the view perception module for shooting view evaluation and image quality evaluation.

[0047] First, the key corners in the images in the multi-view dataset are detected through the 3D mapping module. The key corners refer to the points with prominent image feature attributes, such as the intersection of lines, the end points of line segments, etc. The multi-view images are restored to the 3D space with reference to the shooting angles and positions recorded during the shooting process. Figure 4 shown.

[0048] Through the adaptive spatial detection module, the degree of fit between the angle position of the multi-view images at different spatial positions in the three-dimensional space and the optimal three-dimensional reconstruction standard angle is evaluated, thereby giving each multi-view image an angle confidence. . Angle confidence is as follows Figure 5 As shown, the angle confidence The calculation method is as follows:

[0049] ;

[0050] In the formula, is the angle between the multi-view image and the standard view, ; ; The larger the value, the more consistent the angular position of the multi-view image is with the optimal standard 3D reconstruction angle.

[0051] In this embodiment, by comparing the degree of fit between the acquired multi-view images and the standard view, the accuracy of the angles of the acquired multi-view images can be effectively evaluated, which helps to improve the accuracy of three-dimensional reconstruction.

[0052] The quality perception module evaluates the quality of images in the multi-view dataset, including distortion such as defocus, unevenness, noise interference, and the content integrity of the area of ​​interest, thereby giving each multi-view image a quality confidence level. , ;

[0053] Set the angle confidence and quality confidence After adding, the multi-view image is obtained, which is suitable for the final image confidence of 3D reconstruction. ; Multi-view identification module is used to compare image confidence With confidence threshold and exclude multi-view images whose image confidence is less than a threshold;

[0054] After the multi-view identification module eliminates the multi-view images that do not meet the requirements, it is determined whether the remaining multi-view images meet the minimum number requirements for three-dimensional reconstruction.

[0055] If the requirements are met, the multi-view images and the corresponding image confidences are directly sent to the 3D reconstruction module for reconstruction;

[0056] If the requirements are not met, the conditional diffusion model trained by the text-based image generation method can directly use the angle and position as conditional information to generate the missing multi-view images and corresponding image confidence, and send them to the 3D reconstruction module for reconstruction.

[0057] In this implementation, the text refers to the existing angle and position information. During the training process, the existing angle and position information is used as a condition to directly train the conditional diffusion model with the corresponding image.

[0058] 1.5. The 3D reconstruction module includes two parts: a time-series-based feature point detection module and a feature matching-based 3D reconstruction module. The time-series-based feature point detection module first detects the key feature points in all input multi-view images based on image features, and then calculates the key feature point coordinates in the multi-view images adjacent to each other in time sequence according to the shooting time and position of each multi-view image and the key feature point coordinates. The key feature point coordinates calculated based on time sequence and the key feature point coordinates based on image detection are averaged, and the average coordinate position of the two is updated as the final key feature point.

[0059] In this implementation, the time-series-based feature point detection module can use the time-series information as a supplement to improve the accuracy of feature point detection.

[0060] First, accurate matching is performed based on the feature point information in the multi-view image, and the convolutional deep network is used to efficiently extract the features of the multi-view image. Then, combined with the image confidence, it is multiplied as the weight of the image feature to ensure that in the 3D reconstruction process, the contribution of each multi-view image feature to the 3D reconstruction is proportional to its reliability. Finally, the point cloud generation and surface reconstruction technology are used to fuse these weighted feature point information to accurately reconstruct the 3D model.

[0061] 2. The method of skin scar laser radar 3D scanning is as follows:

[0062] 2.1. First, randomly and uniformly set The seed points are used as the initial center points of the regional division. Then, a high-precision thermometer (with an accuracy of 0.01°) is used to precisely measure the temperature distribution in the scar area, and the temperature at the seed point is used as the center temperature for temperature clustering. The adjacent areas of the scar area are divided into the same area, and the scar area is roughly divided into the following results: Figure 6 As shown:

[0063] ;

[0064] Where S represents the scar area, Represents the roughly divided sub-region.

[0065] 2.2, such as Figure 7 As shown, the scar sub-areas were compared The secondary symmetry region in Figure 6 middle , , and ), the sub-regions after the rough division of the scar area are adjusted twice. If there is no significant difference in the distribution of the secondary symmetric regions, the structural complexity of the scar sub-regions is consistent and no adjustment is required; if there is a significant difference in the distribution of the secondary symmetric regions, the structural complexity of the scar sub-regions is inconsistent, and the sub-regions need to be adjusted twice according to the set adjustment method. The set adjustment method includes: irregularly shrinking the edge contour of the secondary sub-region inward. In this way, by irregularly shrinking inward, the secondary symmetric region with no significant difference in distribution can be better found in a random way.

[0066] The specific method of comparing whether there are significant differences in the secondary symmetric regions in the roughly divided scar sub-regions is as follows:

[0067] Step 1: Use mutually perpendicular area division axes to divide the roughly divided scar sub-areas It is further divided into four secondary sub-areas. ;

[0068] Step 2: Expand the secondary sub-region into a one-dimensional pixel sequence and compare the symmetrical secondary sub-regions respectively and , and The pixel sequence length, if the pixel sequence length or ; represents the length of the pixel sequence, then the pixel sequence length correction is first performed so that , ;

[0069] The pixel sequence length correction includes: drawing a distribution relationship diagram between the pixel values ​​and spatial positions of the pixel sequences with longer pixel sequence lengths in the symmetrical secondary sub-region, averaging the adjacent pixels with similar pixel values ​​in the distribution relationship diagram, thereby reducing the number of pixels.

[0070] Step 3: Calculate the pixel sequence after correction length and , and Pearson product-moment correlation coefficient (Pearson correlation coefficient) and unpaired T-test value;

[0071] Step 4: Rotate the area division axis clockwise, with a step size of , a total rotation of 180°, repeat steps 1 to 3 after each rotation, and calculate the total Group Pearson product-moment correlation coefficient and unpaired T-test value;

[0072] Step 5: Calculation Group Pearson product-moment correlation coefficient and unpaired T-test The average value and :

[0073] like or , it is considered that there is no significant difference between the mutually symmetrical secondary sub-regions;

[0074] like and , it is considered that there are significant differences between the mutually symmetrical secondary sub-regions.

[0075] In this embodiment, the complexity can be taken into consideration and the laser radar can be required to perform quantitative scanning with different scanning emphases for different areas. For areas with low complexity, low-intensity scanning is used to save time; for areas with high complexity, high-intensity scanning is used to help improve the accuracy of three-dimensional reconstruction.

[0076] 2.3. Analyze the structural complexity of the sub-area and set the LiDAR scanning intensity according to the structural complexity.

[0077] Calculate the coefficient of variation of pixel values ​​in the entire scar area And each sub-area Coefficient of variation of internal pixel values , calculated as follows:

[0078] ;

[0079] in, is the standard deviation of the pixel values ​​in the region, is the average value of the pixels in the area;

[0080] Calculate the difference between the coefficient of variation of each sub-area and the coefficient of variation of the entire scar area .

[0081] In the entire scar area and in each sub-area, the pixel value of each dermatoglyphic line is taken along the direction of the dermatoglyphic line, and the correlation coefficient between the pixel values ​​of all dermatoglyphic lines in the entire scar area is calculated, and the average value of the absolute value of the correlation coefficient is calculated. The correlation coefficient between the pixel values ​​of all dermatoglyphic lines in each sub-area is calculated, and the average value of the absolute value of the correlation coefficient is calculated. The average value of the absolute value of the correlation coefficient in the entire scar area is obtained and the average absolute value of the correlation coefficient in each sub-region The correlation coefficient is calculated as follows:

[0082] ;

[0083] Among them, X and Y represent two sets of data for which the correlation coefficient is to be calculated; represents the covariance between X and Y; and Indicates the standard deviation of X and Y.

[0084] According to the coefficient of variation within the entire scar area The average correlation coefficient , set the scanning intensity of the laser radar in the area to I, compare the difference between the average value of the coefficient of variation and the correlation coefficient of each sub-area and the two values ​​in the entire scar area, and determine the scanning intensity in each sub-area respectively, specifically:

[0085] If the difference in coefficient of variation between the scar sub-area and the entire scar area >0, and the average correlation coefficient in this sub-region , then set the scanning intensity in this sub-area to , satisfying the relationship: , represents the scan intensity within the entire scar area;

[0086] If the difference in coefficient of variation between the scar sub-area and the entire scar area >0, and the average correlation coefficient in this sub-region , then set the scanning intensity in this sub-area to , satisfying the relationship: ;

[0087] If the difference in coefficient of variation between the scar sub-area and the entire scar area <0, and the average correlation coefficient in this sub-region , then set the scanning intensity in this sub-area to , satisfying the relationship: ;

[0088] If the difference in coefficient of variation between the scar sub-area and the entire scar area <0, and the average correlation coefficient in this sub-region , then set the scanning intensity in this sub-area to , satisfying the relationship: .

[0089] 2.4. In different sub-areas of the scar area, sparse scanning is performed using lidar according to the set scanning intensities, and the scanned point cloud data is processed using a deep learning model, such as a convolutional neural network, Transformer and other architectures, to convert the point cloud data into a three-dimensional model.

[0090] 3. Fuse the vision-based and lidar-based 3D reconstruction results to obtain the final skin scar 3D reconstruction result. The fusion method is as follows:

[0091] 3.1. The 3D reconstruction results based on vision and lidar are respectively After the size is divided into voxels, the two are matched between each voxel, where H represents the height of the voxel, L represents the length of the voxel, and W represents the width of the voxel;

[0092] If the voxel overlap ratio of the two 3D reconstruction results is greater than 0 after a certain voxel is matched, the overlapped part is retained as the fusion result;

[0093] If the voxel overlap ratio is 0 after a certain voxel of the 3D reconstruction results of the two is matched, the 3D reconstruction result based on the lidar is used as the fusion result.

[0094] The surface color of the three-dimensional reconstruction result based on vision is used as the surface color of the three-dimensional reconstruction result of the skin scar after fusion.

[0095] This embodiment also provides a skin scar 3D reconstruction system based on multimodal fusion of vision and laser radar, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above method when executing the computer program. The skin scar 3D reconstruction system based on multimodal fusion of vision and laser radar can implement various embodiments of the above method and achieve the same beneficial effects, which will not be described in detail here.

[0096] The preferred specific embodiments of the present application are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes according to the concept of the present application without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present application on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for three-dimensional reconstruction of skin scars based on multimodal fusion of vision and laser radar, characterized in that: include: S1: Collect multi-view datasets of skin scar images to be 3D imaged; S2: inputting the multi-view data set into a view perception module to perform shooting view evaluation and image quality evaluation to obtain angle confidence and quality confidence; S3: Screening out a multi-view data set that meets the requirements based on the angle confidence and the quality confidence, and inputting the multi-view data set that meets the requirements into a three-dimensional reconstruction module to obtain a visual reconstruction result; S4: dividing the scar area in the skin scar image to be three-dimensionally imaged into a plurality of sub-areas, and setting a corresponding scanning intensity for each sub-area, performing sparse scanning using a laser radar according to the set different scanning intensities to obtain point cloud data, and converting the point cloud data into a three-dimensional model; S5: fusing the visual reconstruction result and the three-dimensional model to obtain a final three-dimensional reconstruction result of the skin scar; The S4 includes: Dividing a scar region in a skin scar image to be three-dimensionally imaged into a plurality of sub-regions; Calculate the coefficient of variation and correlation coefficient for each sub-region; Determine the scanning intensity in each sub-region according to the coefficient of variation and the correlation coefficient; Using laser radar to perform sparse scanning according to different set scanning intensities, the scanned point cloud data is processed using a deep learning model to convert the point cloud data into a three-dimensional model; The step of dividing the scar region in the skin scar image to be three-dimensionally imaged into a plurality of sub-regions comprises: Randomly set evenly within the scar area A seed point, as the initial center point of the region division; The temperature distribution in the scar area is obtained, and the temperature at the seed point is used as the central temperature for temperature clustering. The adjacent areas whose temperature difference with the central temperature is less than or equal to the temperature threshold are divided into the same scar sub-area, and the scar sub-area after rough division of the scar area is obtained. ; Compare the significant differences of the secondary symmetric regions in the roughly divided scar sub-regions. If there is no significant difference in the distribution of the secondary symmetric regions, the structural complexity of the scar sub-regions is consistent and no adjustment is required. If there is a significant difference in the distribution of the secondary symmetric regions, the sub-regions are adjusted twice according to a set adjustment method, and the set adjustment method includes: irregularly shrinking the edge contour of the secondary sub-region inward; The significant differences of the secondary symmetric regions in the roughly divided scar sub-regions include: Step 1: Use mutually perpendicular area division axes to divide the roughly divided scar sub-areas It is further divided into four secondary sub-areas. ; Step 2: Expand the secondary sub-region into a one-dimensional pixel sequence and compare the symmetrical secondary sub-regions respectively and , and The pixel sequence length, if the pixel sequence length or , then the pixel sequence length is corrected first, so that , ; Indicates the length of the pixel sequence; Step 3: Calculate the pixel sequence after correction length and , and Pearson product-moment correlation coefficient and unpaired T-test value; Step 4: Rotate the area division axis clockwise, with a step size of , a total rotation of 180°, repeat steps 1 to 3 after each rotation, and calculate the total Group Pearson product-moment correlation coefficient and unpaired T-test value; Step 5: Calculation Group Pearson product-moment correlation coefficient and unpaired T-test The average value and : like or , it is considered that there is no significant difference between the mutually symmetrical secondary sub-regions; like and , it is considered that there are significant differences between the mutually symmetrical secondary sub-regions.

2. The method for three-dimensional reconstruction of skin scars based on multimodal fusion of vision and laser radar according to claim 1 is characterized in that: The S1 includes: Taking a skin scar image, and inputting the skin scar image into a preset skin scar image intelligent recognition model, to obtain the scar type and scar area position output by the preset skin scar image intelligent recognition model; Based on the scar type and scar area location, multi-view shooting is performed according to set rules to obtain a multi-view data set of the skin scar to be three-dimensionally imaged; The setting rules are as follows: Step 1: In terms of angle, the vertical axis perpendicular to the center of the scar area is used as the rotation axis, and the distance from the edge of the scar area is Each rotation Shoot at different angles until 360° shooting is completed along the edge of the scar area; Step 2: Update the shooting distance , repeat step 1 to continue shooting until ; Step 3: Update the shooting angle , repeat steps 1 and 2 until .

3. The method for three-dimensional reconstruction of skin scars based on multi-modal fusion of vision and laser radar according to claim 1, characterized in that: The S2 includes: Restoring the multi-view data set to a three-dimensional space to obtain a multi-view image in the three-dimensional space; Determine the standard viewing angle for 3D reconstruction; Determine the degree of fit between the angular positions of the multi-view images at different spatial positions in the three-dimensional space and the standard viewing angle, and calculate the angular confidence of each multi-view image based on the degree of fit , satisfying the following relationship: ； In the formula, is the angle between the multi-view image and the standard view, ; ; Determine the quality confidence for each multi-view image based on the quality parameter , ; The quality parameters include defocus, unevenness, noise interference distortion and content integrity of the region of interest.

4. The method for three-dimensional reconstruction of skin scars based on multi-modal fusion of vision and laser radar according to claim 1, characterized in that: The S3 includes: Add the angle confidence and quality confidence to get the final image confidence; Compare the image confidence with the confidence threshold, and exclude the multi-view images whose image confidence is less than the confidence threshold; Determine whether the number of the remaining multi-view images meets the minimum number requirement for three-dimensional reconstruction, and if the minimum number requirement is met, input the multi-view images and the corresponding image confidences into the three-dimensional reconstruction module for reconstruction to obtain a visual reconstruction result; When the minimum number requirement is not met, the conditional diffusion model trained by the text-based image generation method is used to generate the missing multi-view images and corresponding image confidences according to the angle and position, and then sent to the 3D reconstruction module for reconstruction to obtain the visual reconstruction results; The three-dimensional reconstruction module includes a time-series-based feature point detection module and a feature matching-based three-dimensional reconstruction submodule, and the multi-view data set that meets the requirements is input into the three-dimensional reconstruction module to obtain a visual reconstruction result, including: Inputting the multi-view image into the time-series-based feature point detection module to obtain the coordinates of key feature points in the multi-view image based on image detection; According to the shooting time sequence and position of each multi-view image and the coordinates of the key feature points, the coordinates of the key feature points in the multi-view images adjacent to each other in time sequence are calculated; Averaging the coordinates of the key feature points in the temporally adjacent multi-view images and the coordinates of the key feature points in the multi-view images based on image detection, and updating the average coordinate position of the two as the final key feature point coordinates; The key feature point coordinates in the multi-view image based on image detection are used for accurate matching, and the convolutional deep network is used to efficiently extract the features of the multi-view image; Combine the image confidence of multi-view images and multiply them as the weight of image features; The weighted feature point information is fused with the features extracted from the multi-view images to reconstruct the three-dimensional model.

5. The method for three-dimensional reconstruction of skin scars based on multi-modal fusion of vision and laser radar according to claim 1, characterized in that: The calculation of the coefficient of variation and the correlation coefficient of each sub-region includes: Calculate the coefficient of variation of pixel values ​​in the entire scar area And each sub-area Coefficient of variation of internal pixel values , calculated as follows: ； in, is the standard deviation of the pixel values ​​in the region, is the average value of the pixels in the area; Calculate the difference between the coefficient of variation of each sub-area and the coefficient of variation of the entire scar area ; In the entire scar area and in each sub-area, the pixel value of each dermatoglyphic line is taken along the direction of the dermatoglyphics, and the correlation coefficient between the pixel values ​​of all dermatoglyphic lines in the entire scar area is calculated, and the average value of the absolute value of the correlation coefficient is obtained to obtain the average value of the absolute value of the correlation coefficient in the entire scar area. , calculate the correlation coefficients between the pixel values ​​of all dermatoglyphics in each sub-region, and find the average value of the absolute value of the correlation coefficients in each sub-region , where the correlation coefficient is calculated as follows: ； Among them, X and Y represent two sets of data for which the correlation coefficient is to be calculated; represents the covariance between X and Y; and Indicates the standard deviation of X and Y.

6. The method for three-dimensional reconstruction of skin scars based on multi-modal fusion of vision and laser radar according to claim 1, characterized in that: Determining the scanning intensity in each sub-region according to the coefficient of variation and the correlation coefficient includes: If the difference in coefficient of variation between the scar sub-area and the entire scar area >0, and the average correlation coefficient in this sub-region , then the scanning intensity in this sub-area is set to , satisfying the relationship: , Indicates the scan intensity within the entire scar area; If the difference in coefficient of variation between the scar sub-area and the entire scar area >0, and the average correlation coefficient in this sub-region , then the scanning intensity in this sub-area is set to , satisfying the relationship: ; If the difference in coefficient of variation between the scar sub-area and the entire scar area <0, and the average correlation coefficient in this sub-region , then the scanning intensity in this sub-area is set to , satisfying the relationship: ; If the difference in coefficient of variation between the scar sub-area and the entire scar area <0, and the average correlation coefficient in this sub-region , then the scanning intensity in this sub-area is set to , satisfying the relationship: .

7. The method for three-dimensional reconstruction of skin scars based on multi-modal fusion of vision and laser radar according to claim 1, characterized in that: The S5 includes: The visual reconstruction result and the three-dimensional model are combined to After the size is divided into voxels, the two are matched between each voxel, where H represents the height of the voxel, L represents the length of the voxel, and W represents the width of the voxel; If the voxel overlap ratio of the two 3D reconstruction results is greater than 0 after a certain voxel is matched, the overlapped part is retained as the fusion result; If the voxel overlap ratio is 0 after a certain voxel of the 3D reconstruction results of the two is matched, the 3D model based on lidar is used as the fusion result.

8. A three-dimensional reconstruction system for skin scars based on multimodal fusion of vision and laser radar, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 7 are implemented.

Images