Human body scanning system based on multi-view TOF sensor

Abstract

The invention discloses a human body scanning system based on a multi-view TOF sensor, which relates to the technical field of human body scanning and comprises a sensing module, a data fusion module, a real-time detection module, a natural illumination simulation module, a storage and processing module, a user interaction module, a remote support module, a mode switching module and a hole detection and supplementary scanning module. The sensing module is used for performing multi-view coverage on a scanned object, acquiring point cloud data and preprocessing the acquired data; the data fusion module is used for carrying out seamless fusion on the point cloud data obtained from different visual angles; the real-time detection module is used for generating a temporary 3D model in real time in the scanning process and providing feedback for a user. According to the method, the modes can be freely switched according to different scanning tasks, the method adapts to various scenes, the 3D model with high precision and rich details is generated, the shielding problem possibly caused by a single view angle is reduced through the multi-view-angle layout, and the integrity and accuracy of 3D modeling are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of human body scanning, and in particular to a human body scanning system based on a multi-viewing angle TOF sensor. Background Art

[0002] With the rapid development of 3D scanning technology, human body scanning has shown an increasingly broad application prospect in many fields, such as clothing customization, medical health, animation production and game development. Traditional human body scanning methods, such as laser or photography-based measurement systems, can obtain 3D information of the human body, but they still have certain limitations in accuracy, efficiency and ease of use. Although the laser scanning system has high accuracy, the equipment is expensive, the operation is complicated, and it has high requirements for the scanning environment, which limits its popularity in large-scale commercial applications. Although the scanning system based on photography has low cost, it often requires complex post-processing to restore the 3D shape and is sensitive to light conditions. The TOF sensor is easily affected by shadows and reflections, resulting in inaccurate scanning results. Time-of-flight sensor technology has attracted widespread attention in the field of 3D scanning due to its high precision, high speed and non-contact measurement. The TOF sensor calculates the distance by measuring the time it takes for a light pulse to be reflected and returned by an object, thereby obtaining the 3D information of the object. However, when scanning the human body, the single-view TOF sensor is easily affected by occlusion and self-occlusion, resulting in missing data in some areas, affecting the integrity and accuracy of the scanning results, and having poor flexibility and adaptability. In this regard, we propose a human body scanning system based on a multi-view TOF sensor. Summary of the invention

[0003] In order to solve the above technical problems, a human body scanning system based on a multi-view TOF sensor is provided. This technical solution solves the above problems affecting the integrity and accuracy of the scanning results, and the poor flexibility and adaptability.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is: a human body scanning system based on a multi-view TOF sensor, comprising: a sensing module, a data fusion module, a real-time detection module, a natural light simulation module, a storage and processing module, a user interaction module, a remote support module, a mode switching module and a cavity detection and supplementary scanning module;

[0005] The sensor module is used to cover the scanned object from multiple perspectives, collect point cloud data, and pre-process the collected data;

[0006] The data fusion module is used to seamlessly fuse point cloud data acquired from different perspectives;

[0007] The real-time detection module is used to generate a temporary 3D model in real time during the scanning process and provide feedback to the user;

[0008] The natural lighting simulation module is used to provide natural and uniform lighting conditions during scanning;

[0009] The storage and processing module is used to store the scan data and perform post-processing of the 3D model;

[0010] The user interaction module is used to display real-time scanning progress, temporary 3D models and user feedback;

[0011] The remote support module is used to provide remote diagnosis and maintenance services when a failure occurs during system operation or when technical support is needed;

[0012] The mode switching module is used to switch between single-camera mode and multi-camera mode for scanning;

[0013] The hole detection and supplementary scanning module analyzes the density and geometric features of the point cloud data on the surface of the 3D model to detect the areas that are not fully covered, namely "holes" in real time. It makes judgments based on the point cloud density analysis and optimizes it with the deep learning model to identify potential hole areas. After the hole detection, the user interaction module displays the area and provides visual prompts for users to identify the parts for supplementary scanning.

[0014] Preferably, the sensing module includes a sensor, and the multi-view coverage is performed by fixing the sensor on a moving device at different angles. When the sensor works synchronously and asynchronously, scanning is performed in a preset order, and data preprocessing includes denoising, downsampling and edge detection.

[0015] Preferably, the data fusion module fuses the acquired point cloud data in the following steps:

[0016] S1. Initialization, set two sets of point cloud data as P = {p1, p2, ..., p N} and Q={q1,q2,…,q M}, where P is the point cloud to be registered, Q is the target point cloud, and the rotation matrix R and translation vector t are initially set;

[0017] S2, nearest neighbor search, for each point p i ∈P, find the nearest neighbor point q in the point cloud Q i , the selection of the nearest neighbor pair is realized by the KD tree algorithm, and the formula is:

[0018]

[0019] Among them, ||·|| represents the Euclidean distance, q i is the nearest neighbor point in point cloud Q, p i With q j are two sets of point cloud data points;

[0020] S3, minimize the error, for each pair of matching points (p i ,q i ), minimize the alignment error of the two sets of point clouds, and define the error function E(R,t) as:

[0021]

[0022] Where R is the rotation matrix, t is the translation vector, and p i With q i is the matching point, N is the number of point cloud data, and the goal is to find R and t so that E(R,t) is the smallest;

[0023] S4, SVD solution, use the singular value decomposition SVD method to calculate the rotation matrix R and translation vector t, let: p i The centroid of q i The centroid of The decentralized point cloud is and The covariance matrix is ​​H = P' T Q'

[0024] Next, use SVD to decompose the covariance matrix H = UΣV T , in this decomposition:

[0025] U is a matrix describing the direction of the decentralized quasi-point cloud P' to be matched;

[0026] V is the matrix describing the direction of the decentralized target point cloud Q';

[0027] Σ is a diagonal matrix containing singular values, which reflects the similarity between point clouds;

[0028] Through U and V, the rotation matrix R is determined as:

[0029] R=VU T

[0030] The rotation matrix R is used to rotate the current point cloud to align with the target point cloud;

[0031] The translation vector t is:

[0032]

[0033] in For p i The center of mass, N is the number of point cloud data, p i With q i is the matching point, Q i The centroid of , P' and Q' are decentralized point clouds, P is the point cloud to be registered, Q is the target point cloud, and H is the covariance matrix;

[0034] S5, iterative update, update the position of point cloud P:

[0035] P←RP+t

[0036] S6. Repeat steps S2-S5 until the error function E(R, t) converges to a preset number of iterations, completing the splicing of point cloud data from different perspectives.

[0037] Preferably, the real-time detection module feedback is used to guide the user to adjust posture and expression through the intelligent prompt area; the natural lighting simulation module includes adjusting the light source, which automatically adjusts the brightness, angle and color temperature based on the real-time scanning feedback to ensure that the light is evenly distributed; the storage and processing module compresses and encodes the scanned data to reduce storage space, and the user interaction module displays a temporarily constructed 3D model for the user to view at 360°, and supports voice and gesture interaction; the remote support module is connected to the cloud server through the network, and is connected to the professional support team through the system interface to obtain technical assistance and troubleshooting.

[0038] Preferably, the mode switching module provides switching between two modes, namely, single camera mode and multi-camera mode. The single camera mode uses one camera for data acquisition. According to the initial position and viewing angle of the camera, the camera angle is adjusted according to the planned path, and the viewing angle is gradually adjusted. This process ensures that all important areas are covered through the viewing angle planning algorithm. The camera viewing angle adjustment is adjusted based on the transformation matrix of the three-dimensional coordinate system. The transformation matrix is:

[0039]

[0040] in is the transformation matrix, θ and are the pitch angle and yaw angle respectively;

[0041] The multi-camera mode uses multiple cameras to collect data. The system generates a 3D model through a multi-view fusion algorithm and transforms the homogeneous coordinate matrix T i =[R i |t i ]Unify the time coordinate system of different cameras, and the homogeneous coordinate transformation matrix is ​​in the form of:

[0042] P'=T i P=[R i |t i ]P

[0043] Where P is the original point, P' is the transformed point, and R i is the rotation matrix, t i is the translation vector;

[0044] Users can switch modes autonomously based on the current task volume.

[0045] Preferably, the mode switching module includes a dual-channel switching unit, which is two sets of independent working channels. The working channels can work simultaneously or be used independently, and can be switched according to the task volume requirements;

[0046] The switching method intelligently switches by calculating the working efficiency in each mode. The working efficiency calculation formula of the single-channel mode is:

[0047]

[0048] Where E single is the single-channel working efficiency, t single is the task processing time of a single channel, r single Resource consumption for a single channel;

[0049] The working efficiency calculation formula of dual-channel mode is:

[0050]

[0051] Where E dual is the working efficiency of dual channels, t dual is the dual-channel task processing time, r dual Resource consumption for dual channels.

[0052] Preferably, the point cloud density analysis formula in the hole detection and supplementary scanning module is:

[0053]

[0054] Where ρ(P i ) is the point cloud density, N i is the data of the points in the field, P i is a point in the point cloud, r is the neighborhood radius, if the obtained ρ(P i ) is lower than the set threshold ρ min , then P i Marked as part of the void area.

[0055] Preferably, the deep learning model optimization in the hole detection and supplementary scanning module performs hole detection on the point cloud data by using a convolutional neural network. Assuming that the input point cloud data is X, the network parameter is a, and the convolutional neural network output Y is the predicted hole area label, in the form of:

[0056] Y = CNN(X; a)

[0057] The network is trained by minimizing the loss function L(a), and the cross entropy loss is in the form of:

[0058]

[0059] where y i is the true label, is the predicted value output by the network.

[0060] Preferably, the hole detection and supplementary scanning module provides visual prompts to view and track the automatic movement process of the camera through the interface. When a hole is detected, the system notifies the user of the current hole situation and the progress of the supplementary scanning through the interface, voice or AR augmented reality technology.

[0061] Preferably, the hole detection and supplementary scanning module uses an intelligent filling algorithm to automatically generate a smooth and continuous 3D surface to fill the hole areas that cannot be fully covered by the camera movement. The intelligent filling algorithm analyzes the morphological features of the surrounding area and infers the surface shape of the unscanned area to generate a natural transition. After the hole filling is completed, the system performs a quality check on the entire 3D model to ensure that all holes have been correctly processed, and displays the final results and suggestions on the user interface.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] The present invention can switch modes freely according to different scanning tasks and adapt to various scenarios. Richer data information can be collected in different modes, thereby generating a more complete and realistic 3D model, improving scanning efficiency, and ensuring the integrity of the 3D model. By automatically detecting voids and performing supplementary scanning, human intervention is reduced, the intelligence and automation level of scanning is improved, and the depth data loss problem common in traditional systems is avoided. The multi-view system can capture all-round information of the target object or human body and generate a high-precision, detail-rich 3D model. The multi-view layout reduces the occlusion problem that may be caused by a single view, ensuring the integrity and accuracy of 3D modeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is a framework diagram of the human body scanning system of the present invention;

[0065] Figure 2 It is a schematic diagram of the structure of the human body scanning system of the present invention. DETAILED DESCRIPTION

[0066] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art may think of other obvious variations.

[0067] Reference Figure 1As shown, a human body scanning system based on a multi-view TOF sensor includes: a sensing module, a data fusion module, a real-time detection module, a natural light simulation module, a storage and processing module, a user interaction module, a remote support module, a mode switching module and a cavity detection and supplementary scanning module;

[0068] The sensor module is used to cover the scanned object from multiple perspectives, collect point cloud data, and pre-process the collected data;

[0069] The data fusion module is used to seamlessly fuse point cloud data acquired from different perspectives;

[0070] The real-time detection module is used to generate a temporary 3D model in real time during the scanning process and provide feedback to the user;

[0071] The natural lighting simulation module is used to provide natural and uniform lighting conditions during scanning;

[0072] The storage and processing module is used to store the scan data and perform post-processing of the 3D model;

[0073] The user interaction module is used to display real-time scanning progress, temporary 3D models and user feedback;

[0074] The remote support module is used to provide remote diagnosis and maintenance services when a failure occurs during system operation or when technical support is needed;

[0075] The mode switching module is used to switch between single-camera mode and multi-camera mode for scanning;

[0076] The hole detection and supplementary scanning module analyzes the density and geometric features of the point cloud data on the surface of the 3D model to detect the areas that are not fully covered, namely "holes" in real time. It makes judgments based on the point cloud density analysis and optimizes it with the deep learning model to identify potential hole areas. After the hole detection, the user interaction module displays the area and provides visual prompts for users to identify the parts for supplementary scanning.

[0077] In this application, you can freely switch modes according to different scanning tasks, such as fine surface capture or global contour scanning, to adapt to a variety of scenarios;

[0078] Optimize data collection. In different modes, richer data information can be collected, including depth data, color and texture information, so as to generate more complete and realistic 3D models.

[0079] Improve efficiency. Automatic mode switching can reduce unnecessary data collection steps in specific scenarios and improve scanning efficiency.

[0080] Improve the integrity of the 3D model. During multi-view scanning, some areas may be missing depth data due to occlusion or perspective issues. The hole detection function automatically finds these missing areas to ensure the integrity of the 3D model.

[0081] Automation and intelligence: By automatically detecting voids and performing supplementary scans, human intervention is reduced and the level of intelligence and automation of scanning is improved;

[0082] Improved accuracy: Supplementary scanning can significantly improve the detail accuracy of the model, especially on complex geometric surfaces or recessed areas, avoiding the depth data loss problem common in traditional systems;

[0083] Comprehensive viewing angle coverage: the multi-view system can capture all-round information of the target object or human body and generate a high-precision, detailed 3D model;

[0084] Reduce the impact of occlusion. The multi-view layout reduces the occlusion problem that may be caused by a single view, ensuring the integrity and accuracy of 3D modeling;

[0085] The real-time performance is improved. Combined with the camera mode switching and hole supplement scanning functions, the system can process data and generate 3D models at a higher speed, improving the overall real-time modeling capability.

[0086] The sensing module includes a sensor. The multi-view coverage is achieved by fixing the sensor on a mobile device at different angles. When the sensor works synchronously and asynchronously, it scans in a preset order. Data preprocessing includes denoising, downsampling and edge detection.

[0087] This application uses multiple sensors to obtain depth information of the target from different angles to ensure all-round, high-precision 3D modeling. By fixing the sensors on mobile devices at different angles, all-round, multi-angle monitoring of the monitored object or environment can be achieved. This multi-perspective coverage can capture details that cannot be observed from a single perspective, thereby greatly improving the comprehensiveness of monitoring.

[0088] The steps of fusing the acquired point cloud data by the data fusion module are as follows:

[0089] S1. Initialization, set two sets of point cloud data as P = {p1, p2, ..., p N} and Q={q1,q2,…,q M}, where P is the point cloud to be registered, Q is the target point cloud, and the rotation matrix R and translation vector t are initially set;

[0090] S2, nearest neighbor search, for each point p i ∈P, find the nearest neighbor point q in the point cloud Q i , the selection of the nearest neighbor pair is realized by the KD tree algorithm, and the formula is:

[0091]

[0092] Among them, ||·|| represents the Euclidean distance, q i is the nearest neighbor point in point cloud Q, p i With q j are two sets of point cloud data points;

[0093] S3, minimize the error, for each pair of matching points (p i ,q i ), minimize the alignment error of the two sets of point clouds, and define the error function E(R,t) as:

[0094]

[0095] Where R is the rotation matrix, t is the translation vector, and p i With q i is the matching point, N is the number of point cloud data, and the goal is to find R and t so that E(R,t) is the smallest;

[0096] S4, SVD solution, use the singular value decomposition SVD method to calculate the rotation matrix R and translation vector t, let: p i The centroid of q i The centroid of The decentralized point cloud is and The covariance matrix is ​​H = P' T Q'

[0097] Next, use SVD to decompose the covariance matrix H = UΣV T , in this decomposition:

[0098] U is a matrix describing the direction of the decentralized quasi-point cloud P' to be matched;

[0099] V is the matrix describing the direction of the decentralized target point cloud Q';

[0100] Σ is a diagonal matrix containing singular values, which reflects the similarity between point clouds;

[0101] Through U and V, the rotation matrix R is determined as:

[0102] R=VU T

[0103] The rotation matrix R is used to rotate the current point cloud to align with the target point cloud;

[0104] The translation vector t is:

[0105]

[0106] in For p i The center of mass, N is the number of point cloud data, p i With q i is the matching point, Q i The centroid of , P' and Q' are decentralized point clouds, P is the point cloud to be registered, Q is the target point cloud, and H is the covariance matrix;

[0107] S5, iterative update, update the position of point cloud P:

[0108] P←RP+t

[0109] S6. Repeat steps S2-S5 until the error function E(R, t) converges to a preset number of iterations, completing the splicing of point cloud data from different perspectives.

[0110] The real-time detection module feedback guides users to adjust their posture and expression through the intelligent prompt area; the natural lighting simulation module includes adjusting the light source, which automatically adjusts the brightness, angle and color temperature based on the real-time scanning feedback to ensure even distribution of light; the storage and processing module compresses and encodes the scanned data to reduce storage space, and the user interaction module displays a temporarily constructed 3D model for users to view at 360°, and supports voice and gesture interaction; the remote support module is connected to the cloud server through the network, and is connected to the professional support team through the system interface to obtain technical assistance and troubleshooting.

[0111] In the present application, the natural light simulation module uses a highly sensitive light sensor to scan the ambient light in real time, converts the light signal into an electrical signal, and converts the analog signal into a digital signal through an A / D converter. According to the scanned light data, the brightness, angle and color temperature of the light source are automatically adjusted through a control algorithm; the storage and processing module uses Huffman coding and LZW algorithm lossless compression methods to reduce storage space without losing any information. For image or video data, JPEG and MP3 lossy compression algorithms are used to further reduce storage space.

[0112] The mode switching module provides two modes of switching, namely single camera mode and multi-camera mode. The single camera mode uses one camera for data acquisition. According to the initial position and viewing angle of the camera, the camera angle is adjusted according to the planned path, and the viewing angle is gradually adjusted. This process ensures that all important areas are covered through the viewing angle planning algorithm. The camera viewing angle adjustment is based on the transformation matrix of the three-dimensional coordinate system. The transformation matrix is:

[0113]

[0114] in is the transformation matrix, θ and are the pitch angle and yaw angle respectively;

[0115] The multi-camera mode uses multiple cameras to collect data. The system generates a 3D model through a multi-view fusion algorithm and transforms the homogeneous coordinate matrix T i =[R i |t i ]Unify the time coordinate system of different cameras, and the homogeneous coordinate transformation matrix is ​​in the form of:

[0116] P′=T i P=[R i |t i ]P

[0117] Where P is the original point, P' is the transformed point, and R i is the rotation matrix, t i is the translation vector;

[0118] Users can switch modes autonomously based on the current task volume.

[0119] In the multi-camera mode of this application, the system needs to calibrate each camera to ensure that the relative positions and angles between different cameras are consistent in order to perform data fusion. The cameras can be selected to work simultaneously or sequentially. Different working orders will affect the speed and accuracy of data collection;

[0120] When working at the same time, all cameras synchronously collect data at the same time, ensuring that information from different perspectives is obtained and integrated at the same time, with higher accuracy. When working sequentially, the cameras are started in a preset order, and the next camera starts working after each camera completes collection. This is suitable for scenarios with strict requirements on resource utilization, but the time is longer. In single-camera mode, the system gradually adjusts the position and angle of a camera to collect data from different perspectives in sequence, which is suitable for low-cost, low-precision scenarios. In multi-camera mode, multiple cameras work simultaneously or are started sequentially, which can complete tasks faster and more accurately, and is suitable for high-precision scenarios. At the same time, the dual-channel system can combine the advantages of single and multi-cameras by integrating two working channels, and flexibly select the working mode according to actual needs.

[0121] The mode switching module includes a dual-channel switching unit, which is two sets of independent working channels. The working channels can work simultaneously or independently, and can be switched according to the task volume requirements;

[0122] The switching method intelligently switches by calculating the working efficiency in each mode. The working efficiency calculation formula of the single-channel mode is:

[0123]

[0124] Where E single is the single-channel working efficiency, t single is the task processing time of a single channel, r single Resource consumption for a single channel;

[0125] The working efficiency calculation formula of dual-channel mode is:

[0126]

[0127] Where E dual is the working efficiency of dual channels, t dual is the dual-channel task processing time, r dual Resource consumption for dual channels.

[0128] The point cloud density analysis formula in the void detection and supplementary scanning module is:

[0129]

[0130] Where ρ(P i ) is the point cloud density, N i is the data of the points in the field, P i is a point in the point cloud, r is the neighborhood radius, if the obtained ρ(P i ) is lower than the set threshold ρ min , then P i Marked as part of the void area.

[0131] The design of the dual-channel single instrument in this application can flexibly switch the working mode according to actual needs. In scenarios where large amounts of data and high precision are required, the two channels can work simultaneously, significantly improving work efficiency. In simpler or low-demand tasks, a single channel can meet the requirements, saving resources and power consumption.

[0132] Compared with using two independent instruments, the cost of a dual-channel single instrument is more advantageous, and it can achieve multi-tasking within the same hardware device, avoiding the high cost of purchasing and maintaining multiple devices;

[0133] With the dual-channel design, the two channels can collect data synchronously between different tasks, and integrate and process the data in real time through a unified system to ensure data consistency and real-time performance between different channels. Users can monitor the working status of the two channels through a single interface, improving user experience.

[0134] The single-instrument design allows the device to take up less space, making it particularly suitable for working environments with limited space. It also reduces the complexity of hardware connections and improves the mobility and portability of the device.

[0135] The deep learning model optimization in the hole detection and supplementary scanning module performs hole detection on point cloud data by using a convolutional neural network. Suppose the input point cloud data is X, the network parameter is a, and the convolutional neural network output Y is the predicted hole area label in the form of:

[0136] Y = CNN(X; a)

[0137] The network is trained by minimizing the loss function L(a), and the cross entropy loss is in the form of:

[0138]

[0139] where y i is the true label, is the predicted value output by the network.

[0140] The hole detection and supplementary scanning module provides visual prompts to view and track the automatic movement of the camera through the interface. When a hole is detected, the system notifies the user of the current hole situation and the progress of the supplementary scanning through the interface, voice or AR augmented reality technology.

[0141] The hole detection and supplementary scanning module uses an intelligent filling algorithm to automatically generate a smooth and continuous 3D surface to fill the hole areas that cannot be fully covered by the camera. The intelligent filling algorithm analyzes the morphological characteristics of the surrounding area and infers the surface shape of the unscanned area to generate a natural transition. After the hole is filled, the system performs a quality check on the entire 3D model to ensure that all holes have been correctly processed, and the final results and suggestions are displayed on the user interface.

[0142] This application detects uncovered areas in point cloud data in real time and provides feedback through the user interface. It automatically adjusts the viewing angle and position of the ToF sensor based on the real-time feedback, or allows the user to adjust it manually, and displays the adjusted effect in real time. It can automatically evaluate and adjust the acquisition parameters of the ToF sensor to ensure the quality of the data, support augmented reality display, and superimpose blind spots and quality prompt information in the user's field of view to improve the intuitiveness of the operation.

[0143] Example 1

[0144] Use LiDAR to replace the sensor module for 3D modeling. LiDAR acquires high-precision depth data by emitting laser pulses and measuring the return time, and is usually used for environmental perception and large-scale 3D modeling.

[0145] LiDAR has extremely high ranging accuracy and is particularly suitable for large-scale environment modeling. It can maintain high spatial resolution at long distances, but it is expensive and large in size, making it unsuitable for portable or small high-precision modeling equipment. For close-range high-precision modeling, LiDAR is too complex and redundant. For capturing details of the human body or small objects, LiDAR may be too rough and does not have the flexibility to switch camera modes.

[0146] Example 2

[0147] Using structured light scanning technology, a known pattern, such as stripes or dots, is projected onto the surface of an object and the depth information is calculated by capturing the pattern deformation through a camera.

[0148] It has high precision and detail capture, and is suitable for 3D scanning of complex geometric shapes and surface details. It is widely used in industrial-level measurement and quality control. However, it has high requirements for ambient light, and the effect is poor in strong light or outdoor environments. It also has high requirements for the synchronization of the projection device and the camera. It is not as good as ToF in terms of real-time performance and ease of operation. It has limited capabilities for hole detection and supplementary scanning, and it is difficult to dynamically adjust the mode.

[0149] Example 3

[0150] Sampling multi-view SLAM combines depth cameras and RGB cameras to achieve 3D modeling of objects by tracking camera positions and building environment maps.

[0151] It is capable of real-time 3D modeling and scene reconstruction in dynamic environments, can handle moving objects, and track them in combination with pose estimation. However, the system is highly complex and consumes large amounts of computing resources. It has limitations in hole detection and supplementary scanning, and it is difficult to accurately supplement data. Mode switching and high-precision real-time modeling are difficult to achieve, especially in rapidly changing scenes.

[0152] Example 4

[0153] By using binocular cameras or multi-cameras, left and right perspective images of the object are obtained, and depth information is calculated through the difference in images, thereby achieving 3D modeling.

[0154] No additional sensors are required, depth information can be collected only by relying on cameras. It is low-cost and suitable for scenes with rich textures. It performs well outdoors or in places with sufficient light. However, the depth data accuracy is low, especially on smooth surfaces or textureless objects. The effect is not good, and multi-view data fusion, hole detection and supplementary scanning functions cannot be easily realized. The flexibility of camera mode switching and high-precision 3D modeling is weak.

[0155] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A human body scanning system based on a multi-view TOF sensor, characterized in that: include: Sensing module, data fusion module, real-time detection module, natural light simulation module, storage and processing module, user interaction module, remote support module, mode switching module and cavity detection and supplementary scanning module; The sensor module is used to cover the scanned object from multiple perspectives, collect point cloud data, and pre-process the collected data; The data fusion module is used to seamlessly fuse point cloud data acquired from different perspectives; The real-time detection module is used to generate a temporary 3D model in real time during the scanning process and provide feedback to the user; The natural lighting simulation module is used to provide natural and uniform lighting conditions during scanning; The storage and processing module is used to store the scan data and perform post-processing of the 3D model; The user interaction module is used to display real-time scanning progress, temporary 3D models and user feedback; The remote support module is used to provide remote diagnosis and maintenance services when a failure occurs during system operation or when technical support is needed; The mode switching module is used to switch between single-camera mode and multi-camera mode for scanning; The hole detection and supplementary scanning module analyzes the density and geometric features of the point cloud data on the surface of the 3D model to detect the areas that are not fully covered, namely "holes" in real time. It analyzes and judges based on the point cloud density and optimizes it with the deep learning model to identify potential hole areas. After the hole is detected, the user interaction module displays the area and provides visual prompts for users to identify the parts for supplementary scanning.

2. A human body scanning system based on a multi-view TOF sensor according to claim 1, characterized in that: The sensing module includes a sensor. The multi-view coverage is achieved by fixing the sensor on a mobile device at different angles. When the sensor works synchronously and asynchronously, it scans in a preset order. Data preprocessing includes denoising, downsampling and edge detection.

3. The human body scanning system based on a multi-view TOF sensor according to claim 1, characterized in that: The steps of fusing the acquired point cloud data by the data fusion module are as follows: S1. Initialization, set two sets of point cloud data as P = {p1, p2, ..., p N } and Q={q1,q2,…,q M }, where P is the point cloud to be registered, Q is the target point cloud, and the rotation matrix R and translation vector t are initially set; S2, nearest neighbor search, for each point p i ∈P, find the nearest neighbor point q in the point cloud Q i , the selection of the nearest neighbor pair is realized by the KD tree algorithm, and the formula is: Among them, ||·|| represents the Euclidean distance, q i is the nearest neighbor point in point cloud Q, p i With q j are two sets of point cloud data points; S3, minimize the error, for each pair of matching points (p i ,q i ), minimize the alignment error of the two sets of point clouds, and define the error function E(R,t) as: Where R is the rotation matrix, t is the translation vector, and p i With q i is the matching point, N is the number of point cloud data, and the goal is to find R and t so that E(R,t) is the smallest; S4, SVD solution, use the singular value decomposition SVD method to calculate the rotation matrix R and translation vector t, let: p i The centroid of q i The centroid of The decentralized point cloud is and The covariance matrix is ​​H = P' T Q' Next, use SVD to decompose the covariance matrix H = UΣV T , in this decomposition: U is a matrix describing the direction of the decentralized quasi-point cloud P' to be matched; V is the matrix describing the direction of the decentralized target point cloud Q'; Σ is a diagonal matrix containing singular values, which reflects the similarity between point clouds; Through U and V, the rotation matrix R is determined as: R=VU T The rotation matrix R is used to rotate the current point cloud to align with the target point cloud; The translation vector t is: in For p i The center of mass, N is the number of point cloud data, p i With q i is the matching point, for q i The centroid of , P' and Q' are decentralized point clouds, P is the point cloud to be registered, Q is the target point cloud, and H is the covariance matrix; S5, iterative update, update the position of point cloud P: P←RP+t S6. Repeat steps S2-S5 until the error function E(R, t) converges to a preset number of iterations, completing the splicing of point cloud data from different perspectives.

4. The human body scanning system based on a multi-view TOF sensor according to claim 1, characterized in that: The real-time detection module feedback guides users to adjust their posture and expression through the intelligent prompt area; the natural lighting simulation module includes adjusting the light source, which automatically adjusts the brightness, angle and color temperature based on the real-time scanning feedback to ensure even distribution of light; the storage and processing module compresses and encodes the scanned data to reduce storage space, and the user interaction module displays a temporarily constructed 3D model for users to view at 360°, and supports voice and gesture interaction; the remote support module is connected to the cloud server through the network, and is connected to the professional support team through the system interface to obtain technical assistance and troubleshooting.

5. The human body scanning system based on multi-view TOF sensor according to claim 1, characterized in that: The mode switching module provides two modes of switching, namely single camera mode and multi-camera mode. The single camera mode uses one camera for data acquisition. According to the initial position and viewing angle of the camera, the camera angle is adjusted according to the planned path, and the viewing angle is gradually adjusted. This process ensures that all important areas are covered through the viewing angle planning algorithm. The camera viewing angle adjustment is based on the transformation matrix of the three-dimensional coordinate system. The transformation matrix is: in is the transformation matrix, θ and are the pitch angle and yaw angle respectively; The multi-camera mode uses multiple cameras to collect data. The system generates a 3D model through a multi-view fusion algorithm and transforms the homogeneous coordinate matrix T i =[R i |t i ]Unify the time coordinate system of different cameras, and the homogeneous coordinate transformation matrix is ​​in the form of: P′=T i P=[R i |t i ]P Where P is the original point, P' is the transformed point, and R i is the rotation matrix, t i is the translation vector; Users can switch modes autonomously based on the current task volume.

6. The human body scanning system based on multi-view TOF sensor according to claim 5, characterized in that: The mode switching module includes a dual-channel switching unit, which is two sets of independent working channels. The working channels can work simultaneously or independently, and can be switched according to the task volume requirements; The switching method intelligently switches by calculating the working efficiency in each mode. The working efficiency calculation formula of the single-channel mode is: Where E single is the single-channel working efficiency, t single is the task processing time of a single channel, r single Resource consumption for a single channel; The working efficiency calculation formula of dual-channel mode is: Where E dual is the working efficiency of dual channels, t dual is the dual-channel task processing time, r dual Resource consumption for dual channels.

7. The human body scanning system based on a multi-view TOF sensor according to claim 1, characterized in that: The point cloud density analysis formula in the void detection and supplementary scanning module is: Where ρ(P i ) is the point cloud density, N i is the data of the points in the field, P i is a point in the point cloud, r is the neighborhood radius, if the obtained ρ(P i ) is lower than the set threshold ρ min , then P i Marked as part of the void area.

8. The human body scanning system based on multi-view TOF sensor according to claim 1, characterized in that: The deep learning model optimization in the hole detection and supplementary scanning module performs hole detection on point cloud data by using a convolutional neural network. Suppose the input point cloud data is X, the network parameter is a, and the convolutional neural network output Y is the predicted hole area label in the form of: Y = CNN(X; a) The network is trained by minimizing the loss function L(a), and the cross entropy loss is in the form of: where y i is the true label, is the predicted value output by the network.

9. The human body scanning system based on multi-view TOF sensor according to claim 1, characterized in that: The hole detection and supplementary scanning module provides visual prompts to view and track the automatic movement of the camera through the interface. When a hole is detected, the system notifies the user of the current hole situation and the progress of the supplementary scanning through the interface, voice or AR augmented reality technology.

10. The human body scanning system based on multi-view TOF sensor according to claim 1, characterized in that: The hole detection and supplementary scanning module uses an intelligent filling algorithm to automatically generate a smooth and continuous 3D surface to fill the hole areas that cannot be fully covered by the camera. The intelligent filling algorithm analyzes the morphological characteristics of the surrounding area and infers the surface shape of the unscanned area to generate a natural transition. After the hole is filled, the system performs a quality check on the entire 3D model to ensure that all holes have been correctly processed, and the final results and suggestions are displayed on the user interface.

Images