Model generation method and device based on implicit neural representation, equipment and medium

Abstract

The invention relates to the technical field of ultrasonic detection, and discloses an implicit neural representation-based model generation method and device, equipment and a medium, and the method comprises the steps: obtaining a two-dimensional image sequence collected by detection equipment and spatial pose information corresponding to image frames, analyzing periodic motion, screening the image frames in a stable motion stage, and generating a to-be-reconstructed image sequence. And processing the to-be-reconstructed image frame, and extracting segmentation data of the target object. And training an implicit neural representation model based on the to-be-reconstructed image sequence, the spatial pose information and the target object segmentation data, inputting the three-dimensional space coordinates into the model, generating corresponding image information and semantic information, and generating a three-dimensional model of the target object. According to the method, the stable image frames are screened and the implicit neural representation model is trained by analyzing the spatial pose information, so that the data utilization efficiency and the precision and continuity of three-dimensional model reconstruction are improved, and high-precision continuous three-dimensional reconstruction is realized without external registration or breathing cooperation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of ultrasonic detection, and in particular to a model generation method and device based on implicit neural representation, equipment and a storage medium. BACKGROUND

[0002] Currently, as a non-invasive, non-ionizing radiation, real-time and cost-effective medical imaging method, ultrasonic imaging has been widely used in clinical practice, especially for the examination and follow-up of soft tissue structures such as the abdomen, heart and blood vessels. However, traditional ultrasonic examination is highly dependent on the experience and technical level of the operator, and there is significant inter-operator variability and low repeatability, which makes it difficult to meet the needs of large-scale screening and standardized clinical pathways. This human dependence not only affects the image quality and diagnostic stability, but also restricts the application and promotion of ultrasound in chronic disease follow-up and screening.

[0003] To reduce the influence of the operator and improve the standardization of the operation, robot-assisted ultrasonic systems have become a research hotspot. Related systems usually carry the probe through a mechanical arm to achieve path planning, contact force control and image acquisition automation. Some studies introduce image feedback and visual servoing control technology, combining deep learning algorithms to segment target structures in real time in ultrasonic images, which is used to guide the adjustment of the probe posture. Although these works have made some progress in the consistency of path planning and image acquisition, in a free-breathing environment, the coupling between device motion and patient body surface motion is complex, and the position and posture of ultrasonic image frames are easily unstable due to respiratory motion, which seriously affects the subsequent data utilization and image stability. The existing methods for screening image frames generally lack detailed analysis of the respiratory cycle motion and high-precision stable frame screening, resulting in low-quality image sequences for subsequent modeling, which increases the three-dimensional model reconstruction error.

[0004] In addition, the synchronization acquisition process of the ultrasonic images and their corresponding spatial pose information for training the three-dimensional model in the prior art relies on special hardware or external optical tracking systems, which not only increases the integration complexity and cost of the system, but also has poor adaptability to clinical environments. The current training process cannot guarantee high-precision alignment between image frames, timestamps and spatial poses, limiting the training effect and robustness of the deep learning model, especially in dynamic scanning and complex motion environments.

[0005] Therefore, the prior art still has deficiencies in image acquisition standardization, respiratory cycle motion compensation, stable frame screening, spatio-temporal alignment of training data and generation of high-quality data sets. These deficiencies directly lead to poor continuity, large registration error and lack of sensitivity to respiratory motion during subsequent three-dimensional model reconstruction, limiting the precision and robustness of three-dimensional reconstruction in robot-assisted ultrasonic systems, and making it difficult to meet the clinical demand for high-quality, stable and automated three-dimensional model reconstruction. SUMMARY

[0006] The main purpose of the present application is to provide a model generation method, device and equipment based on implicit neural representation and storage medium, aiming at solving the technical problem that the prior art lacks high-precision periodic motion analysis and stable frame screening mechanism based on spatial pose information of a detection device in a free breathing environment, resulting in low-quality and poor continuity of image sequence data for training and reconstruction.

[0007] To achieve the above-mentioned purpose, the present application provides a model generation method based on implicit neural representation, comprising:

[0008] obtaining a two-dimensional image sequence containing a target object collected by a detection device and spatial pose information corresponding to image frames of the two-dimensional image sequence;

[0009] analyzing periodic motion of the detection device based on the spatial pose information, and screening out image frames in a preset stable motion stage from the two-dimensional image sequence according to the analysis result of the periodic motion to generate a to-be-reconstructed image sequence;

[0010] processing image frames in the to-be-reconstructed image sequence to identify and extract the target object, and obtaining target object segmentation data corresponding to the to-be-reconstructed image sequence;

[0011] training an implicit neural representation model based on the to-be-reconstructed image sequence, the spatial pose information and the target object segmentation data;

[0012] inputting three-dimensional space coordinates in a preset three-dimensional space range into the implicit neural representation model to generate image information and semantic information corresponding to the three-dimensional space coordinates;

[0013] generating a three-dimensional model of the target object according to the image information and semantic information.

[0014] Further, to achieve the above-mentioned purpose, the present application provides a model generation device based on implicit neural representation, comprising:

[0015] a data acquisition module for obtaining a two-dimensional image sequence containing a target object collected by a detection device and spatial pose information corresponding to image frames of the two-dimensional image sequence;

[0016] a motion analysis module for analyzing periodic motion of the detection device based on the spatial pose information, and screening out image frames in a preset stable motion stage from the two-dimensional image sequence according to the analysis result of the periodic motion to generate a to-be-reconstructed image sequence;

[0017] a target segmentation module, configured to process image frames in the image sequence to be reconstructed to identify and extract the target object, and obtain target object segmentation data corresponding to the image sequence to be reconstructed;

[0018] a model training module, configured to train an implicit neural representation model based on the image sequence to be reconstructed, the spatial pose information, and the target object segmentation data;

[0019] a model inference module, configured to input a three-dimensional space coordinate in a preset three-dimensional space range into the implicit neural representation model to generate image information and semantic information corresponding to the three-dimensional space coordinate;

[0020] a three-dimensional reconstruction module, configured to generate a three-dimensional model of the target object according to the image information and the semantic information.

[0021] Further, to achieve the above object, the present application further provides a computer device, which comprises a memory, a processor, and a model generation program based on implicit neural representation stored in the memory and executable on the processor, and the model generation program based on implicit neural representation, when executed by the processor, implements the steps of the model generation method based on implicit neural representation.

[0022] Further, to achieve the above object, the present application further provides a computer readable storage medium, which stores a model generation program based on implicit neural representation, and the model generation program based on implicit neural representation, when executed by a processor, implements the steps of the model generation method based on implicit neural representation.

[0023] Beneficial Effects: This invention relates to the field of ultrasound detection technology and discloses a method, apparatus, device, and medium for model generation based on implicit neural representation. The method includes: acquiring a two-dimensional image sequence collected by a detection device and spatial pose information corresponding to the image frames; analyzing the periodic motion of the detection device based on the spatial pose information; selecting image frames in a stable motion phase from the two-dimensional image sequence to generate an image sequence to be reconstructed; processing the image frames in the image sequence to be reconstructed to extract segmentation data of the target object; training an implicit neural representation model based on the image sequence to be reconstructed, spatial pose information, and target object segmentation data; inputting three-dimensional spatial coordinates into the implicit neural representation model to generate image information and semantic information corresponding to the three-dimensional spatial coordinates; and generating a three-dimensional model of the target object based on the image information and semantic information. This invention accurately analyzes the periodic motion of the detection device using spatial pose information and selects high-quality image frames in a stable motion phase, ensuring the continuity and accuracy of training and reconstruction data. By combining the implicit neural representation model with training and three-dimensional reconstruction of stable frames, it can obtain a high-precision, continuous three-dimensional model of the target object without additional external registration or patient breathing coordination, improving data utilization efficiency and the stability of reconstruction results. Attached Figure Description

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0025] Figure 1 This is a schematic diagram of an application environment for a model generation method based on implicit neural representation in one embodiment of the present invention;

[0026] Figure 2 This is a flowchart illustrating an embodiment of the model generation method based on implicit neural representation of the present invention.

[0027] Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the model generation device based on implicit neural representation of the present invention;

[0028] Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0029] Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0030] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0031] The model generation method based on implicit neural representation provided in this invention can be applied to, for example... Figure 1In an application environment of the present application, a user terminal communicates with a server terminal through a network. The server terminal can obtain a two-dimensional image sequence and spatial pose information corresponding to the image frames from the detection device through the user terminal, analyze the periodic motion of the detection device based on the spatial pose information, filter the image frames in the stable motion stage from the two-dimensional image sequence, and generate a to-be-reconstructed image sequence. The image frames in the to-be-reconstructed image sequence are processed to extract the segmentation data of the target object. Based on the to-be-reconstructed image sequence, the spatial pose information and the segmentation data of the target object, an implicit neural representation model is trained, the three-dimensional space coordinates are input into the implicit neural representation model, the image information and semantic information corresponding to the three-dimensional space coordinates are generated, and the three-dimensional model of the target object is generated based on the image information and the semantic information. The present application accurately analyzes the periodic motion of the detection device and filters the image frames in the stable motion stage with high quality by using the spatial pose information, ensures the continuity and accuracy of the training data and the reconstruction data, and combines the implicit neural representation model to train and three-dimensionally reconstruct the stable frames, so that a high-precision and continuous three-dimensional model of the target object can be obtained without additional external registration or patient breathing cooperation, and the data utilization efficiency and the stability of the reconstruction effect are improved. The user terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The server terminal can be implemented by an independent server or a server cluster composed of multiple servers. The present application will be described in detail through specific embodiments.

[0032] Please refer to Figure 2 , Figure 2 The flowchart of an embodiment of the model generation method based on implicit neural representation provided by the present application is shown. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown.

[0033] As Figure 2 shown, the model generation method based on implicit neural representation provided by the present application includes the following steps:

[0034] S10, obtaining a two-dimensional image sequence containing a target object and spatial pose information corresponding to the image frames of the two-dimensional image sequence collected by a detection device;

[0035] In the embodiment, a sequence of two-dimensional images containing a target object acquired by a detection device and spatial pose information corresponding to the image frames of the sequence of two-dimensional images are obtained, involving multiple key contents. The detection device is usually an imaging device with high-frequency ultrasonic transducers, which can be integrated with a mechanical arm to reduce human interference. The sequence of two-dimensional images refers to a set of grayscale images obtained by the detection device in a continuous scanning process, arranged in chronological order, and each frame records the instantaneous echo intensity distribution of the imaging plane. The operation of obtaining the sequence of two-dimensional images needs to cover the target area in space and ensure that the frame rate is sufficient to describe continuous motion in time. The spatial pose information indicates the position and orientation of the detection device in three-dimensional space at the time each two-dimensional image is acquired, which can be described using six degrees of freedom displacement and rotation parameters, for example, obtained in real time by mechanical arm joint sensor. The spatial pose information must be accurately corresponding to the image frames, and each frame of image should be associated with the three-dimensional coordinates and attitude information corresponding to the time of acquisition to realize the binding of time and space. Combining the process of obtaining the sequence of two-dimensional images and the spatial pose information, the detection device can be controlled by applying a stable contact force, and the image frames and pose data streams can be recorded synchronously in continuous trajectory scanning, thereby ensuring the integrity and time consistency of the data.

[0036] To implement this process, the detection device can be installed at the end of the mechanical arm, and a force sensor is used to feedback and adjust the contact pressure to ensure stable contact with the surface of the human body. During the data acquisition stage, the detection device is triggered in real time to continuously acquire two-dimensional B-mode image frames at a certain frame rate, and at the same time, the spatial position and orientation of the mechanical arm end at each time are calculated through the joint angle encoder of the mechanical arm and the kinematics model. By timestamping the image frame acquisition time and the corresponding pose calculation result, after threshold checking time synchronization deviation, the two-dimensional image frames and their spatial pose information are packaged and encapsulated into one-to-one data units. During the acquisition process, the entire target area should be covered along the planned scanning trajectory, and whether the scanning is completed is determined by accumulating the length of the trajectory coordinates and the preset path coverage condition, and finally a complete sequence of two-dimensional images and their corresponding spatial pose information set are formed.

[0037] Example description: In practical applications, the KUKA LBR iiwa robot arm can be controlled to carry the Civco ultrasonic probe to scan the target area of the patient's abdomen along the planned path, and the joint angle data at the image acquisition time is recorded in real time. The acquisition time of each frame of image is time-stamped by the hardware clock of the detection device, and the joint angle data is read in real time from the robot arm controller interface, and the three-dimensional coordinates and attitude angle of the robot arm end at that time are solved based on the kinematics model of the robot arm. After binding the calculated position and attitude with the corresponding time stamp, a spatial pose data unit corresponding to each frame of two-dimensional B-mode image frame is formed. Combined with continuous trajectory scanning control, the complete coverage of the entire abdominal scanning area can be completed in a short time, and a complete two-dimensional image sequence and spatial pose information set is formed, ensuring the integrity and reliability of the data basis for subsequent processing links.

[0038] By associating and binding the two-dimensional image sequence obtained by the detection device with the real-time synchronized spatial pose information, the embodiment can effectively reduce the problem of image and spatial data asynchronization caused by unstable operator action or non-ideal robot arm movement, and ensure the spatial consistency and temporal continuity of the data. This process avoids the dependence on external optical tracking devices and improves the accurate correspondence between spatial positioning and imaging results, thereby providing a high-quality input data set for subsequent data screening, segmentation, training and three-dimensional reconstruction.

[0039] S20, based on the spatial pose information, analyzing the periodic motion of the detection device, and according to the analysis result of the periodic motion, screening out image frames in the preset stable motion stage from the two-dimensional image sequence to generate a to-be-reconstructed image sequence;

[0040] In the embodiment, the periodic motion of the detection device is analyzed based on the spatial pose information, which involves extracting dynamic data reflecting the respiratory motion from the spatial pose information. The spatial pose information usually includes the position coordinates of the detection device in three-dimensional space and its attitude angle, and the vertical displacement change is highly related to the patient's respiratory period. In order to accurately describe the periodic motion, the spatial pose information needs to be arranged in time sequence first, and the vertical displacement component is focused on.

[0041] Subsequently, the time series is smoothed by the moving average filtering method to reduce transient fluctuations and high-frequency interference, so that the periodic change is more clear. The smoothed curve can intuitively show the periodic up-and-down motion process, and the local maximum and minimum points can be identified by using the extreme point detection algorithm, which correspond to the end of inspiration and expiration times, respectively.

[0042] According to these extreme points, the interval between adjacent maximum points is defined as a breathing cycle, and further, the maximum fluctuation amplitude of displacement change in this interval is calculated. If the fluctuation amplitude is less than a preset threshold, the interval is marked as a stable motion stage. In this way, the stability of device motion can be dynamically determined from spatial pose information, and the time range of each stable interval can be accurately located.

[0043] On the basis of the known time range, the image frames with acquisition timestamps located in the stable interval can be correspondingly screened out in the two-dimensional image sequence. These image frames represent data collected in a state where the spatial pose is most stable and the relative position between the detection device and the target object remains consistent, and thus are aggregated as a to-be-reconstructed image sequence to improve subsequent data consistency and reconstruction accuracy.

[0044] One implementation is to record and associate the spatial pose information and the two-dimensional image sequence in real time during data acquisition. First, the vertical direction displacement component in the spatial pose information is extracted to form time series data. A sliding average filter with adjustable window length, for example, a window length of 10 frames, is applied to the time series to smooth the data and suppress transient noise. Then, the start and end points of each breathing cycle are determined by detecting the local extreme points of the curve. For each cycle, the displacement fluctuation range is calculated, and if it is less than a set threshold (such as 1 mm), the cycle is marked as a stable motion stage.

[0045] In each interval marked as stable, the two-dimensional image frames corresponding to their timestamps are extracted, and these image frames are aggregated into a continuous to-be-reconstructed image sequence after timestamp comparison. This aggregation process allows adaptive adjustment of threshold parameters, for example, a stricter fluctuation threshold can be used when the patient's breathing is shallow, and the threshold can be relaxed when the breathing is deep, to ensure that it can be applied to patients of different body types and different breathing patterns.

[0046] Example: In a robotic ultrasound scan of the abdominal region, the spatial pose information recorded by the mechanical arm during the acquisition process is synchronously associated with the two-dimensional B-mode image sequence. The vertical direction displacement is selected as the periodic motion indicator, and after being arranged in time sequence, a sliding average filter with a window length of 10 frames is used to smooth the sequence. The end of inspiration and expiration positions are determined by the extreme value detection method, and the fluctuation range between adjacent maximum values is calculated. When the fluctuation is less than 1 mm, it is marked as a stable motion stage. In these stable motion stages, the two-dimensional image frames corresponding to the timestamps are extracted as high-stability data, and are aggregated to form a continuous to-be-reconstructed image sequence input into subsequent training and three-dimensional reconstruction processing links, thereby improving the accuracy and consistency of model training and reconstruction results.

[0047] The embodiment reduces geometric distortion and blurred data caused by patient respiratory motion, probe jitter, etc. by analyzing periodic motion characteristics in spatial pose information and screening image frames corresponding to stable motion stages. The stable motion screening process can be completed using only spatial pose information, without relying on external respiratory monitoring devices or additional registration steps, thereby reducing system complexity and clinical implementation difficulty, improving data acquisition process standardization and result reliability.

[0048] S30, processing the image frames in the image sequence to be reconstructed to identify and extract the target object, and obtaining target object segmentation data corresponding to the image sequence to be reconstructed;

[0049] In the embodiment, the image frames in the image sequence to be reconstructed are processed to identify and extract the target object corresponding to the anatomical structure of interest in the image, and finally form the segmentation data corresponding to the image sequence to be reconstructed. The image sequence to be reconstructed refers to a set of two-dimensional image frames with high consistency and spatial stability after stable motion screening. The image frame is a two-dimensional gray image, usually a B-mode ultrasound image.

[0050] In the process of identifying the target object, each frame of image needs to be mapped to a representation domain suitable for processing by a segmentation algorithm, such as an intermediate representation domain, which is characterized by reducing the interference of speckle noise and insufficient contrast of ultrasound images through image domain conversion. This representation domain can be realized by a trained conversion network, which maps the original ultrasound image to a domain with high contrast and structural saliency. This conversion operation allows standardization of the pre-processing of images under different devices, different patients and different imaging conditions.

[0051] The identification process needs to run a semantic segmentation algorithm in the intermediate representation domain, such as a pixel-level classification network based on a deep convolutional neural network, to predict the class probability of each pixel in the image. The class probability prediction result is a semantic class probability map, which contains the probability value of each pixel belonging to the target object or the background. Through a post-processing step such as a morphological closing operation, small holes and boundary discontinuities in the segmentation result can be compensated for, making the segmentation region more complete. Calculating the connected domain information can further select the largest connected region as the effective target object region, avoiding the interference of discrete small regions to the result.

[0052] After obtaining the complete target object region, its contour boundary needs to be further extracted, and the class probability value of the boundary pixel is calculated to generate a boundary mask with confidence. This mask can be indexed and bound with the timestamp of each frame of image to form complete segmentation data in the time sequence. After performing the above processing on all image frames, the boundary mask set is integrated in the order of time sequence to form the target object segmentation data corresponding to the image sequence to be reconstructed.

[0053] The image sequence to be reconstructed can be input frame by frame into a pre-trained deep learning model. Each frame of image is first input into a specially trained conversion network, for example, a network based on an encoder-decoder structure, and the output is an intermediate representation domain image with a more balanced gray scale distribution and enhanced edge features. The converted image is then input into a semantic segmentation model to generate a corresponding semantic class probability map.

[0054] The semantic class probability map is processed by threshold setting and morphological operation to retain high confidence areas and fill in possible small gaps. Connected component analysis is used to identify the main region of the target object, and the largest connected component is marked as the target region. Then the contour curve of the region is extracted and the probability values of the corresponding pixels are recorded to form a boundary mask with confidence. Each frame of boundary mask is stored in association with its acquisition timestamp to form a segmented data set sorted in time sequence. According to specific requirements, the input resolution of the segmentation model, the network depth or different loss functions can be adjusted to adapt to the image characteristics of different anatomical sites.

[0055] Example: In the abdominal aorta ultrasound image processing scenario, two-dimensional B-mode image frames that have been screened for stable motion are input frame by frame into a pre-trained image conversion network to obtain intermediate representation domain images that enhance the contrast between blood vessels and surrounding tissues. A deep learning segmentation network is run on the converted images to output the probability values of each pixel being a blood vessel or not. By selecting regions with probability values above a certain threshold and performing a closing operation, a complete blood vessel segmentation mask is obtained. Connected component analysis helps to select the largest blood vessel region, extract the boundary of the region, and combine the probability values to form a confidence mask. The mask is stored after being bound to the image frame timestamp in a sequence to form the target object segmentation data, laying a solid foundation for accurate two-dimensional structural information for subsequent high-precision three-dimensional model reconstruction.

[0056] This embodiment improves segmentation consistency and accuracy by performing target object recognition and extraction on the image sequence to be reconstructed frame by frame, reducing segmentation errors caused by changes in image quality and differences in detection angle. Through standardized processing of the intermediate representation domain and the high robustness of the deep segmentation model, the adaptability and universality of automated segmentation are improved. The introduction of post-processing steps enhances the completeness and boundary continuity of the segmented region, ensuring that the target object segmentation data generated can be used as an important input for subsequent training of implicit neural representation models and three-dimensional reconstruction, improving the overall data quality and stability of the processing flow.

[0057] S40, training an implicit neural representation model based on the image sequence to be reconstructed, the spatial pose information, and the target object segmentation data;

[0058] In this embodiment, the goal of training the implicit neural representation model is to enable the model to predict image information and semantic information at arbitrary three-dimensional positions by establishing a continuous mapping from three-dimensional spatial coordinates to grayscale values and semantic label probabilities. The input data for training includes a sequence of images to be reconstructed, spatial pose information, and target object segmentation data. The sequence of images to be reconstructed is a set of stable image frames selected to have low motion artifacts. The spatial pose information provides the spatial position and orientation of the detection device at the time of each image capture, which is used to align the image frames with the three-dimensional spatial coordinates. The target object segmentation data contains pixel labels or confidence information for the target object region in each image frame, serving as supervision data during the training process.

[0059] The training process first needs to determine the training range. The three-dimensional spatial bounding box of the target object can be resolved using the spatial pose information and the target object segmentation data, and a set of three-dimensional coordinate points can be uniformly sampled within the bounding box to ensure that the training samples cover the entire target region and the background space. For each sampling point, its three-dimensional spatial coordinates are converted into a high-dimensional feature vector through the sine function positional encoding method to enhance the model's ability to express high-frequency details. Positional encoding ensures that the model can capture boundary and texture features at a small scale by performing multi-scale sine and cosine transformations on the coordinates.

[0060] The high-dimensional feature input multi-layer perceptron neural network serves as the input end of the implicit neural representation model, and the model output includes predicted voxel grayscale values and predicted semantic label probabilities. The supervision signals for training come from the two-dimensional image frame grayscale values and target object segmentation labels corresponding to the three-dimensional coordinates. By resolving the image frame corresponding to the three-dimensional sampling point and the timestamp from the spatial pose information, the corresponding grayscale values and labels are obtained as the supervision true values.

[0061] A joint loss function is defined during the training process to jointly optimize the predicted voxel grayscale values and the predicted semantic label probabilities. The reconstruction error is used to measure the difference between the model's predicted grayscale values and the true image grayscale values, and the cross-entropy error is used to measure the matching degree of the predicted semantic label probabilities and the segmentation labels. The joint loss function integrates these two parts of the error with weights as the optimization target. The model weight parameters are continuously updated through the backpropagation algorithm and the gradient descent method to optimize the model performance. When the fluctuation of the loss function is below a certain threshold, the training is terminated, and the trained implicit neural representation model is obtained.

[0062] The three-dimensional training area can be determined based on the spatial pose information and the target object segmentation data, and coordinate points can be sampled at a fixed interval in the area. The sampling density can be adjusted to adapt to different reconstruction resolution requirements. The sampled coordinates are encoded by a multi-scale sine function. The number of frequency series and the range can be adjusted according to the structural complexity of the target object, for example, 10 frequency series are used to ensure sufficient spatial resolution. The multi-layer perception neural network can be set to a four-layer structure, each layer has 256 neurons, and a sine activation function is used. The prediction end outputs two channels, corresponding to the gray value and the semantic label probability respectively.

[0063] The supervised data is projected into the two-dimensional B-mode image frame corresponding to the time stamp through the spatial pose information, the gray value of the pixel at the position is extracted, and the label is extracted from the target object segmentation data. The loss function can be defined as the mean square error between the predicted gray value and the true value plus the cross-entropy loss between the predicted semantic label probability and the label. The weighting coefficient can be set according to experimental requirements. The training process uses the Adam optimization algorithm, and the learning rate can be set to 0.001, which is dynamically adjusted according to the convergence curve, and the training is stopped when the loss value changes less than 0.001.

[0064] Example: In the abdominal aorta ultrasound reconstruction scene, the three-dimensional spatial range covering the abdominal aorta is calculated through the spatial pose information and the segmentation data, and the three-dimensional coordinates are sampled at an interval of 0.5mm in the range. Each sampled coordinate is input into a four-layer multi-layer perception network after being encoded by a sine function, and the voxel gray value and the probability of belonging to a blood vessel at the coordinate are predicted and output. During training, the sampled coordinates are projected onto the corresponding two-dimensional B-mode image frame calculated from the spatial pose information, and the gray value is collected as the true value, and the target object segmentation data is used as semantic supervision. A joint loss function is used for optimization, and after training, the model can predict the gray value and semantic probability of the blood vessel at any three-dimensional coordinate position, realizing the expression and reconstruction of continuous three-dimensional space.

[0065] The present embodiment realizes the continuous mapping capability from three-dimensional space coordinates to image gray value and semantic label probability by training an implicit neural representation model jointly using an image sequence to be reconstructed, spatial pose information and target object segmentation data. This method enhances the spatial expression capability of the model through high-dimensional position encoding, improves the reconstruction accuracy and semantic consistency in the presence of respiratory motion and non-rigid deformation, and reduces geometric mismatch caused by motion artifacts between frames. Supervised training ensures that the model not only has accurate gray reconstruction capability, but also maintains semantic recognition capability for anatomical structures, laying the foundation for subsequent continuous three-dimensional model generation.

[0066] S50, inputting three-dimensional space coordinates in a preset three-dimensional space range into the implicit neural representation model to generate image information and semantic information corresponding to the three-dimensional space coordinates;

[0067] In this embodiment, the preset three-dimensional space range is a three-dimensional space region defined by combining the spatial pose information of the detection device and the target object segmentation data, which can cover the target object and its adjacent range to ensure the integrity and accuracy of subsequent prediction. The three-dimensional space coordinates refer to sampling points uniformly distributed in the preset three-dimensional space range, each coordinate corresponding to a unique spatial position, serving as a sampling reference for image reconstruction in space. The implicit neural representation model is a multi-layer perceptron network that obtains a continuous mapping relationship from three-dimensional space coordinates to image gray value and semantic label probability through training.

[0068] In actual processing, all three-dimensional space coordinates in the preset three-dimensional space range are input into the trained implicit neural representation model in sequence. The model performs forward calculation on each coordinate position, and outputs the predicted voxel gray value and the predicted semantic label probability. All output results are summarized according to the coordinate positions to form an output set corresponding to the input coordinates.

[0069] The output predicted voxel gray value constitutes the gray scale information of the continuous space, which is used to express the image intensity distribution of the target object in the three-dimensional space. The predicted semantic label probability expresses the probability that the coordinate position belongs to the target object, which is used to reflect the distribution range of the anatomical structure in the space. Through this process, the model can generate image information and semantic information consistent with the spatial position at any position in the preset three-dimensional space range, supporting subsequent three-dimensional reconstruction and analysis tasks.

[0070] The boundary box of the preset three-dimensional space range can be defined first, and a uniform three-dimensional coordinate grid can be generated at a fixed interval, for example, covering the entire range at an interval of 0.5 mm. The same sine function position encoding operation as in the training phase is performed on each three-dimensional coordinate to ensure the consistency of the input feature distribution with the training. The encoded high-dimensional features are input into the implicit neural representation model one by one, and the model performs forward inference on each input coordinate to output the predicted voxel gray value and the predicted semantic label probability at that position. All prediction results are rearranged into a data structure corresponding to the three-dimensional coordinates to obtain continuous image information and semantic information.

[0071] In implementation, multiple coordinate inputs can be processed in batches to fully utilize the GPU parallel computing capability and speed up the inference. After prediction, the results can be stored in a three-dimensional data format, such as a multi-dimensional array or a voxel grid, for direct use by the subsequent three-dimensional model generation module.

[0072] Example description: In the abdominal aorta three-dimensional reconstruction scene, first generate a uniform three-dimensional coordinate grid with an interval of 0.5 mm within the spatial range defined by the spatial pose information and the segmentation data. Map each coordinate to a high-dimensional feature input implicit neural representation model through a sine function position encoding. After the model performs forward inference on all coordinates, it outputs the predicted voxel gray value and semantic probability value of each coordinate position. Finally, a three-dimensional gray body data and semantic probability body data covering the abdominal region are formed, providing basic data for subsequent abdominal aorta geometric model reconstruction and analysis.

[0073] By using the trained implicit neural representation model to predict all three-dimensional space coordinates within a predetermined three-dimensional space range, the embodiment can realize high-fidelity image information and semantic information reconstruction from sparse two-dimensional image sequences to continuous three-dimensional space. This process does not rely on external registration systems or auxiliary data, fully utilizes the gray and semantic statistical characteristics captured during the training process, and consistently predicts any spatial position, effectively solving the problems of inter-frame position error and data discontinuity, improving the integrity and continuity of the reconstructed data, and ensuring the accuracy of subsequent three-dimensional model generation.

[0074] S60, generating a three-dimensional model of the target object according to the image information and semantic information.

[0075] In this embodiment, the image information represents the gray value distribution of each sampling point in the continuous three-dimensional space predicted by the implicit neural representation model, which reflects the image intensity characteristics of the target object in the three-dimensional space. The semantic information represents the probability of the corresponding sampling point belonging to the target object, which reflects the spatial distribution of the target object in space. The combination of the two provides a double constraint for the accurate description of the target object in the three-dimensional space.

[0076] In the processing process, first, the spatial position correspondence relationship of the image information and the semantic information needs to be analyzed, and the spatial points are filtered one by one within the entire reconstruction space range. According to the probability value in the semantic information, it is judged whether it belongs to the target object, which can be used as the only basis for judgment, or it can be jointly judged with the gray value in the image information to improve the robustness of the judgment. For the set of spatial points that meet the conditions, form a dense three-dimensional point cloud data to represent the spatial geometric distribution of the target object.

[0077] In order to recover the continuous geometric surface from the point cloud data, the normal vector of the point cloud data is estimated, and the main direction is fitted through the local neighborhood in space to obtain the normal information of each point. Then, using the Poisson surface reconstruction algorithm, the closed, continuous and smooth surface mesh model is recovered according to the spatial position and normal vector data of the points. This surface mesh model is further smoothed by Laplace smoothing to further improve the surface smoothness and visual continuity, and finally a high-quality three-dimensional model suitable for three-dimensional analysis and measurement is obtained.

[0078] The spatial points with probability values greater than a preset threshold (e.g., 0.5) can be extracted from the semantic information to form a basic point cloud dataset. In a high-noise scene, the spatial points with a gray value lower than a certain threshold can be further removed in combination with the image information. The k-nearest neighbor or radius search method is used on the point cloud dataset, and the covariance matrix of the local neighborhood of each point is calculated to extract the principal eigenvector as the normal vector of the point.

[0079] The calculated point cloud data and normal vector are used as inputs to calculate a multi-resolution implicit function using the Poisson reconstruction algorithm, and the isosurface is extracted as a triangular mesh model. The triangular mesh model obtained is subjected to Laplace smoothing processing to eliminate small noise and improve the continuity of the surface, so that the model has consistent boundary curvature and surface quality in vision. The final output model can be saved in a standard three-dimensional format, such as an STL or OBJ file, for clinical analysis and visualization.

[0080] Example: In the task of robot-assisted three-dimensional reconstruction of the abdominal aorta, the gray-scale image information and semantic probability information predicted by the implicit neural representation model are analyzed. The spatial points with a semantic probability greater than 0.5 are extracted as abdominal aorta point cloud data in three-dimensional space, and the points with too low gray values are filtered out in combination with the gray information to reduce noise. The principal component analysis normal vector estimation is performed on the obtained point cloud data, and the Poisson reconstruction algorithm is input to generate a closed three-dimensional curved mesh of the abdominal aorta. Finally, through the Laplace smoothing operation, the model surface boundary is smooth and continuous, generating a fine abdominal aorta three-dimensional model with high fidelity and meeting the requirements of three-dimensional visualization and clinical measurement.

[0081] The present embodiment generates a three-dimensional model of a target object by combining image information and semantic information, which can accurately identify the spatial distribution of the target object while preserving the details of the image gray intensity, improving the reconstruction geometry and texture quality. Through the processing processes of point cloud screening, normal estimation, Poisson reconstruction, and Laplace smoothing, the three-dimensional model obtained has high resolution, high continuity, and low noise characteristics, effectively avoiding the geometric distortion caused by inter-frame offset and motion artifacts in traditional reconstruction methods, and ensuring the true reproduction of the geometric shape of the target object in three-dimensional space.

[0082] The application relates to the technical field of ultrasonic detection and discloses a model generation method and device based on implicit neural representation, equipment and a medium, which comprises the following steps: acquiring a two-dimensional image sequence collected by a detection device and spatial pose information corresponding to image frames, analyzing periodic motion of the detection device based on the spatial pose information, screening image frames in a stable motion stage from the two-dimensional image sequence, and generating a to-be-reconstructed image sequence. Image frames in the to-be-reconstructed image sequence are processed, and segmentation data of a target object is extracted. An implicit neural representation model is trained based on the to-be-reconstructed image sequence, the spatial pose information and the segmentation data of the target object, three-dimensional space coordinates are input into the implicit neural representation model, image information and semantic information corresponding to the three-dimensional space coordinates are generated, and a three-dimensional model of the target object is generated based on the image information and the semantic information. According to the application, the periodic motion of the detection device is accurately analyzed by using the spatial pose information, and image frames in a high-quality stable motion stage are screened, so that the continuity and accuracy of training data and reconstruction data are ensured. The stable frames are trained and three-dimensionally reconstructed in combination with the implicit neural representation model, a high-precision and continuous three-dimensional model of the target object can be obtained without additional external registration or patient breathing cooperation, and the data utilization efficiency and the stability of the reconstruction effect are improved.

[0083] In one embodiment, the above step S10 comprises:

[0084] S101, controlling the end effector of the mechanical arm to carry the detection device to establish physical contact with the surface of the target region, and monitoring the contact pressure between the detection device and the surface of the target region in real time, dynamically adjusting the posture of the end effector of the mechanical arm, maintaining the component of the contact pressure between the detection device and the surface of the target region in the normal direction of the contact surface within a preset threshold range, and generating a device stable contact state flag;

[0085] S102, when the device stable contact state flag is activated, triggering the detection device to continuously collect two-dimensional B-mode image frames containing the target object at a preset scanning speed, and transmitting the two-dimensional B-mode image frames to a processing terminal in real time through a video interface;

[0086] S103, when each two-dimensional B-mode image frame is collected, a time stamp is intercepted through a hardware clock of the detection device, joint angle values are obtained by analyzing a joint encoder data stream of the mechanical arm in real time, three-dimensional coordinates and posture angles of the end effector of the mechanical arm at corresponding time stamps are calculated based on the joint angle values and kinematic parameters of the mechanical arm, the three-dimensional coordinates, the posture angles and the time stamps of the end effector of the mechanical arm are bound, and spatial pose information marked with the time stamps is generated;

[0087] S104, each two-dimensional B-mode image frame is bound with the spatial pose information marked with the time stamps corresponding to the time when each two-dimensional B-mode image frame is collected, and when the time stamp deviation is less than a tolerance threshold, the two-dimensional B-mode image frame and the spatial pose information are packaged as a space-time aligned data unit and stored in a cache database.

[0088] S105, repeatedly performing image acquisition and transmission, pose synchronization calculation and space-time binding operation along the preset trajectory in the device stable contact state, terminating the operation through trajectory coordinate accumulation to determine that the scanning path completely covers the target area, and outputting a two-dimensional image sequence composed of space-time aligned data units containing the target object and a set of spatial pose information corresponding to the image frames of the two-dimensional image sequence.

[0089] In this embodiment, the mechanical arm end effector carrying the detection device establishes physical contact with the target area surface is a necessary condition to ensure that the detection device and the patient surface are closely attached, in order to ensure the stability and effectiveness of the subsequent image acquisition. In this process, real-time monitoring of contact pressure is completed by force sensors integrated in the detection device or the mechanical arm end effector, which acquires the pressure component in the normal direction of the contact surface between the detection device and the target area surface, and compares it with the preset threshold range in real time. Dynamic adjustment of the mechanical arm end effector posture is to use the multi-degree-of-freedom adjustment capability of the mechanical arm to adjust the position and posture of the probe in space according to the real-time feedback pressure data, so that the contact pressure is stabilized within a reasonable range set, avoiding weak image signals due to too light contact, or affecting the imaging effect due to tissue compression caused by too heavy contact. The device stable contact state flag is generated after the pressure is monitored to be stable, which is used as a condition for starting subsequent image acquisition.

[0090] When the device stable contact state flag is activated, the detection device works continuously according to the set scanning speed, and the B-mode image frame data is transmitted in real time to the processing terminal through the video interface for caching and analysis, ensuring the real-time performance of image data acquisition and processing. When each frame of image is acquired, the hardware clock of the detection device is used as a unified time reference to accurately record the current timestamp, and the angle data output by the mechanical arm joint encoder is analyzed in real time. Based on the acquired angle data and the kinematic parameters of the mechanical arm, the three-dimensional spatial position and attitude angle information of the mechanical arm end effector are calculated. The pose information is bound by timestamp to establish an accurate space-time correspondence with the currently acquired image frame, ensuring that each image frame can describe the spatial geometric information at the time of acquisition through its corresponding spatial pose.

[0091] After the binding is completed, the timestamp deviation needs to be checked to ensure that the synchronization accuracy of image frame acquisition and pose solving is within the tolerance threshold. Qualified data is packaged as a space-time aligned data unit and stored as a basis for subsequent reconstruction data. When the detection device is in a stable contact state, the above acquisition, transmission, solving, binding and storage operations are automatically repeated along the set trajectory. Path coverage is determined by real-time accumulation of trajectory coordinates to confirm whether the scanning path completely covers the target area. Only after complete coverage, the entire acquisition process is terminated, and a dataset composed of multiple space-time aligned data units is finally output, including a continuous two-dimensional B-mode image sequence and a strictly corresponding spatial pose information set, which is used for subsequent motion compensation and three-dimensional reconstruction.

[0092] The embodiment ensures the contact stability of the detection device and the human body surface during image acquisition by real-time monitoring and dynamic adjustment of the contact pressure of the detection device and the surface of the target area, and reduces image distortion caused by contact pressure fluctuations. By synchronously collecting spatial pose information through joint encoders of the mechanical arm and a hardware clock, combining timestamp binding and tolerance checking, the two-dimensional image frame is accurately corresponding to the spatial geometric state at the time of acquisition. The accumulation of path coordinates ensures complete coverage of the target area during scanning, effectively avoiding missing image data due to path omission. The final output of the space-time aligned data unit set provides standardized, continuous and complete data input for subsequent motion compensation processing, semantic segmentation and three-dimensional reconstruction, improving spatial geometric consistency and data utilization efficiency.

[0093] In one embodiment, the above step S20 comprises:

[0094] S201, extracting a displacement component of the detection device in the vertical direction from the spatial pose information;

[0095] S202, performing time window sliding average filtering on the displacement component to generate a periodic respiratory waveform;

[0096] S203, detecting local maximum points and local minimum points of the periodic respiratory waveform;

[0097] S204, marking a platform interval between adjacent local maximum points that satisfies a displacement fluctuation threshold as a preset stable motion phase;

[0098] S205, extracting a two-dimensional image frame corresponding to a timestamp when the periodic respiratory waveform is in the preset stable motion phase;

[0099] S206, aggregating all two-dimensional image frames in the preset stable motion phase to generate a to-be-reconstructed image sequence.

[0100] In the embodiment, the spatial pose information includes the coordinates and attitude angle information of the detection device in three-dimensional space. The extraction of the vertical displacement component refers to selecting the displacement value in the normal direction relative to the reference plane in the spatial pose data. The direction is usually perpendicular to the human body surface and reflects the up-and-down motion trajectory of the detection device over time. The purpose of extracting this component is to capture the periodic motion law caused by breathing and provide basic data for subsequent motion state analysis.

[0101] The time window moving average filter is used to smooth the vertical displacement sequence. By defining a fixed length time window, the average value of the displacement values in the window is calculated. The sliding window is pushed forward point by point on the time axis to generate a smoothed time sequence curve, called the periodic breathing waveform. The waveform reflects the periodic motion change of the detection device caused by the patient's breathing, which helps to distinguish the stable motion phase.

[0102] The local maximum and minimum points of the periodic breathing waveform are detected by using the first-order difference or neighborhood comparison method. The points on the waveform curve that are significantly higher or lower than the data points in the neighborhood are marked as local maximum and local minimum, respectively. This detection process clearly identifies the peak and valley positions of the breathing motion, providing time anchor points for segment analysis and platform interval determination.

[0103] The platform interval between adjacent local maximum points that satisfies the displacement fluctuation threshold is marked as the preset stable motion phase. This is achieved by calculating the maximum and minimum difference of all displacement values in the interval. If the difference is lower than the preset threshold, it is considered that the motion in the interval is small and stable. This discrimination criterion selects the relatively stable section of the breathing motion, i.e., the time period with less image acquisition jitter and minimal device motion.

[0104] When the periodic breathing waveform is in the preset stable motion phase, the two-dimensional image frames collected in this time period are extracted based on the correspondence of the time stamp, ensuring that the selected image frames correspond to the time when the device is in a stable state. All image frames that meet the conditions are aggregated to form a sequence of images to be reconstructed, which serves as the input for subsequent training and three-dimensional reconstruction. The entire process is based on accurate analysis of time series spatial data, ensuring the continuity and stability of the data in space and time.

[0105] The embodiment effectively captures the periodic motion law of the detection device affected by breathing by extracting and smoothing the vertical displacement component in the spatial pose information. By using local extreme point detection and fluctuation threshold determination, the time interval when the device is in a stable motion state is accurately selected. The two-dimensional image frames in this stage are extracted to generate a sequence of images to be reconstructed, reducing image blur and spatial misalignment caused by device jitter or breathing interference, and improving the spatiotemporal consistency and accuracy of subsequent training and three-dimensional model reconstruction.

[0106] In one embodiment, the step 30 above comprises:

[0107] S301, input each two-dimensional B-mode image frame in the image sequence to be reconstructed into a pre-trained conversion network, and map the two-dimensional B-mode image frame to an intermediate representation domain through the conversion network;

[0108] S302, run a semantic segmentation model in the intermediate representation domain to generate a semantic class probability map for each pixel;

[0109] S303, perform a morphological closing operation on the semantic class probability map to generate a probability map after closing operation processing;

[0110] S304, calculate the maximum connected domain of the probability map after closing operation processing, and mark the maximum connected domain as a complete target object region;

[0111] S305, extract the contour boundary of the complete target object region, and calculate the class probability value of the contour boundary pixel to generate a boundary mask with probability confidence;

[0112] S306, establish an index mapping table of the boundary mask and the image frame timestamp;

[0113] S307, integrate all boundary masks in time sequence to generate target object segmentation data corresponding to the image sequence to be reconstructed.

[0114] In this embodiment, each two-dimensional B-mode image frame in the image sequence to be reconstructed is input into a pre-trained conversion network. The core of this step is to map the original grayscale ultrasound image to an intermediate representation domain which is more suitable for the semantic segmentation model. The conversion network generally adopts a convolutional neural network architecture, and the pre-trained model can be derived from a public data set or trained on related tasks in an unsupervised or self-supervised manner to enhance the adaptability to the characteristics of low contrast, noise, etc. of the ultrasound image. The intermediate representation domain refers to a high-dimensional feature space introduced between the original image and the semantic label. In this space, the boundary and structure information of the target object is clearer, thereby improving the robustness and accuracy of the downstream segmentation task.

[0115] Running a semantic segmentation model in the intermediate representation domain means that each pixel in the intermediate representation domain image is classified based on a deep convolutional neural network, and the probability value of each pixel belonging to each semantic class is output to form a semantic class probability map. The semantic segmentation model can adopt an encoder-decoder architecture, such as the U-Net structure, to extract context information in a multi-scale feature fusion manner, which is particularly suitable for handling the problems of texture blur and boundary blur in ultrasound images.

[0116] A morphological closing operation is performed on the semantic class probability map, that is, a combination operation of expansion and then corrosion, to effectively fill small holes and connect adjacent but discontinuous target regions, thereby improving the completeness and connectivity of the segmentation result. The operation usually uses a structure element, such as a rectangular or elliptical kernel, and the kernel size is configured according to the typical morphological parameters of the target object.

[0117] Calculating the largest connected domain of the probability map after the closing operation processing means determining the connectivity between pixels in the binary image and selecting the largest connected region as the target region, so as to avoid mistaking a pseudo-target or a small noise region in the image as the result. The largest connected domain is marked as a complete target object region, so that the subsequent steps focus on the segmentation boundary of the real anatomical structure.

[0118] The contour boundary of the complete target object region is extracted, the external boundary of the region is determined through an edge detection algorithm, and the class probability value of each pixel point on the boundary is further calculated to generate a boundary mask with a probability confidence. The boundary mask not only marks the shape contour of the target object, but also provides the classification confidence of each contour point, which is used for subsequent optimization and uncertainty analysis.

[0119] An index mapping table of the boundary mask and the image frame timestamp is established to bind the boundary mask with its corresponding acquisition time, ensuring the timing consistency and providing a precise time and space matching basis for subsequent multi-frame fusion and three-dimensional reconstruction. All boundary masks are integrated in time sequence to generate target object segmentation data corresponding to the image sequence to be reconstructed, forming an ordered, continuous, and spatiotemporally aligned segmentation result sequence, which serves as an important input for subsequent implicit neural network training and three-dimensional reconstruction.

[0120] The embodiment maps two-dimensional B-mode image frames to an intermediate representation domain using a conversion network, combines a semantic segmentation model for fine pixel-by-pixel classification of the target object, and then performs post-processing steps such as morphological closing operation, maximum connected domain extraction, and boundary confidence calculation, effectively overcoming problems such as noise, blurred boundaries, and uneven gray scale in ultrasound images, improving the accuracy and continuity of the target object segmentation data, and providing high-quality spatiotemporally consistent input data for subsequent training of an implicit neural representation model and three-dimensional reconstruction.

[0121] In one embodiment, the above step S40 includes:

[0122] S401, based on the spatial pose information and the target object segmentation data, analyzing a target object space bounding box, and generating a uniform three-dimensional coordinate point set within the target object space bounding box;

[0123] S402, constructing a multi-layer perception network as an implicit neural representation model;

[0124] S403, performing a sine function position encoding on the three-dimensional spatial coordinate point set to generate a high-dimensional position encoding feature;

[0125] S404, inputting the high-dimensional position encoding feature into an implicit neural representation model to output a predicted voxel gray value and a predicted semantic label probability;

[0126] S405, extracting a two-dimensional B-mode image frame corresponding to the timestamp in the image sequence to be reconstructed, and defining the extracted two-dimensional B-mode image frame as a predicted voxel gray value supervision true value;

[0127] S406, extracting target object segmentation data corresponding to the timestamp, and defining the extracted target object segmentation data as a predicted semantic label probability supervision true value;

[0128] S407, calculating the reconstruction error between the predicted voxel gray value and the predicted voxel gray value supervision true value, and calculating the cross-entropy error between the predicted semantic label probability and the predicted semantic label probability supervision true value;

[0129] S408, weighted summing the reconstruction error and the cross-entropy error to form a joint loss value;

[0130] S409, updating the network weight parameters of the implicit neural representation model based on the joint loss value through a back propagation algorithm;

[0131] S410, repeating the prediction and parameter updating operations until the joint loss function fluctuation is less than a convergence threshold, obtaining a trained implicit neural representation model.

[0132] In this embodiment, the implicit neural representation model is trained based on the image sequence to be reconstructed, spatial pose information, and target object segmentation data. First, the range of the target object in three-dimensional space needs to be analyzed. Using the spatial pose information and the target object segmentation data, the spatial bounding box of the target object in the imaging space is determined, which refers to the smallest cuboid region that completely covers the target object, serving as the boundary range for subsequent three-dimensional coordinate sampling. Within the bounding box, a uniformly distributed three-dimensional coordinate point set is generated to ensure that the training data covers all regions of the target object. The spatial point distribution can use regular grid or variable density sampling to adapt to the object shape.

[0133] A multi-layer perceptron network is constructed as the implicit neural representation model. The multi-layer perceptron is a feedforward neural network that usually includes multiple fully connected layers and nonlinear activation functions, and has the ability to express the relationship between spatial coordinates and target physical properties, and can fit the mapping relationship from any three-dimensional spatial coordinates to the gray value and semantic probability at that position.

[0134] In the input stage, a sine function position encoding operation is performed on the three-dimensional spatial coordinate point set to convert low-dimensional coordinates into high-dimensional features. Position encoding enables the model to capture high-frequency details, such as fine edges and texture information in an image, by applying sine and cosine functions of different frequencies to coordinate components. The generated high-dimensional position encoding features are provided as input to a multi-layer perceptron network.

[0135] The model output predicts voxel intensity values and semantic label probabilities. The predicted voxel intensity values correspond to the intensity information of the three-dimensional spatial coordinate points, and the predicted semantic label probabilities represent the probability distribution of each coordinate point belonging to different semantic categories. To supervise the model output, a two-dimensional B-mode image frame consistent with the timestamp of the training coordinate points is extracted from the image sequence to be reconstructed as the supervised true value, and the target object segmentation data corresponding to the timestamp is extracted as the supervised true value of the predicted semantic label probability. The definition of the supervised true value ensures the spatio-temporal consistency between the training samples and the model input spatial points.

[0136] The construction of the loss function includes two parts: calculating the reconstruction error between the predicted voxel intensity values and the supervised true values, using mean square error as the measurement standard; calculating the cross-entropy error between the predicted semantic label probabilities and the segmentation supervised true values, measuring the fitting degree of the model to the classification. By combining the two errors through weighted summation, a joint loss value is formed, and the weight coefficient can be adjusted according to the demand for reconstruction and segmentation accuracy of the task.

[0137] During training, the model network weight parameters are updated based on the joint loss value through the backpropagation algorithm. Each training round performs a complete forward prediction, loss calculation, and weight update operation, and monitors the fluctuation of the joint loss function during training. When the loss value of consecutive training rounds fluctuates less than the preset convergence threshold, the training is stopped and the trained implicit neural representation model is output, ensuring that the model converges stably and has good generalization ability.

[0138] This embodiment establishes a continuous mapping relationship from three-dimensional spatial position to voxel intensity and semantic label probability by analyzing the spatial bounding box of the target object and uniformly sampling three-dimensional spatial coordinate points inside it, combining sine function position encoding and multi-layer perceptron network input. The training process optimizes the predicted values through joint supervision signals for multi-objective optimization, enabling the model to effectively express the geometric details and semantic attributes of the target object and generate continuous and consistent reconstruction results in three-dimensional space, improving training stability and reconstruction accuracy.

[0139] In one embodiment, the above step S50 includes:

[0140] S501, based on the spatial pose information and the target object segmentation data, analyzing the maximum boundary of the scanning trajectory, and combining the anatomical structure space to generate a preset three-dimensional spatial range;

[0141] S502, performing a uniform grid sampling operation in the preset three-dimensional space range to generate reconstructed target three-dimensional space coordinates;

[0142] S503, performing a sine function position encoding operation on the reconstructed target three-dimensional space coordinates to generate reconstructed target high-dimensional position encoding features;

[0143] S504, inputting the reconstructed target high-dimensional position encoding features into the implicit neural representation model;

[0144] S505, outputting, by the implicit neural representation model, a reconstructed target predicted voxel gray value and a reconstructed target predicted semantic label probability corresponding to the reconstructed target three-dimensional space coordinates;

[0145] S506, generating a reconstructed target three-dimensional gray field as image information according to the reconstructed target predicted voxel gray value;

[0146] S507, generating a reconstructed target three-dimensional probability field as semantic information according to the reconstructed target predicted semantic label probability;

[0147] S508, combining the reconstructed target three-dimensional gray field and the reconstructed target three-dimensional probability field to form a three-dimensional reconstruction data set covering the preset three-dimensional space range.

[0148] In the present embodiment, in the process of inputting the three-dimensional space coordinates in the preset three-dimensional space range into the implicit neural representation model, first, the maximum boundary of the scanning trajectory is analyzed by using the spatial pose information and the target object segmentation data. The maximum boundary of the scanning trajectory refers to the circumscribed bounding box formed by the movement path of the detection device in space. By combining the anatomical structure range in the segmentation data, the maximum boundary is limited to ensure that the spatial range covers the complete target object and excludes irrelevant areas, thereby generating the preset three-dimensional space range as the spatial boundary condition for three-dimensional reconstruction.

[0149] The uniform grid sampling operation is performed in the preset space range to construct a three-dimensional regular point set. The uniform grid sampling ensures the consistent distribution density of points in space, which helps to ensure the spatial uniformity and resolution consistency of the subsequent reconstruction results. The point set obtained by sampling is the reconstructed target three-dimensional space coordinates, which is a set of points corresponding to the spatial distribution in the training phase.

[0150] Subsequently, the sine function position encoding operation is performed on the reconstructed target three-dimensional space coordinates. By performing sine and cosine transformation on the coordinate values at different frequencies, the expression ability of the model for high-frequency details of space is enhanced, especially in target regions containing complex structure boundaries. After position encoding processing, the reconstructed target high-dimensional position encoding features are generated. These features contain multi-scale information of the space coordinates.

[0151] The generated high-dimensional position encoding features are input into an implicit neural representation model. The model performs forward calculation on each input feature according to the trained parameters, and outputs predicted voxel grayscale values and predicted semantic label probabilities corresponding to the reconstruction target three-dimensional space coordinates. The former represents the grayscale information of the space point, and the latter represents the probability distribution of the space point belonging to different anatomical structures.

[0152] According to the predicted voxel grayscale values, a reconstruction target three-dimensional grayscale field is generated in the entire preset space range. The three-dimensional grayscale field is formed by the corresponding collection of all predicted grayscale values in three-dimensional coordinates, and is output as image information to express the texture and intensity distribution of the reconstruction result. According to the predicted semantic label probabilities, a reconstruction target three-dimensional probability field is also generated in the three-dimensional space range. The probability field reflects the attribution probability of each space point to the anatomical semantics, and is output as semantic information.

[0153] Finally, the reconstruction target three-dimensional grayscale field and the reconstruction target three-dimensional probability field are combined to form a three-dimensional reconstruction data set covering the preset three-dimensional space range. The data set contains the grayscale value and semantic label probability of each space point, supporting subsequent three-dimensional model reconstruction and visualization analysis.

[0154] The embodiment jointly limits the space range by using the spatial pose information and the target object segmentation data, combines uniform grid sampling, sinusoidal function position encoding, and the forward prediction mechanism of the implicit neural representation model, continuously generates image information and semantic information covering the entire preset range in the three-dimensional space, and can effectively solve the problems of uneven scanning path, inconsistent resolution, and incomplete local space information, thereby improving the spatial consistency, texture detail expression ability, and semantic distribution integrity of the three-dimensional reconstruction data.

[0155] In one embodiment, the above step S60 includes:

[0156] S601, extracting a reconstruction target predicted semantic label probability set in the reconstruction target three-dimensional space range whose probability value in the semantic information is greater than a preset probability threshold;

[0157] S602, determining whether to extract a reconstruction target predicted voxel grayscale value set in the reconstruction target three-dimensional space range whose grayscale value in the image information is greater than a preset grayscale threshold;

[0158] S603, when the reconstruction target predicted voxel grayscale value set is extracted, combining the reconstruction target predicted voxel grayscale value set and the reconstruction target predicted semantic label probability set to mark a reconstruction target three-dimensional space coordinate set belonging to the target object;

[0159] S604, when the reconstruction target predicted voxel grayscale value set is not extracted, marking a reconstruction target three-dimensional space coordinate set belonging to the target object based on the reconstruction target predicted semantic label probability set;

[0160] S605, taking the reconstructed target three-dimensional space coordinate set belonging to the target object as three-dimensional point cloud data;

[0161] S606, calculating a point cloud normal vector for the three-dimensional point cloud data;

[0162] S607, performing Poisson surface reconstruction processing based on the three-dimensional point cloud data and the point cloud normal vector thereof to generate a three-dimensional closed curved surface mesh of the target object;

[0163] S608, performing Laplace smoothing processing on the three-dimensional closed curved surface mesh of the target object to generate a three-dimensional model of the target object.

[0164] In the embodiment, in the process of generating a three-dimensional model of a target object according to image information and semantic information, a set of predicted semantic label probabilities in the semantic information that are located within the range of the reconstructed target three-dimensional space and have a probability value greater than a preset probability threshold are first extracted, the semantic information representing the possibility of each space coordinate point belonging to the target object, and a set of space points greater than the threshold value is screened to determine a preliminary target region candidate set. Subsequently, it is determined whether to extract a set of predicted voxel grayscale values in the image information that are located within the same space range and have a grayscale value greater than a preset grayscale threshold, the image information providing the grayscale value of each space coordinate point, and a high grayscale value set is screened to enhance the judgment of the strong echo region of the target tissue. Through the judgment logic of whether to select the grayscale threshold, two screening paths of relying only on semantic information or comprehensively using grayscale information and semantic information are distinguished, and the adaptation ability to different data quality or use scenarios is possessed.

[0165] When the grayscale value set is extracted, the grayscale value set and the semantic label probability set are fused by spatial intersection, and a set of space coordinates that satisfy both the grayscale and semantic conditions is marked out, which more accurately represents the region of the target object in space. When the grayscale value set is not extracted, the region of the target object is directly marked based on the semantic label probability set, which is suitable for scenarios without reliable grayscale contrast.

[0166] The set of space coordinates marked by the above screening constitutes three-dimensional point cloud data, which is the geometric input of subsequent surface reconstruction. The position of each point in the point cloud reflects the spatial distribution after screening. The point cloud normal vector is calculated for the three-dimensional point cloud data, and the point cloud normal vector describes the direction information of the local surface of each point, providing a normal constraint condition for the Poisson surface reconstruction algorithm. The Poisson surface reconstruction processing interpolates a continuous closed curved surface mesh in the point cloud by solving the Poisson equation, and the closed curved surface mesh can completely express the spatial shape and surface continuity of the target object.

[0167] After the surface reconstruction is completed, a Laplace smoothing process is performed to reduce local sawtooth and irregular noise by iteratively adjusting the grid vertex position, improve the surface quality and smoothness, and generate the final target object three-dimensional model. In the overall process, through the joint screening of image information and semantic information, point cloud normal estimation, Poisson surface reconstruction and ordered connection of Laplace smoothing, high-quality reconstruction of the target object three-dimensional model from the original three-dimensional reconstruction data to the geometric continuous and smooth detail is realized.

[0168] For example, to realize high-quality three-dimensional ultrasound reconstruction of abdominal blood vessels (such as abdominal aorta) in a free breathing state, a unified method framework combining robot operation, breathing compensation screening, image segmentation and implicit neural representation (INR) reconstruction is proposed, which has high automation, good clinical adaptability and reconstruction accuracy.

[0169] The scheme relies on a convex array ultrasonic probe carried by a medical manipulator, and reads the real-time pose information of the end effector in space through an encoder. At the same time, the system acquires two-dimensional B-mode ultrasonic images in real time, and completes a wide range of continuous scanning under the premise of maintaining a safe contact force (such as 5N). The whole system does not need external optical or electromagnetic tracking equipment, but only relies on ultrasonic images and built-in sensor information of the manipulator, which can realize the image-space joint mapping required for three-dimensional reconstruction, and has good deployment flexibility.

[0170] Under the condition of free breathing, the up and down fluctuation of the patient's abdominal wall causes non-rigid deformation between image frames, especially between the end of expiration and the beginning of inspiration, the image pose offset is severe, which directly affects the continuity of three-dimensional reconstruction. Therefore, the invention introduces a breathing state screening mechanism in the image acquisition stage. By analyzing the periodic motion trend of the end of the manipulator in the vertical direction (z axis), combined with time sliding window smoothing processing, a periodic function representing the breathing state can be constructed. The stable platform segment at the end of expiration is identified as the best opportunity for image acquisition, and only the image frames at this time are retained for subsequent reconstruction, which greatly reduces the artifacts and frame skipping phenomenon caused by respiratory motion. This mechanism not only does not need additional hardware, but also can improve the spatial consistency of the image sequence while ensuring real-time.

[0171] Subsequently, semantic segmentation is performed on the filtered image sequence to extract the target blood vessel structure. A domain adaptive network is used to project the original B-mode image to an intermediate representation (IR) domain, which is trained using a computed tomography (CT) label composite image and has better semantic alignment capabilities. The ultrasound segmentation network trained in this IR space can directly process real ultrasound images, achieving high-accuracy target structure extraction. By extracting the maximum connected domain of the segmentation result in each image and tracking its spatial position and geometric size, an abnormal slice detection mechanism is also introduced to automatically remove frames with position mutations or inconsistent shapes, thereby enhancing the stability and effectiveness of the overall training data.

[0172] To break through the limitations of traditional linear interpolation in three-dimensional ultrasound reconstruction, such as sawtooth, fault, and lack of frame recovery capability, a three-dimensional volume modeling method based on implicit neural representation (INR) is proposed. This method uses a multi-layer perceptron (MLP) network structure to learn a continuous mapping function that maps to its corresponding voxel grayscale value and semantic label probability, i.e.:

[0173] f Θ (x)=(i(x),s(x))

[0174] where x is the coordinate point of the reconstructed target three-dimensional space, generated during the "uniform grid sampling operation" stage, used to query the implicit neural representation model for the point attributes.

[0175] fθ(·): The implicit neural representation model composed of a multi-layer perceptron performs continuous mapping on the input coordinates, and θ represents the network weight parameters updated by joint loss iteration.

[0176] i(x): The predicted voxel grayscale value output by the model at x, corresponding to the "reconstructed target predicted voxel grayscale value".

[0177] s(x): The predicted semantic label probability output by the model at x, corresponding to the "reconstructed target predicted semantic label probability".

[0178] To enhance the network's ability to capture high-frequency details, a position encoding mechanism is introduced at the input layer, which maps the spatial position x to a high-dimensional periodic function space, commonly expressed as:

[0179] γ(x)=(sin(2 0 πx),cos(2 0 πx),…,sin(2 L-1 πx),cos(2 L-1 πx))

[0180] wherein, γ(x): a sinusoidal function position encoding operator, is used to enhance the network's ability to express high-frequency details.

[0181] Thus, the curvature information of the voxel boundary is preserved, and the blurring and discontinuity of the reconstructed image are reduced. The network structure only needs to train sparse image frames in space to complete the dense sampling prediction of the entire three-dimensional space. Especially in the scanning process, due to the occlusion or unstable contact of the body surface, the image frame is missing, and INR can naturally complete the missing part through the learned continuous function to generate a coherent and complete three-dimensional grid. In the training process, an adaptive loss function is designed to jointly optimize the gray reconstruction error and the semantic segmentation error. The loss function for each spatial point is defined as follows:

[0182]

[0183] wherein, i*(x): the two-dimensional B-mode image frame gray true value extracted from the image sequence to be reconstructed under the timestamp index, as a supervised reference for i(x).

[0184] s*(x): the semantic label true value obtained from the target object segmentation data under the timestamp index, as a supervised reference for s(x).

[0185] λ i : reconstruction error weight coefficient, used to adjust the proportion of voxel gray reconstruction error in the joint loss; in the form of "weighted sum".

[0186] λ s : segmentation error weight coefficient, used to adjust the proportion of semantic cross-entropy error in the joint loss.

[0187] ‖i(x)–i*(x)‖ 2 : reconstruction error term between the predicted voxel gray value and the supervised true value; corresponding to "calculate reconstruction error".

[0188] CE(s(x),s*(x)): cross-entropy error term between the predicted semantic label probability and the supervised true value; corresponding to "calculate cross-entropy error".

[0189] L(x): joint loss function, which comprehensively calculates the reconstruction error and the segmentation error for a single sampling point; the overall training process takes its convergence as the termination condition.

[0190] After reconstruction, the volume data is sampled and the normal is estimated, and the final three-dimensional mesh model is generated by using the Poisson surface reconstruction algorithm, which has smooth, continuous, and interactive visualization effect. Compared with the traditional linear interpolation scheme, the reconstruction method has obvious advantages in maintaining the integrity of details and removing respiratory artifacts. For example, under the condition of free breathing, the grid Laplace roughness index of the reconstructed three-dimensional model of the aorta is reduced by more than 40%, and the error in manual annotation diameter measurement is controlled within 1mm, which has reached the clinical follow-up interpretation standard.

[0191] The whole system is deployed on a general industrial robot platform and supports mainstream medical data protocols such as DICOM or HL7-FHIR, realizing seamless connection with hospital picture archiving and communication system (PACS) and electronic medical record system (EMR). Its software and hardware structure has good universality and scalability, which is not only suitable for abdominal large blood vessels, but also can be extended to three-dimensional scanning and intelligent analysis of soft tissue organs such as liver, pancreas, gallbladder and kidney, and has broad application prospects in remote ultrasound, intraoperative navigation and intelligent auxiliary diagnosis and other application scenarios.

[0192] Example: In an ultrasound-assisted blood vessel reconstruction clinical application scenario, the mechanical arm carries an ultrasound detection device and automatically executes path scanning in the doctor's planned scanning task. First, through the precise mechanical arm control strategy, the detection device establishes reliable physical contact with the patient's abdominal skin, and uses the pressure sensor embedded in the mechanical arm end to monitor the contact pressure in real time, and dynamically adjusts the mechanical arm end effector posture to maintain the contact pressure within the preset safety threshold range in the normal direction. When the real-time monitoring confirms that the contact pressure is stable, the system generates a stable contact state flag and activates the continuous data acquisition process.

[0193] Under the condition of stable contact of the device, the ultrasound detection device continuously acquires two-dimensional B-mode image frames containing the main blood vessels of the abdomen (such as the abdominal aorta) according to the preset scanning speed, and simultaneously transmits the image frames to the backend processing terminal in real time through the video interface. At the same time of acquiring each frame of image frame, the system uses the hardware clock to record the time stamp, and analyzes the encoder data of each joint of the mechanical arm in real time, and calculates the three-dimensional space coordinates and attitude angle of the mechanical arm end effector at the corresponding time stamp by combining the kinematics model of the mechanical arm, to complete the generation of the space pose information marked with the time stamp. Each two-dimensional B-mode image frame and the corresponding space pose information are bound, and after the time alignment error of the image data and the pose data is less than the tolerance threshold, the time and space aligned data unit is packaged and stored in the cache database. The data acquisition, pose synchronization calculation and time and space binding operations are continuously performed along the preset trajectory until the scanning path accumulatively covers the complete target area planned by the doctor.

[0194] Then, the system extracts the stored spatio-temporal alignment data unit from the cache database and extracts the displacement component of the ultrasound probe in the vertical direction from the spatial pose information. A continuous periodic respiratory waveform is generated by time window sliding average filtering, local maximum and local minimum points are detected, and a platform interval with a displacement fluctuation between adjacent maximum points below a preset threshold is identified as a stable motion phase. Only image frames where the respiratory curve is in the stable motion phase are extracted, aggregated into a sequence of images to be reconstructed, to ensure that the image data used in subsequent modeling is less affected by respiratory motion artifacts.

[0195] In the image frame processing link, each two-dimensional B-mode image frame in the sequence of images to be reconstructed is input into a pre-trained image conversion network, which maps the B-mode image to a standardized intermediate representation domain through domain transfer learning. A semantic segmentation model is run in the intermediate representation domain to output a semantic class probability map for each pixel, and a morphological closing operation is applied to the probability map to eliminate small holes and noise artifacts to obtain a closed probability map. On this basis, the largest connected domain is extracted to locate the complete blood vessel target region, and then the contour boundary is calculated and the semantic probability value of the contour boundary pixel is recorded to generate a boundary mask with probability confidence. All boundary masks are integrated in time sequence to form target object segmentation data corresponding to each frame of the sequence of images to be reconstructed, which is used in the subsequent training stage.

[0196] When training the implicit neural representation model, first, the spatial pose information and the target object segmentation data are used to jointly analyze a three-dimensional spatial bounding box containing the blood vessel anatomical region, and a set of three-dimensional spatial coordinate points is generated by sampling in a uniform grid manner in the bounding box. The three-dimensional coordinate points are subjected to sine function position encoding to enhance the expression ability of spatial details (such as blood vessel boundary curvature), generating high-dimensional position encoding features. The high-dimensional position encoding features are input into the multi-layer perceptron network model as the implicit neural representation model, and the model outputs predicted voxel gray values and predicted semantic label probabilities. The system extracts the two-dimensional B-mode image frame corresponding to the timestamp from the training data as the voxel gray value supervision truth value, and the target object segmentation data corresponding to the timestamp as the semantic label probability supervision truth value, calculates the reconstruction error and cross-entropy error between the predicted value and the supervision truth value, and weightedly sums them to form a joint loss value. Based on the joint loss value, the network weight parameters are updated by back propagation, and the training is repeated iteratively until the joint loss value fluctuates below a preset convergence threshold, obtaining a trained implicit neural representation model that can be used for spatial continuous mapping.

[0197] After the training is completed, the maximum boundary of the scanning track is parsed based on the spatial pose information and the target object segmentation data, a preset three-dimensional space range is limited in combination with the spatial information of the anatomical region, and the three-dimensional space coordinates of the reconstruction target are uniformly generated in the range. The three-dimensional space coordinates are again executed with the sine function position coding to generate high-dimensional features and input the implicit neural representation model, and the output is the predicted voxel gray value of the reconstruction target corresponding to the space coordinates and the predicted semantic label probability of the reconstruction target. The system generates a three-dimensional gray field of the reconstruction target as image information according to the predicted voxel gray value, generates a three-dimensional probability field of the reconstruction target as semantic information according to the predicted semantic label probability, and combines the two into a three-dimensional reconstruction dataset covering the entire three-dimensional space range.

[0198] Finally, in the three-dimensional model generation link, a set of space points with a semantic probability value greater than a preset probability threshold in the three-dimensional space range of the reconstruction target is extracted from the semantic information. The system determines whether to extract a set of space points with a gray value greater than a preset gray threshold according to the application requirement. When the gray value set is extracted, it is combined with the semantic label probability set to mark the set of three-dimensional space coordinates of the reconstruction target belonging to the blood vessel target object; when the gray value set is not extracted, only the semantic label probability set is used to mark the target object region. The finally marked space coordinate set forms the three-dimensional point cloud data of the target object. The normal vector of each point is calculated for the point cloud data, and the Poisson surface reconstruction is performed based on the point cloud data and the normal vector to generate a closed triangular mesh surface. The Laplace smoothing processing is performed on the mesh surface to further improve the continuity and smoothness of the surface, and finally a high-quality blood vessel three-dimensional model is obtained, which can be used for surgical planning or clinical visual diagnosis.

[0199] The embodiment can improve the accuracy and robustness of the spatial positioning of the target object by using semantic information for spatial screening, combining optional gray information enhancement constraints, and flexibly selecting the screening strategy according to data reliability. The closed continuous surface structure is formed by point cloud normal estimation and Poisson surface reconstruction, and the irregularity of the surface is eliminated by combining Laplace smoothing. The finally generated three-dimensional model has high geometric precision, continuity and smoothness, and is particularly suitable for ultrasound reconstruction scenes with irregular anatomical structures and uneven data quality.

[0200] In an embodiment, a model generation device based on implicit neural representation is provided, which corresponds to the model generation method based on implicit neural representation in the above embodiment. Referring to Figure 3 , Figure 3 The functional module schematic diagram of a preferred embodiment of the model generation device based on implicit neural representation of the present application is shown in FIG. 1. The data acquisition module 10, the motion analysis module 20, the target segmentation module 30, the model training module 40, the model inference module 50 and the three-dimensional reconstruction module 60. The detailed description of each functional module is as follows:

[0201] a data acquisition module 10, configured to acquire a sequence of two-dimensional images containing a target object and spatial pose information corresponding to image frames of the sequence of two-dimensional images, which are collected by a detection device;

[0202] a motion analysis module 20, configured to analyze periodic motion of the detection device based on the spatial pose information, and select image frames in a preset stable motion phase from the sequence of two-dimensional images according to an analysis result of the periodic motion, to generate a sequence of images to be reconstructed;

[0203] a target segmentation module 30, configured to process image frames in the sequence of images to be reconstructed, to identify and extract the target object, and obtain target object segmentation data corresponding to the sequence of images to be reconstructed;

[0204] a model training module 40, configured to train an implicit neural representation model based on the sequence of images to be reconstructed, the spatial pose information, and the target object segmentation data;

[0205] a model inference module 50, configured to input a three-dimensional spatial coordinate in a preset three-dimensional spatial range into the implicit neural representation model, to generate image information and semantic information corresponding to the three-dimensional spatial coordinate;

[0206] a three-dimensional reconstruction module 60, configured to generate a three-dimensional model of the target object according to the image information and the semantic information.

[0207] In an embodiment, the data acquisition module 10 is specifically configured to:

[0208] control an end effector of a robot arm to carry the detection device to establish physical contact with a surface of a target region, and monitor contact pressure between the detection device and the surface of the target region in real time, dynamically adjust a pose of the end effector of the robot arm, so that a component of the contact pressure between the detection device and the surface of the target region in a normal direction of a contact surface is maintained in a preset threshold range, and generate a device stable contact state flag;

[0209] when the device stable contact state flag is activated, trigger the detection device to continuously collect two-dimensional B-mode image frames containing the target object at a preset scanning speed, and transmit the two-dimensional B-mode image frames to a processing terminal in real time through a video interface;

[0210] when each two-dimensional B-mode image frame is collected, a timestamp is intercepted through a hardware clock of the detection device, joint angle values are analyzed in real time by analyzing a joint encoder data stream of the robot arm, three-dimensional coordinates and pose angles of the end effector of the robot arm at corresponding timestamps are calculated based on the joint angle values and kinematic parameters of the robot arm, the three-dimensional coordinates, the pose angles, and the timestamps of the end effector of the robot arm are bound, and spatial pose information marked with the timestamps is generated;

[0211] Correlate and bind each two-dimensional B-mode image frame with the spatial pose information marked with the time stamp corresponding to the moment when each two-dimensional B-mode image frame is collected, and when the timestamp deviation is less than the tolerance threshold, encapsulate as a space-time aligned data unit and store to the cache database;

[0212] Repeat the image acquisition and transmission, pose synchronization calculation and space-time binding operation along the preset trajectory under the condition of stable contact of the device, terminate the operation after judging that the scanning path completely covers the target region through trajectory coordinate accumulation, and output a two-dimensional image sequence containing the target object composed of space-time aligned data units and a set of spatial pose information corresponding to the image frames of the two-dimensional image sequence.

[0213] In an embodiment, the motion analysis module 20 is specifically configured to:

[0214] Extract the displacement component of the detection device in the vertical direction from the spatial pose information;

[0215] Perform time window sliding average filtering on the displacement component to generate a periodic respiratory waveform;

[0216] Detect the local maximum points and local minimum points of the periodic respiratory waveform;

[0217] Mark the platform interval between adjacent local maximum points that satisfy the displacement fluctuation threshold as a preset stable motion phase;

[0218] When the periodic respiratory waveform is in the preset stable motion phase, extract the two-dimensional image frame corresponding to the time stamp;

[0219] Aggregate all two-dimensional image frames in the preset stable motion phase to generate a to-be-reconstructed image sequence.

[0220] In an embodiment, the target segmentation module 30 is specifically configured to:

[0221] Input each two-dimensional B-mode image frame in the to-be-reconstructed image sequence into a pre-trained conversion network, and map the two-dimensional B-mode image frame to an intermediate representation domain through the conversion network;

[0222] Run a semantic segmentation model in the intermediate representation domain to generate a semantic class probability map for each pixel;

[0223] Perform a morphological closing operation on the semantic class probability map to generate a probability map after closing operation processing;

[0224] Calculate the maximum connected domain of the probability map after closing operation processing, and mark the maximum connected domain as a complete target object region;

[0225] extracting a contour boundary of the complete target object region, and calculating a class probability value of a contour boundary pixel to generate a boundary mask with a probability confidence;

[0226] establishing an index mapping table of the boundary mask and an image frame timestamp;

[0227] integrating all boundary masks in a time sequence to generate target object segmentation data corresponding to the image sequence to be reconstructed.

[0228] In an embodiment, the model training module 40 is specifically configured to:

[0229] analyzing a target object space bounding box based on the spatial pose information and the target object segmentation data, and generating a uniform three-dimensional coordinate point set within the target object space bounding box;

[0230] constructing a multi-layer perception network as an implicit neural representation model;

[0231] performing a sine function position encoding on the three-dimensional spatial coordinate point set to generate a high-dimensional position encoding feature;

[0232] inputting the high-dimensional position encoding feature into the implicit neural representation model to output a predicted voxel gray value and a predicted semantic label probability;

[0233] extracting a two-dimensional B-mode image frame corresponding to the timestamp in the image sequence to be reconstructed, and defining the extracted two-dimensional B-mode image frame as a predicted voxel gray value supervision truth value;

[0234] extracting target object segmentation data corresponding to the timestamp, and defining the extracted target object segmentation data as a predicted semantic label probability supervision truth value;

[0235] calculating a reconstruction error between the predicted voxel gray value and the predicted voxel gray value supervision truth value, and calculating a cross-entropy error between the predicted semantic label probability and the predicted semantic label probability supervision truth value;

[0236] weighting and summing the reconstruction error and the cross-entropy error to form a joint loss value;

[0237] updating network weight parameters of the implicit neural representation model based on the joint loss value through a back propagation algorithm;

[0238] repeating the prediction and parameter update operations until the joint loss function fluctuation is less than a convergence threshold, and obtaining a trained implicit neural representation model.

[0239] In an embodiment, the model inference module 50 is specifically configured to:

[0240] Based on the spatial pose information and the target object segmentation data, a scanning track maximum boundary is parsed, and a preset three-dimensional space range is generated in combination with an anatomical structure space limitation;

[0241] A uniform grid sampling operation is performed in the preset three-dimensional space range to generate a reconstructed target three-dimensional space coordinate;

[0242] A sine function position encoding operation is performed on the reconstructed target three-dimensional space coordinate to generate a reconstructed target high-dimensional position encoding feature;

[0243] The reconstructed target high-dimensional position encoding feature is input into the implicit neural representation model;

[0244] The implicit neural representation model outputs a reconstructed target predicted voxel gray value and a reconstructed target predicted semantic label probability corresponding to the reconstructed target three-dimensional space coordinate;

[0245] A reconstructed target three-dimensional gray field is generated as image information according to the reconstructed target predicted voxel gray value;

[0246] A reconstructed target three-dimensional probability field is generated as semantic information according to the reconstructed target predicted semantic label probability;

[0247] The reconstructed target three-dimensional gray field and the reconstructed target three-dimensional probability field are combined to form a three-dimensional reconstruction data set covering the preset three-dimensional space range.

[0248] In an embodiment, the three-dimensional reconstruction module 60 is specifically configured to:

[0249] A reconstructed target predicted semantic label probability set in which a probability value is greater than a preset probability threshold in the reconstructed target three-dimensional space range is extracted from the semantic information;

[0250] It is determined whether to extract a reconstructed target predicted voxel gray value set in which a gray value is greater than a preset gray threshold in the reconstructed target three-dimensional space range from the image information;

[0251] When the reconstructed target predicted voxel gray value set is extracted, a reconstructed target three-dimensional space coordinate set belonging to a target object is marked in combination with the reconstructed target predicted voxel gray value set and the reconstructed target predicted semantic label probability set;

[0252] When the reconstructed target predicted voxel gray value set is not extracted, a reconstructed target three-dimensional space coordinate set belonging to a target object is marked based on the reconstructed target predicted semantic label probability set;

[0253] The reconstructed target three-dimensional space coordinate set belonging to the target object is taken as three-dimensional point cloud data;

[0254] A point cloud normal vector is calculated for the three-dimensional point cloud data;

[0255] performing Poisson surface reconstruction processing based on the three-dimensional point cloud data and the point cloud normal vectors thereof, to generate a three-dimensional closed curved surface mesh of the target object;

[0256] performing Laplace smoothing processing on the three-dimensional closed curved surface mesh of the target object, to generate a three-dimensional model of the target object.

[0257] In an embodiment, a computer device is provided, which can be a server, and an internal structure diagram thereof can be as shown in Figure 4 The computer device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with an external client through a network connection. The computer program, when executed by the processor, implements the functions or steps of a server side of a model generation method based on implicit neural representation.

[0258] In an embodiment, a computer device is provided, which can be a client, and an internal structure diagram thereof can be as shown in Figure 5 The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with an external server through a network connection. The computer program, when executed by the processor, implements the functions or steps of a client side of a model generation method based on implicit neural representation

[0259] In an embodiment, a computer device is provided, which includes a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the following steps when executing the computer program:

[0260] obtaining a two-dimensional image sequence containing a target object and spatial pose information corresponding to image frames of the two-dimensional image sequence collected by a detection device;

[0261] analyze periodic motion of the detection device based on the spatial pose information, and screen image frames in a preset stable motion stage from the two-dimensional image sequence according to an analysis result of the periodic motion to generate a to-be-reconstructed image sequence;

[0262] process image frames in the to-be-reconstructed image sequence to identify and extract the target object, and obtain target object segmentation data corresponding to the to-be-reconstructed image sequence;

[0263] train an implicit neural representation model based on the to-be-reconstructed image sequence, the spatial pose information, and the target object segmentation data;

[0264] input three-dimensional spatial coordinates in a preset three-dimensional spatial range into the implicit neural representation model to generate image information and semantic information corresponding to the three-dimensional spatial coordinates;

[0265] generate a three-dimensional model of the target object according to the image information and the semantic information.

[0266] In one embodiment, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0267] obtain a two-dimensional image sequence containing a target object collected by a detection device and spatial pose information corresponding to image frames of the two-dimensional image sequence;

[0268] analyze periodic motion of the detection device based on the spatial pose information, and screen image frames in a preset stable motion stage from the two-dimensional image sequence according to an analysis result of the periodic motion to generate a to-be-reconstructed image sequence;

[0269] process image frames in the to-be-reconstructed image sequence to identify and extract the target object, and obtain target object segmentation data corresponding to the to-be-reconstructed image sequence;

[0270] train an implicit neural representation model based on the to-be-reconstructed image sequence, the spatial pose information, and the target object segmentation data;

[0271] input three-dimensional spatial coordinates in a preset three-dimensional spatial range into the implicit neural representation model to generate image information and semantic information corresponding to the three-dimensional spatial coordinates;

[0272] generate a three-dimensional model of the target object according to the image information and the semantic information.

[0273] It should be noted that the above functions or steps that can be implemented by the computer readable storage medium or the computer device can correspond to the related descriptions of the server side and the user side in the foregoing method embodiments, and will not be described here to avoid repetition.

[0274] A person of ordinary skill in the art can understand that all or part of the processes in the foregoing method embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a nonvolatile computer readable storage medium, and when the computer program is executed, the processes of the foregoing embodiments of the method can be included. Any reference to a memory, storage, database or other medium used in the embodiments provided in the present application can include nonvolatile and / or volatile memory. The nonvolatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. The volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0275] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0276] It should be noted that if a non-company software tool or component appears in the embodiments of the present application, it is only used for example introduction and does not represent actual use. The above described embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent substitutions for part of the technical features; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A model generation method based on implicit neural representation, characterized in that, The method comprises the following steps: acquiring a two-dimensional image sequence containing a target object collected by a detection device and spatial pose information corresponding to image frames of the two-dimensional image sequence; analyzing periodic motion of the detection device based on the spatial pose information, and screening out image frames in a preset stable motion stage from the two-dimensional image sequence according to an analysis result of the periodic motion to generate a to-be-reconstructed image sequence; processing image frames in the to-be-reconstructed image sequence to identify and extract the target object, and obtaining target object segmentation data corresponding to the to-be-reconstructed image sequence; training an implicit neural representation model based on the to-be-reconstructed image sequence, the spatial pose information, and the target object segmentation data; inputting a three-dimensional space coordinate in a preset three-dimensional space range into the implicit neural representation model to generate image information and semantic information corresponding to the three-dimensional space coordinate; generating a three-dimensional model of the target object according to the image information and the semantic information.

2. The model generating method based on implicit neural representation according to claim 1, wherein, The method comprises the following steps: controlling an end effector of a mechanical arm to carry a detection device to establish physical contact with a target region surface, and monitoring contact pressure between the detection device and the target region surface in real time, dynamically adjusting a posture of the end effector of the mechanical arm, maintaining a component of the contact pressure between the detection device and the target region surface in a normal direction of a contact surface within a preset threshold range, and generating a device stable contact state flag; when the device stable contact state flag is activated, triggering the detection device to continuously collect two-dimensional B-mode image frames containing the target object at a preset scanning speed, and transmitting the two-dimensional B-mode image frames to a processing terminal in real time through a video interface; when each two-dimensional B-mode image frame is collected, obtaining joint angle values by analyzing a joint encoder data stream of the mechanical arm in real time through a hardware clock timestamp of the detection device, calculating three-dimensional coordinates and posture angles of the end effector of the mechanical arm at corresponding timestamps based on the joint angle values and kinematic parameters of the mechanical arm, binding the three-dimensional coordinates, the posture angles, and the timestamps of the end effector of the mechanical arm, and generating timestamp-marked spatial pose information; binding each two-dimensional B-mode image frame and the timestamp-marked spatial pose information corresponding to a time when each two-dimensional B-mode image frame is collected, and when a timestamp deviation is less than a tolerance threshold, encapsulating the two-dimensional B-mode image frame and the timestamp-marked spatial pose information as a space-time aligned data unit and storing the space-time aligned data unit in a cache database; repeating the image collection and transmission, the pose synchronization calculation, and the space-time binding operations along a preset trajectory under the device stable contact state, terminating the operations when a scanning path completely covers the target region by accumulating trajectory coordinates, and outputting a two-dimensional image sequence containing the target object and a set of spatial pose information corresponding to image frames of the two-dimensional image sequence composed of the space-time aligned data units.

3. The model generating method based on implicit neural representation according to claim 1, wherein, The method comprises the following steps: Extracting a displacement component of the detection device in a vertical direction in the spatial pose information; Performing time window sliding average filtering on the displacement component to generate a periodic respiratory waveform; Detecting local maximum points and local minimum points of the periodic respiratory waveform; Marking a platform interval between adjacent local maximum points that satisfies a displacement fluctuation threshold as a preset stable motion phase; When the periodic respiratory waveform is in the preset stable motion phase, extracting a two-dimensional image frame corresponding to a time stamp; Aggregating all two-dimensional image frames in the preset stable motion phase to generate a to-be-reconstructed image sequence.

4. The model generating method based on implicit neural representation according to claim 1, wherein, Processing image frames in the to-be-reconstructed image sequence to identify and extract the target object, and obtaining target object segmentation data corresponding to the to-be-reconstructed image sequence, including: Inputting each two-dimensional B-mode image frame in the to-be-reconstructed image sequence into a pre-trained conversion network, and mapping the two-dimensional B-mode image frame to an intermediate representation domain through the conversion network; Running a semantic segmentation model in the intermediate representation domain to generate a semantic class probability map for each pixel; Performing a morphological closing operation on the semantic class probability map to generate a probability map after closing operation processing; Calculating the maximum connected domain of the probability map after closing operation processing, and marking the maximum connected domain as a complete target object region; Extracting the contour boundary of the complete target object region, calculating the class probability value of the contour boundary pixel, and generating a boundary mask with probability confidence; Establishing an index mapping table of the boundary mask and the image frame time stamp; Integrating all boundary masks in time sequence to generate target object segmentation data corresponding to the to-be-reconstructed image sequence.

5. The model generating method based on implicit neural representation according to claim 1, wherein, Training an implicit neural representation model based on the to-be-reconstructed image sequence, the spatial pose information, and the target object segmentation data, including: Analyzing the target object space bounding box based on the spatial pose information and the target object segmentation data, and generating a uniform three-dimensional coordinate point set within the target object space bounding box; Constructing a multi-layer perception network as an implicit neural representation model; Performing a sine function position encoding on the three-dimensional spatial coordinate point set to generate a high-dimensional position encoding feature; Inputting the high-dimensional position encoding feature into the implicit neural representation model to output a predicted voxel gray value and a predicted semantic label probability; Extracting a two-dimensional B-mode image frame corresponding to a time stamp in the to-be-reconstructed image sequence, and defining the extracted two-dimensional B-mode image frame as a predicted voxel gray value supervision true value; Extracting target object segmentation data corresponding to the time stamp, and defining the extracted target object segmentation data as a predicted semantic label probability supervision true value; Calculating the reconstruction error between the predicted voxel gray value and the predicted voxel gray value supervision true value, and calculating the cross-entropy error between the predicted semantic label probability and the predicted semantic label probability supervision true value; Weighted sum of the reconstruction error and the cross-entropy error to form a joint loss value; Updating the network weight parameters of the implicit neural representation model based on the joint loss value through a back propagation algorithm; Repeating the prediction and parameter updating operations until the joint loss function fluctuation is less than a convergence threshold, and obtaining a trained implicit neural representation model.

6. The model generating method based on implicit neural representation according to claim 1, wherein, Inputting a three-dimensional space coordinate in a preset three-dimensional space range into the implicit neural representation model generates image information and semantic information corresponding to the three-dimensional space coordinate, including: Based on the spatial pose information and the target object segmentation data, the scanning trajectory maximum boundary is parsed, and the preset three-dimensional space range is generated in combination with the anatomical structure space limitation; Performing a uniform grid sampling operation in the preset three-dimensional space range generates a reconstructed target three-dimensional space coordinate; Performing a sine function position encoding operation on the reconstructed target three-dimensional space coordinate generates a reconstructed target high-dimensional position encoding feature; Inputting the reconstructed target high-dimensional position encoding feature into the implicit neural representation model; Through the implicit neural representation model, a reconstructed target predicted voxel gray value and a reconstructed target predicted semantic label probability corresponding to the reconstructed target three-dimensional space coordinate are outputted; According to the reconstructed target predicted voxel gray value, a reconstructed target three-dimensional gray field is generated as image information; According to the reconstructed target predicted semantic label probability, a reconstructed target three-dimensional probability field is generated as semantic information; Combining the reconstructed target three-dimensional gray field and the reconstructed target three-dimensional probability field forms a three-dimensional reconstruction data set covering the preset three-dimensional space range.

7. The model generating method based on implicit neural representation according to claim 1, wherein, According to the image information and the semantic information, a three-dimensional model of the target object is generated, including: Extracting a reconstructed target predicted semantic label probability set in the reconstructed target three-dimensional space range whose probability value is greater than a preset probability threshold in the semantic information; Determining whether to extract a reconstructed target predicted voxel gray value set in the reconstructed target three-dimensional space range whose gray value is greater than a preset gray threshold in the image information; When the reconstructed target predicted voxel gray value set is extracted, the reconstructed target three-dimensional space coordinate set belonging to the target object is marked in combination with the reconstructed target predicted voxel gray value set and the reconstructed target predicted semantic label probability set; When the reconstructed target predicted voxel gray value set is not extracted, the reconstructed target three-dimensional space coordinate set belonging to the target object is marked based on the reconstructed target predicted semantic label probability set; The reconstructed target three-dimensional space coordinate set belonging to the target object is taken as three-dimensional point cloud data; Calculating a point cloud normal vector for the three-dimensional point cloud data; Based on the three-dimensional point cloud data and its point cloud normal vector, a Poisson surface reconstruction process is performed to generate a three-dimensional closed curved surface mesh of the target object; Performing a Laplace smoothing process on the three-dimensional closed curved surface mesh of the target object generates a three-dimensional model of the target object.

8. A model generating apparatus based on implicit neural representations, characterized by, The model generation device based on the implicit neural representation includes: A data acquisition module is configured to acquire a two-dimensional image sequence containing a target object and spatial pose information corresponding to image frames of the two-dimensional image sequence collected by a detection device; A motion analysis module is configured to analyze the periodic motion of the detection device based on the spatial pose information, and select image frames in a preset stable motion stage from the two-dimensional image sequence according to the analysis result of the periodic motion to generate a to-be-reconstructed image sequence; The target segmentation module is configured to process image frames in the image sequence to be reconstructed to identify and extract the target object, and obtain target object segmentation data corresponding to the image sequence to be reconstructed. The model training module is configured to train an implicit neural representation model based on the image sequence to be reconstructed, the spatial pose information, and the target object segmentation data. The model inference module is configured to input a three-dimensional spatial coordinate within a preset three-dimensional spatial range into the implicit neural representation model to generate image information and semantic information corresponding to the three-dimensional spatial coordinate. The three-dimensional reconstruction module is configured to generate a three-dimensional model of the target object according to the image information and the semantic information.

9. A computer device, comprising: The computer device comprises a memory, a processor, and a model generation program based on implicit neural representation stored on the memory and capable of running on the processor, and the model generation program based on implicit neural representation, when executed by the processor, implements the steps of the model generation method based on implicit neural representation in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a model generation program based on implicit neural representation, and the model generation program based on implicit neural representation, when executed by the processor, implements the steps of the model generation method based on implicit neural representation in any one of claims 1-7.

Images