A bone data calibration method and device, electronic equipment and storage medium

Abstract

The application is suitable for the technical field of image processing, and provides a bone data calibration method and device, electronic equipment and storage medium, comprising: acquiring a three-dimensional bone model of a target object; in response to an editing operation on the three-dimensional bone model, determining target data corresponding to the editing operation in the three-dimensional bone model; if the target data meets a preset abnormal calibration condition, determining a calibration strategy corresponding to the three-dimensional bone model according to an abnormal type corresponding to the target data; and generating a calibration result of the target object based on the editing operation and the calibration strategy. The above method can adjust a plurality of three-dimensional data corresponding to an abnormal type in batches according to the corresponding abnormal type, so as to update the three-dimensional bone model, reduce the difficulty of adjusting the three-dimensional bone model, and improve the calibration efficiency.

Description

Technical Field

[0001] This application belongs to the field of equipment control technology, and in particular relates to a method, apparatus, electronic device and storage medium for calibrating bone data. Background Technology

[0002] With the continuous development of imaging technology, users can project two-dimensional images into three-dimensional space to construct three-dimensional models. This technology is gradually being applied in the medical field. For example, after obtaining ultrasound images of the interior of a biological organism using an ultrasound tracer, a bone model can be constructed in three-dimensional space based on the two-dimensional ultrasound images, allowing users to easily observe the bone structure inside the organism through the three-dimensional bone model. However, existing bone model construction technologies suffer from low accuracy due to potential obstructions or inaccurate shooting angles during the imaging process. The accuracy of the constructed three-dimensional bone models is relatively low, requiring users to manually adjust abnormal three-dimensional voxels, which is difficult and complex. In particular, when constructing a three-dimensional bone model, a large number of ultrasound images are used, and checking for anomalies frame by frame is time-consuming.

[0003] It is evident that existing bone model construction technologies require users to manually check for abnormal voxels in the bone model, which is difficult to operate and inefficient. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and storage medium for calibrating bone data, which can solve the problems of existing bone model construction technology, which requires users to manually check abnormal voxels in the bone model, resulting in high operational difficulty and low efficiency.

[0005] In a first aspect, embodiments of this application provide a method for calibrating bone data, including: A three-dimensional bone model of the target object is obtained; the three-dimensional bone model is generated based on multiple ultrasound images of the target object. In response to an editing operation on the three-dimensional bone model, target data corresponding to the editing operation in the three-dimensional bone model is determined; the target data includes target three-dimensional data located in the three-dimensional bone model and a target image; the target image is an ultrasound image used to construct the target three-dimensional data; If the target data meets the preset anomaly calibration conditions, then the calibration strategy corresponding to the three-dimensional bone model is determined according to the anomaly type corresponding to the target data; the anomaly type includes at least one of image quality anomaly, pose anomaly, or segmentation anomaly. Based on the editing operation and the calibration strategy, the calibration result of the target object is generated; the calibration result includes an updated three-dimensional bone model and multiple updated ultrasound images.

[0006] In one possible implementation of the first aspect, if the target data meets preset anomaly calibration conditions, then determining the calibration strategy corresponding to the three-dimensional bone model based on the anomaly type corresponding to the target data includes: Based on the target image and the tracking matrix corresponding to the target image, a pose anomaly coefficient is calculated; the tracking matrix is ​​used to project the target image into a three-dimensional space. If the pose abnormality coefficient is greater than or equal to the first abnormality threshold, then the calibration strategy corresponding to the three-dimensional bone model is determined to be the first strategy. If the pose anomaly coefficient is less than the first anomaly threshold, then the segmentation anomaly coefficient is calculated based on the target image and the neighboring images of the target image; If the segmentation anomaly coefficient is greater than or equal to the second anomaly threshold, then the calibration strategy corresponding to the three-dimensional bone model is determined to be the second strategy. If the segmentation anomaly coefficient is less than the second anomaly threshold, then the quality anomaly coefficient is calculated based on the model parameters of the target image in the three-dimensional bone model; If the quality anomaly coefficient is greater than or equal to the third anomaly threshold, then the calibration strategy corresponding to the three-dimensional bone model is determined to be the third strategy.

[0007] In one possible implementation of the first aspect, calculating the pose anomaly coefficients based on the target image and the tracking matrix corresponding to the target image includes: For any of the target images, determine the translation jump coefficient and rotation jump coefficient corresponding to the target image; the translation jump coefficient is determined based on the translation vector of the tracking matrix of the target image; the rotation jump coefficient is determined based on the rotation matrix of the tracking matrix of the target image. The pose anomaly coefficient is determined based on the pose score corresponding to the target image, the translation jump coefficient, and the rotation jump coefficient; the pose anomaly coefficient is specifically:

[0008] Among them, the The pose anomaly coefficient is the coefficient of the position. The translation jump coefficient corresponding to the target image; The rotation jump coefficient corresponding to the target image; The pose score corresponding to the target image; Here, represents the basic anomaly parameters corresponding to the target image; norm(x) is the normalization function. , , and is the weighting coefficient; i is the frame number corresponding to the target image; the basic anomaly parameter is any one of the timestamp jump coefficient, tracking anomaly coefficient, or device quality coefficient.

[0009] In one possible implementation of the first aspect, if the pose anomaly coefficient is less than the first anomaly threshold, then calculating the segmentation anomaly coefficient based on the target image and its neighboring images includes: Based on the editing region where the target image is located, determine the contribution ratio of the target image in the editing region and the consistency coefficient of the editing region in the three-dimensional bone model; If the contribution ratio is greater than a first threshold and the consistency coefficient is less than a second threshold, then the segmentation anomaly coefficient is calculated based on the anomaly scores of the target image in multiple segmentation dimensions; the segmentation dimensions include at least one of the following: The first segmentation dimension is determined based on the area of ​​the connected component corresponding to the edited region and the shape score corresponding to the connected component. The second segmentation dimension is determined based on the aspect ratio of the connected components. The third segmentation dimension is determined based on the distance between the editing area and the main bone surface area in the three-dimensional bone model. The fourth segmentation dimension is determined based on the distance between the target image and the segmentation boundary corresponding to the three-dimensional bone model. The fifth segmentation dimension is determined based on the distance between a pixel in the target image and the first bone surface point, and the distance between that pixel and the segmentation line of the last bone surface. The sixth segmentation dimension; the sixth segmentation dimension is determined based on the segmentation confidence of the segmentation region where the target image is located; The seventh segmentation dimension is determined based on the distance between the position of the back projection of the target image in the three-dimensional bone model and the bone surface.

[0010] In one possible implementation of the first aspect, if the segmentation anomaly coefficient is less than the second anomaly threshold, then calculating the quality anomaly coefficient based on the model parameters of the target image in the three-dimensional bone model includes: The quality anomaly coefficient is calculated based on the anomaly scores of the target image in multiple imaging quality dimensions; the imaging quality dimensions include at least one of the following: The first imaging dimension is determined based on the echo response intensity of the target image on the bone surface in the three-dimensional bone model. The second imaging dimension is determined based on the continuous score of the bone surface lines in the three-dimensional bone model of the target image. The third imaging dimension is determined based on the bone surface segmentation thickness of the target image in the three-dimensional bone model. The fourth imaging dimension is determined based on the number of multi-layer structures corresponding to the three-dimensional bone model. The fifth imaging dimension; the fifth imaging dimension is determined based on the image noise of the target image; The sixth imaging dimension; the sixth imaging dimension is determined based on the area of ​​the shadow region of the target image; The seventh imaging dimension is determined based on the number of abnormal pixels in the target image; the number of abnormal pixels is the pixels deleted according to the editing operation and / or the pixels identified by a preset anomaly algorithm.

[0011] In one possible implementation of the first aspect, generating the calibration result of the target object based on the editing operation and the calibration strategy includes: If the calibration strategy is the first strategy, then the target image is deleted and the first three-dimensional data corresponding to the target image is removed from the three-dimensional bone model, and the three-dimensional bone model is updated based on the deleted first three-dimensional data; the first strategy is the calibration strategy executed when the anomaly type is pose anomaly; If the calibration strategy is the second strategy, then the two-dimensional segmentation region corresponding to the target image and the second three-dimensional data of the two-dimensional segmentation region in the three-dimensional bone model are deleted, and the three-dimensional bone model is updated based on the deleted second three-dimensional data; the second strategy is the calibration strategy executed when the anomaly type is segmentation anomaly; If the calibration strategy is the third strategy, then the target image is deleted; the third strategy is the calibration strategy executed when the anomaly type is image quality anomaly.

[0012] In one possible implementation of the first aspect, after generating the calibration result of the target object based on the editing operation and the calibration strategy, the method further includes: For any three-dimensional data in the updated three-dimensional bone model, the support corresponding to the three-dimensional data is determined based on the two-dimensional data source corresponding to the three-dimensional data; the two-dimensional data source is determined based on the pixels corresponding to the three-dimensional data in the updated multiple ultrasound images. Based on the support level corresponding to each of the three-dimensional data, the updated three-dimensional bone model is adjusted a second time to obtain the target bone model.

[0013] In one possible implementation of the first aspect, obtaining the three-dimensional bone model of the target object includes: Acquire the multiple ultrasound images; For any of the ultrasound images, the three-dimensional projection data of each pixel in the ultrasound image in three-dimensional space is determined according to the calibration matrix and the tracking matrix corresponding to the ultrasound image. Based on the three-dimensional projection data of all pixels in the three-dimensional space, a three-dimensional bone model of the target object is obtained; the three-dimensional bone model records the source information corresponding to each three-dimensional projection data; the source information includes the pixels of at least one ultrasound image corresponding to the three-dimensional projection data.

[0014] Secondly, embodiments of this application provide a bone data calibration device, the device comprising: A model acquisition unit is used to acquire a three-dimensional bone model of a target object; the three-dimensional bone model is generated based on multiple ultrasound images of the target object. A model editing unit is configured to, in response to an editing operation on the three-dimensional bone model, determine target data in the three-dimensional bone model corresponding to the editing operation; the target data includes target three-dimensional data located in the three-dimensional bone model and a target image; the target image is an ultrasound image used to construct the target three-dimensional data; The strategy determination unit is used to determine the calibration strategy corresponding to the three-dimensional bone model according to the anomaly type corresponding to the target data if the target data meets the preset anomaly calibration conditions; the anomaly type includes at least one of image quality anomaly, pose anomaly, or segmentation anomaly. The strategy execution unit is used to generate calibration results for the target object based on the editing operation and the calibration strategy; the calibration results include an updated three-dimensional bone model and multiple updated ultrasound images.

[0015] Thirdly, embodiments of this application provide an electronic device, the electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in any of the first aspects above.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the first aspects above.

[0017] Fifthly, embodiments of this application provide a computer program product that, when run on a robotic arm, causes a drone to perform the method described in any one of the first aspects above.

[0018] Compared with existing technologies, the beneficial effects of this application embodiment are as follows: The electronic device can construct a three-dimensional bone model of a target object based on multiple ultrasound images, realizing the conversion from two-dimensional images to three-dimensional models. Users can edit this three-dimensional bone model. Responding to the editing operation, the electronic device determines the target data corresponding to the editing operation. If the target data meets the abnormal calibration conditions, it determines the corresponding calibration strategy. The three-dimensional bone model is calibrated using the calibration strategy and the specified target data, thereby obtaining a calibration result. This calibration result includes the updated three-dimensional bone model and multiple updated ultrasound images, achieving the purpose of automatically calibrating the three-dimensional bone model. Unlike existing bone model construction technologies, this application embodiment does not require users to manually edit all abnormal three-dimensional voxels one by one. Instead, the electronic device can determine in real time whether the user-specified target data meets the abnormal calibration conditions based on the user-initiated operation, i.e., whether there are one or more abnormalities such as image quality abnormalities, pose abnormalities, or segmentation abnormalities. Therefore, it can batch adjust multiple three-dimensional data corresponding to the corresponding abnormality type to update the three-dimensional bone model, reducing the difficulty of adjusting the three-dimensional bone model and improving calibration efficiency. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the relevant modules involved in the implementation of the above-described bone data calibration method by an electronic device provided in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the implementation of a bone data calibration method provided in an embodiment of this application; Figure 3 This is a flowchart illustrating the specific implementation of a bone data calibration method S203 provided in the second embodiment of this application. Figure 4 This is a schematic diagram of the calibration process for various anomaly types provided in an embodiment of this application; Figure 5 This is a schematic diagram of a calibration strategy for different anomaly types provided in an embodiment of this application; Figure 6 This is a flowchart illustrating the specific implementation of a bone data calibration method provided in the third embodiment of this application after step S204. Figure 7 This is a flowchart illustrating the specific implementation of a bone data calibration method S201 provided in the fourth embodiment of this application; Figure 8 This is a schematic diagram of the structure of a bone data calibration device provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0021] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0022] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0023] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0024] Ultrasound imaging technology is widely used in orthopedic surgical navigation, robot-assisted surgery, and other fields. Two-dimensional ultrasound images can be used to generate three-dimensional bone models of biological interiors, thereby obtaining surface information of bone structures. This embodiment provides a method for generating bone structure models based on ultrasound images. Specifically, it involves using a tracking device (such as an ultrasound tracer) to obtain the spatial pose matrix corresponding to each frame of a two-dimensional ultrasound image, performing bone surface segmentation on the two-dimensional ultrasound image to obtain a two-dimensional bone surface mask corresponding to each frame of the ultrasound image, and then projecting each pixel in the two-dimensional ultrasound image into three-dimensional space according to the probe calibration matrix and tracking matrix corresponding to the ultrasound tracer, thereby obtaining a three-dimensional bone structure model. This three-dimensional bone structure model may include bone surface point clouds, voxel models, or surface models.

[0025] While existing bone structure modeling techniques can construct three-dimensional bone models from two-dimensional ultrasound images, these techniques still have the following limitations: First, while users can edit erroneous regions in the 3D bone structure, there is a lack of effective correlation between the 3D editing results and the original 2D ultrasound images. After a user deletes or modifies an erroneous region in 3D space, the electronic device can only modify the 3D bone model itself, and cannot determine which 2D ultrasound images the erroneous region originated from, or which pixels, scan lines, or segmented connected regions within those ultrasound images. Therefore, when re-performing 3D reconstruction based on the same ultrasound images, the same error may reappear, requiring the user to manually edit and adjust it again.

[0026] Second, existing bone structure construction methods cannot identify the sources of the aforementioned errors. Errors in 3D bone models can stem from a variety of causes, including 2D segmentation errors, tracking matrix jumps, probe calibration errors, time synchronization errors, ultrasound imaging artifacts, strong echo artifacts behind the bone, and thick shell structures resulting from low-quality image frames or multi-frame fusion. Existing methods typically cannot automatically identify the types of these errors based on the 3D editing operations performed by the user.

[0027] Third, existing methods require manual editing of the 3D bone model, resulting in low efficiency and significant difficulty in model calibration. Frame-by-frame inspection of 2D ultrasound images requires substantial manual operation, and operators find it difficult to determine from a single 2D image whether it has a severe impact on 3D reconstruction. While manual deletion on the 3D model is intuitive, it cannot correct 2D segmentation results or mark abnormal ultrasound images as valid.

[0028] Fourth, existing methods for constructing 3D bone models typically employ the same or similar fusion strategies for all valid frames, making it difficult to apply different processing methods to different error types. For example, using the same method to fuse ultrasound images with tracking matrix anomalies with images of local segmentation anomalies reduces the accuracy of the resulting 3D bone model and makes it impossible to accurately calibrate the model according to different anomaly types using corresponding strategies.

[0029] Therefore, existing bone structure construction technologies lack a traceable relationship between 3D editing operations and the source of 2D frames in the process of building 3D models based on 2D ultrasound images. They cannot automatically locate relevant 2D ultrasound images based on the user's 3D editing operations, nor can they classify abnormal types and perform corresponding corrections.

[0030] The bone data calibration method provided in this application can be applied to scenarios where a three-dimensional bone structure model is constructed based on two-dimensional ultrasound images. This bone data calibration method can be applied to electronic devices, such as computers, laptops, servers, etc. The type of electronic device used can be determined according to the actual application scenario and is not limited here. Specifically, the electronic device can be equipped with an application program for constructing three-dimensional structures. Within this application, a bone structure model of the target object can be constructed, and ultrasound images sent by other devices can be received.

[0031] In this embodiment, the above-mentioned bone data calibration method can be applied to the observation scenario of any object with bone structure. That is, the target object can be any living organism, such as cats, dogs and tigers, or humans. The specific type of target object can be determined according to the actual observation needs and is not limited here.

[0032] For example, Figure 1 This diagram illustrates relevant modules involved in implementing the bone data calibration method described above using an electronic device according to an embodiment of this application. See also... Figure 1 As shown, the aforementioned electronic device may include: a data acquisition module 11, a 3D modeling module 12, a mapping module 13, an interactive editing module 14, a scoring module 15, a correction module 16, and an output module 17. The functions and connections of each module are described in detail below: Data acquisition module 11: Used to acquire multiple ultrasound images, as well as the segmented image, tracking matrix, and probe calibration parameters of the ultrasound probe of the ultrasound tracer corresponding to each ultrasound image. This data acquisition module 11 can send the acquired data to the 3D modeling module 12 and the mapping module 13.

[0033] The 3D modeling module 12 is used to construct a 3D bone model of the target object. Specifically, the 3D modeling module 12 can extract 2D bone surfaces based on data sent by the data acquisition module 11, project the 2D bone surface data into 3D space, and fuse the 2D bone surface data obtained from multiple ultrasound images to obtain a 3D bone model. Then, the 3D modeling module 12 can send the generated 3D bone model to the interactive editing module 14 for subsequent editing operations by the user.

[0034] Mapping module 13: Used to establish the mapping relationship between each pixel in the two-dimensional ultrasound image and the three-dimensional data in the three-dimensional bone model. Mapping module 13 can establish the above mapping relationship based on the data sent by data acquisition module 11 and the three-dimensional bone model generated by three-dimensional modeling module 12. The mapping relationship can be associated with stored source information of the three-dimensional data, which may include: frame number, pixel coordinates of the source pixel, scan line number, connected component number, and contribution weight, etc. The number of source pixels for a single three-dimensional data point can be one or multiple.

[0035] Interactive editing module 14: Used to receive editing operations initiated by the user. The editing operations may include deletion, addition and modification of three-dimensional data in the three-dimensional bone model, and send the editing results to the scoring module 15.

[0036] Scoring module 15: This module determines the anomaly coefficients corresponding to each anomaly type based on the user's editing operations. For example, scoring module 15 can calculate the quality anomaly coefficient corresponding to imaging quality anomalies, the pose anomaly coefficient corresponding to pose anomalies, and the segmentation anomaly coefficient corresponding to segmentation anomalies. Furthermore, after updating the 3D bone model, it can also calculate the consistency score for each 3D data point. The quality anomaly coefficients calculated by scoring module 15 can be sent to correction module 16 for updating the 3D bone model.

[0037] Optionally, the scoring module 15 can also determine the calibration strategy corresponding to each anomaly type based on the anomaly coefficient, and send the calibration strategy to the correction module 16.

[0038] Correction module 16: This module corrects the 3D bone model and multiple ultrasound images based on the calibration strategy and user editing operations. Correction operations may include deleting abnormal full-frame ultrasound images, correcting segmentation anomalies, reducing contribution weights, and correcting scan lines. Correction module 16 can send the corrected calibration results to mapping module 13 to update the established mapping relationship, and can also send the calibration results to output module 17.

[0039] Output module 17: Used to display calibration results. These results may include the updated 3D bone model, updated ultrasound images, quality scores, and frame indices for each ultrasound image.

[0040] Please see Figure 2 , Figure 2 This illustration shows a schematic diagram of a bone data calibration method provided in an embodiment of this application, specifically describing how to determine the calibration strategy for controlling bone data and output calibration results. Specifically, the method includes the following steps: In S201, a three-dimensional bone model of the target object is obtained; the three-dimensional bone model is generated based on multiple ultrasound images of the target object.

[0041] In this embodiment, before constructing the three-dimensional bone model of the target object, multiple ultrasound images of the target object's interior can be acquired using an ultrasound tracer. Each ultrasound image may include a corresponding frame number, which can be used to determine the order in which the multiple ultrasound images were captured. The electronic device can map the two-dimensional image data into a three-dimensional space based on the frame number and the multiple ultrasound images, and construct the three-dimensional bone model of the target object in the three-dimensional space based on the bone surface pixels in the ultrasound images.

[0042] In this embodiment, the target object can be any organism with a skeletal structure, such as a human, or other organisms, such as a pig or a mouse. There is no limitation on the type of organism.

[0043] In S202, in response to the editing operation on the three-dimensional bone model, target data corresponding to the editing operation in the three-dimensional bone model is determined; the target data includes target three-dimensional data and target image located in the three-dimensional bone model; the target image is an ultrasound image used to construct the target three-dimensional data.

[0044] In this embodiment, after acquiring the three-dimensional bone model of the target object, the electronic device can display the three-dimensional bone model on the interactive interface. Since the ultrasound image may be obstructed during capture, or the accuracy of the constructed three-dimensional bone model may be affected by the shooting posture, the user can edit the three-dimensional bone model while viewing it on the interactive interface to adjust any abnormal areas.

[0045] In some possible implementations, the above editing operation includes at least one of the following operations: Operation 1: Delete operation. The delete operation can be used to delete one or more three-dimensional voxels, point cloud clusters, or local surface regions in a three-dimensional bone model.

[0046] Operation 2: Add operation. The add operation is used to add new bone surface points, three-dimensional voxels or local surfaces in the three-dimensional bone model.

[0047] Operation 3: Movement operation. The movement operation is specifically used to translate the local bone surface along the direction, the direction of the ultrasound scan line, or any three-dimensional direction.

[0048] It should be noted that when editing a 3D bone model, users can perform any of the above-mentioned editing operations, or a combination of one or more of these operations. For example, after adding a bone facet point, the added bone facet point can be moved. The specific editing operations performed by the user can be determined according to the actual situation, and no restrictions are imposed on the editing operations here.

[0049] In this embodiment, the electronic device can convert the user's manual editing behavior in three-dimensional space into two-dimensional frame correction information. For deletion operations, the system can determine whether it is an abnormal pose of the entire frame, a false positive in local segmentation, or low-quality imaging; for addition operations, the system can check for possible missed two-dimensional frames and pixel regions; thus, the three-dimensional manual editing results not only modify the final three-dimensional model, but also correct the original two-dimensional segmentation or frame-level validity markers, and reduce the cost of frame-by-frame manual inspection.

[0050] In this embodiment, after receiving an editing operation from a user on a three-dimensional bone model, the electronic device can determine the target data corresponding to the editing operation, i.e., the updated data generated based on the editing operation. This updated data can include three-dimensional data located in the three-dimensional bone model, or it can include source two-dimensional data of the three-dimensional data, such as a source ultrasound image or a set of pixels in a source ultrasound image. For example, the area in the three-dimensional bone model corresponding to the editing operation is the editing area, which can be represented by E. E represents the combination of three-dimensional voxels, a set of three-dimensional points, or a set of surface triangles within the editing area.

[0051] For example, the electronic device can determine target data based on the aforementioned editing region E. The target data can be a three-dimensional voxel in the editing region E, and the source two-dimensional data corresponding to the aforementioned three-dimensional voxel, such as a source ultrasound image and pixels in the ultrasound image.

[0052] In some possible implementations, the process of determining the target data can be as follows: the electronic device determines all three-dimensional data belonging to or located in the neighborhood of the editing area E, based on the editing area E. And determine the source data of each of the above three-dimensional data, and obtain the above three-dimensional data. Corresponding source set That is, the two-dimensional observation set involved in the above editing operation can be represented as , in This includes the frame number of all ultrasound images associated with the 3D editing region, the location of the source pixel of the 3D data, the scan line number, the connected component number, and the contribution weight. The contribution weight is specifically used to determine the degree of contribution of the 2D observations to the 3D data.

[0053] In this embodiment, the electronic device can determine the support coefficient of each frame of ultrasound image in the aforementioned two-dimensional observation set for the edited region. The support coefficient of the ultrasound image of the i-th frame for the edited region can be expressed as:

[0054] in Let represent the support coefficient of the i-th frame for the edit region E. u and v represent the coordinates of the source pixel of the 3D data; c is the scan line number; l is the number of the 2D segmented connected component to which the source pixel belongs; and w represents the contribution weight of the 2D observation to the 3D element. Wherein, if A higher value indicates that the frame makes a significant contribution to the user's editing area and should be a key focus of inspection.

[0055] In this embodiment, the target data may include target 3D data and target image, and the target 3D data specifically refers to the aforementioned 3D data. The aforementioned target image is specifically an ultrasound image used to construct the aforementioned three-dimensional data, for example, based on... The frame number of all ultrasound images associated with the 3D editing area, the location of the source pixel of the 3D data, the scan line number, and the connected component number are determined.

[0056] In S203, if the target data meets the preset abnormal calibration conditions, then the calibration strategy corresponding to the three-dimensional bone model is determined according to the abnormality type corresponding to the target data; the abnormality type includes at least one of image quality abnormality, pose abnormality, or segmentation abnormality.

[0057] In this embodiment, the electronic device can determine whether the target data corresponding to the user's editing operation meets the preset abnormal calibration conditions based on the user's editing operation. That is, whether it is necessary to adjust other three-dimensional data in the three-dimensional bone model in addition to the three-dimensional data edited by the user, thereby achieving the purpose of batch calibration of abnormal data and reducing the difficulty of operation for the user.

[0058] In this embodiment, the target data satisfying the above-mentioned anomaly calibration conditions can be specifically achieved by the electronic device calculating the anomaly coefficients corresponding to each anomaly type based on the target data, such as calculating the pose anomaly coefficient, segmentation anomaly coefficient, and command anomaly coefficient, and then comparing each anomaly coefficient with its corresponding threshold. If any anomaly coefficient is greater than the threshold corresponding to its corresponding anomaly type, the target data is identified as satisfying the above-mentioned anomaly calibration conditions, and the operation of S203 is executed.

[0059] Optionally, if the target data does not meet the abnormal calibration conditions—for example, if all the abnormal coefficients determined based on the target data are less than their corresponding thresholds—then the target data is identified as not meeting the abnormal calibration conditions. In this case, the electronic device can update the three-dimensional bone model according to the editing operation, without needing to calibrate other three-dimensional data besides the three-dimensional data specified in the editing operation. Optionally, after updating the three-dimensional bone model according to the above editing operation, the electronic device can also update the source data corresponding to the three-dimensional data specified in the editing operation. For example, it can mark the source ultrasound images and mark the frame numbers of the source ultrasound images. It can also update the mapping relationship between the three-dimensional data and the source data. The specific update process can be determined according to the actual situation.

[0060] In this embodiment, when the electronic device identifies that the target data meets the abnormal calibration conditions, the electronic device can determine the abnormal type corresponding to the target data. Since different abnormal types can correspond to different calibration strategies, the electronic device can select the corresponding calibration strategy according to the above abnormal type.

[0061] In some possible implementations, the target data can correspond to an anomaly type. That is, if the anomaly coefficient of one of the anomaly types of the target data is greater than its corresponding threshold, then the anomaly type with an anomaly coefficient greater than the threshold can be taken as the anomaly type corresponding to the target data. For example, if the electronic device detects that the pose anomaly coefficient of the target data is greater than the first threshold, while the segmentation anomaly coefficient and the quality anomaly coefficient are less than their corresponding thresholds, then the anomaly type of the target data can be identified as pose anomaly, and the calibration strategy corresponding to the pose anomaly can be selected and executed.

[0062] In some possible implementations, the target data may correspond to two or more anomaly types, meaning the anomaly coefficients of the two anomaly types of the target data are greater than their corresponding thresholds. In this case, there can be multiple corresponding strategies, or a single anomaly strategy can be determined based on multiple anomaly types. For example, if the pose anomaly coefficient of the target data is greater than a first threshold and the segmentation anomaly coefficient is greater than a second threshold, the electronic device can be calibrated using the first strategy corresponding to the pose anomaly and the second strategy corresponding to the segmentation anomaly. Optionally, the electronic device can determine the calibration strategy corresponding to the above anomaly combination based on the pose anomaly and the segmentation anomaly, and then perform calibration using the calibration strategy corresponding to the above anomaly combination.

[0063] In S204, based on the editing operation and the calibration strategy, a calibration result for the target object is generated; the calibration result includes an updated three-dimensional bone model and multiple updated ultrasound images.

[0064] In this embodiment, after determining the calibration strategy, the electronic device can determine the calibration data corresponding to the calibration strategy. The calibration data includes other three-dimensional data besides the three-dimensional data specified in the editing operation, as well as related ultrasound images. The electronic device can adjust the three-dimensional bone model according to the other three-dimensional data corresponding to the calibration strategy and the three-dimensional data specified in the editing operation to obtain an updated three-dimensional bone model. It can also update the two-dimensional ultrasound images from the relevant sources according to the updated three-dimensional bone model to obtain updated ultrasound images, thereby obtaining the above-mentioned calibration results.

[0065] In some possible implementations, the calibration result may include at least one of the following: Result 1: The updated 3D bone model, which may include the updated bone surface point cloud, voxel model or surface model.

[0066] Result 2: In the case of an anomaly type of pose anomaly, a list of frames that are determined to be pose anomalies can be included, i.e., multiple ultrasound images involved in pose anomalies.

[0067] Result 3: In the case of anomaly type being imaging quality anomaly, a list of frames that are determined to be low-quality imaging or artifact-related can be included. Result 4: In the case of anomaly type segmentation anomaly, it can include the completed or corrected two-dimensional segmented image; Result 5: The mapping relationships determined based on the revised 3D bone model, such as the mapping relationship between the 3D editing area and the 2D source frame.

[0068] Result 6: Correction Report. This report displays adjustment suggestions and automatic correction results for the types of anomalies that users can review.

[0069] In some implementations, the calibration results described above can be applied to human-machine collaborative verification scenarios. Specifically, after generating the calibration results, the electronic device can first output at least one anomaly type, the corresponding two-dimensional frame position, the anomaly region range, and suggested handling methods, without immediately updating the three-dimensional bone model. The user can verify the calibration content based on the calibration results. If the verification is correct, the electronic device then updates the frame-level availability markers, local two-dimensional segmentation, or three-dimensional reconstruction results. This method can be applied to medical software applications that require the retention of manual review and operation records, thereby improving the reliability of the calibration results.

[0070] As can be seen from the above, the bone data calibration method provided in this application embodiment allows an electronic device to construct a three-dimensional bone model of a target object based on multiple ultrasound images, realizing the conversion from two-dimensional images to a three-dimensional model. The user can edit the three-dimensional bone model. Responding to the editing operation, the electronic device determines the target data corresponding to the editing operation. If the target data meets the abnormal calibration conditions, it determines the corresponding calibration strategy. The three-dimensional bone model is calibrated using the calibration strategy and the specified target data, thereby obtaining a calibration result. This calibration result includes the updated three-dimensional bone model and multiple updated ultrasound images, achieving the purpose of automatically calibrating the three-dimensional bone model. Unlike existing bone model construction technologies, this application embodiment does not require the user to manually edit all abnormal three-dimensional voxels one by one. Instead, the electronic device can determine in real time whether the user-specified target data meets the abnormal calibration conditions based on the user-initiated operation, i.e., whether there are one or more abnormalities such as image quality abnormalities, pose abnormalities, or segmentation abnormalities. Therefore, it can batch adjust multiple three-dimensional data corresponding to the corresponding abnormality type to update the three-dimensional bone model, reducing the difficulty of adjusting the three-dimensional bone model and improving calibration efficiency. Because the embodiments of this application have the above-mentioned beneficial effects, they can be applied to the following multiple scenarios, specifically: (1) Applied to medical image processing systems, three-dimensional fusion reconstruction is performed on two-dimensional ultrasound image sequences, two-dimensional bone surface segmentation results and corresponding tracking poses. Combined with three-dimensional voxel fusion, bone surface segmentation post-processing and medical image interactive editing, three-dimensional expression, error localization and local correction of two-dimensional ultrasound observation results are realized.

[0071] (2) Applied to orthopedic surgical navigation system, it acquires two-dimensional ultrasound images with tracking pose information during surgery, and then reconstructs three-dimensional bone models of femur, tibia, vertebra or other bone structures. Based on the calibration results generated by the user's editing operation, it automatically traces back to the corresponding abnormal two-dimensional frame or abnormal segmentation area, thereby improving the spatial consistency and reliability of the intraoperative navigation model.

[0072] (3) Applied to robot-assisted surgical systems and CT-free bone surface modeling scenarios, ultrasound images are obtained through intraoperative ultrasound tracers, and then three-dimensional bone models of local bone surfaces are obtained to determine the observation information of local bone surfaces. In the absence of complete preoperative three-dimensional images, the patient-specific bone surface model is reconstructed, enabling the system to perform navigation, planning or registration based on the corrected bone surface data.

[0073] (4) Applied to medical image annotation, training data correction and related industrial testing systems, by back-mapping the calibration results generated by the user's editing operation to the original two-dimensional ultrasound image or obtaining the sensor data of the ultrasound image, low-quality frame markers or local segmentation correction data are automatically generated, thereby assisting in the construction of high-value training datasets and supporting error tracing and quality assessment of the original observation data.

[0074] Figure 3 A flowchart illustrating the specific implementation of a bone data calibration method S203 provided in the second embodiment of this application is shown. See also... Figure 3 As shown, relative to Figure 2 In the embodiment provided in this application, bone data calibration method, step S203 includes steps S2031 to S2036, which are described in detail below: Specifically, if the target data meets preset anomaly calibration conditions, then based on the anomaly type corresponding to the target data, a calibration strategy corresponding to the three-dimensional bone model is determined, including: In S2031, pose anomaly coefficients are calculated based on the target image and the tracking matrix corresponding to the target image; the tracking matrix is ​​used to project the target image into three-dimensional space.

[0075] In this embodiment, after acquiring the target data, the electronic device can determine the target image contained in the target data and the tracking matrix corresponding to the acquisition of the target image. The target image is the ultrasound image from which the target three-dimensional data is sourced among the multiple ultrasound images. It should be noted that since there can be multiple target three-dimensional data sets, the corresponding number of target images can also be multiple.

[0076] In some implementations, when there are multiple target images, the electronic device can calculate the anomaly score corresponding to each target image based on each target image and its corresponding tracking matrix, and then determine the pose anomaly coefficient based on the anomaly score corresponding to each target image. For example, the maximum value of the anomaly score of each target image can be used as the pose anomaly coefficient, or the average value of the anomaly score of the target images can be used as the pose anomaly coefficient.

[0077] In some implementations, when there are multiple target images, the electronic device can calculate the pose anomaly coefficient corresponding to each target image separately. If the pose anomaly coefficient corresponding to any target image is greater than the first threshold, the target image is identified as an abnormal image, and the anomaly type of the target data is identified as pose anomaly, and the operation of S2032 is executed.

[0078] Furthermore, as another embodiment of this application, the above-described S2031 may specifically include the following steps: In S2031.1, for any of the target images, the translation jump coefficient and rotation jump coefficient corresponding to the target image are determined; the translation jump coefficient is determined based on the translation vector of the tracking matrix of the target image; the rotation jump coefficient is determined based on the rotation matrix of the tracking matrix of the target image.

[0079] In this embodiment, the electronic device can acquire adjacent images with consecutive frame numbers to the target image, and determine the translation jump coefficient and rotation jump coefficient corresponding to the target image based on the target image and the adjacent images.

[0080] Specifically, the above translation jump coefficient can be expressed as:

[0081] in, is the translation jump coefficient of the target image; i is the frame number corresponding to the target image; The tracking matrix of the target image Translation vector in; Tracking matrix of adjacent images Translation vector in the text.

[0082] Specifically, the aforementioned rotational jump coefficient can be expressed as:

[0083] in, The translation jump coefficient of the target image; The tracking matrix of the target image The rotation matrix in; Tracking matrix of adjacent images The rotation matrix in the matrix.

[0084] In S2031.2, the pose anomaly coefficient is determined based on the pose score corresponding to the target image, the translation jump coefficient, and the rotation jump coefficient; the pose anomaly coefficient is specifically:

[0085] Among them, the The pose anomaly coefficient is the coefficient of the position. The translation jump coefficient corresponding to the target image; The rotation jump coefficient corresponding to the target image; The pose score corresponding to the target image; Here, represents the basic anomaly parameters corresponding to the target image; norm(x) is the normalization function. , , and is the weighting coefficient; i is the frame number corresponding to the target image; the basic anomaly parameter is any one of the timestamp jump coefficient, tracking anomaly coefficient, or device quality coefficient.

[0086] In this embodiment, the electronic device can determine whether there is an overall offset between the target image and adjacent images, and then determine whether there is a pose abnormality when the target image was captured. Specifically, the electronic device can make anomaly judgments based on the two-dimensional similarity between the target image and adjacent images. Specifically, if the content or segmentation shape of the target image and adjacent images are similar, but their three-dimensional back-projection positions are obviously discontinuous, the anomaly type of the target image can be identified as a pose abnormality. This can be determined using the pose abnormality coefficient mentioned above.

[0087] In this embodiment, the aforementioned pose anomaly coefficient can be calculated based on the translational jump coefficient and rotational jump coefficient corresponding to the target image. These two jump coefficients can determine the overall offset between the target image and adjacent images. Furthermore, the electronic device can also obtain the pose score of the target image, i.e. The higher the posture score, the better the match between the target image and the standard posture during image capture. This posture score can be determined by the probe posture of the ultrasonic tracer when acquiring the target image. The electronic device can also acquire basic anomaly parameters, where a longer time interval between adjacent ultrasonic images (i.e., the time difference between the target image and adjacent images) corresponds to a larger timestamp jump coefficient and a larger basic anomaly parameter. The aforementioned tracking anomaly coefficient and device quality coefficient are determined based on the hardware of the ultrasonic tracer and tracking module. The electronic device can calculate the posture anomaly coefficient corresponding to the target image based on the various data acquired above, thereby determining whether there is a posture anomaly when capturing the target image.

[0088] In S2032, if the pose abnormality coefficient is greater than or equal to the first abnormality threshold, then the calibration strategy corresponding to the three-dimensional bone model is determined to be the first strategy.

[0089] In this embodiment, when the electronic device identifies that the pose abnormality coefficient of the target image is greater than the first abnormality threshold, it can determine that the pose abnormality is corresponding to the capture of the target image. Then, the target image can be identified as a pose abnormality image. When there are multiple target images, the pose abnormality coefficient corresponding to each target image can be calculated separately. Thus, all pose abnormality images with pose abnormality coefficients greater than or equal to the first abnormality threshold can be adjusted through the first strategy described above, thereby achieving the purpose of batch processing.

[0090] In this embodiment, the electronic device can first calculate the pose anomaly coefficient corresponding to the target data. Since the computational load for pose anomaly identification is relatively small, but its impact on the overall three-dimensional bone model is significant, by first identifying whether the pose anomaly coefficient is greater than a first anomaly threshold, and then calibrating the anomaly, subsequent anomaly type identification and calibration are performed.

[0091] In some possible implementations, the process of executing the first strategy can specifically be as follows: if the calibration strategy is the first strategy, then delete the target image and remove the first three-dimensional data corresponding to the target image from the three-dimensional bone model, and update the three-dimensional bone model based on the deleted first three-dimensional data. Specifically, when the electronic device detects that the pose anomaly coefficient of the target image is greater than or equal to a first anomaly threshold, that is, when... If the target image is determined to be an image with an abnormal pose, the image with the abnormal pose is set to unusable and all 3D points or voxels contributed by that frame are deleted; alternatively, the image with the abnormal pose is designated as an ultrasound image that requires re-acquiring the localization matrix or re-registration.

[0092] In S2033, if the pose anomaly coefficient is less than the first anomaly threshold, then the segmentation anomaly coefficient is calculated based on the target image and the neighboring images of the target image.

[0093] In this embodiment, when the electronic device detects that the pose anomaly coefficients corresponding to each target image are all less than the first anomaly threshold, it can perform segmentation anomaly identification. Specifically, the electronic device can calculate the segmentation anomaly coefficient corresponding to the target image based on the target image and adjacent images. The adjacent images can be other ultrasound images whose frame numbers are connected to the target image. For example, if the frame number of the target image is i, the frame number of the adjacent images can be i-1 or i+1.

[0094] Furthermore, as another embodiment of this application, the above-described S2033 may specifically include the following steps: In S2033.1, based on the editing region where the target image is located, the contribution ratio of the target image in the editing region and the consistency coefficient of the editing region in the three-dimensional bone model are determined.

[0095] In this embodiment, the aforementioned editing region specifically refers to the area specified by the editing operation in the three-dimensional bone model. The electronic device can determine the contribution ratio of the target image within the aforementioned editing region. The contribution ratio can be expressed as:

[0096] Where J is the total number of all target images corresponding to the above editing area; Let represent the support coefficient of the target image (i.e., the ultrasound image of frame i) for the edit region E, and let represent the support coefficient of the ultrasound image of frame j for the edit region E.

[0097] In this embodiment, the electronic device can also determine the consistency coefficient corresponding to the target image. The consistency coefficient is specifically used to determine that there is a lack of consistency support in the neighboring frames or the global fusion result, that is, the target image has not been verified by the neighboring frames or the global fusion result. Specifically, the electronic device can determine the above consistency coefficient based on the image similarity between the target image and the adjacent images, or it can calculate the above consistency coefficient in other ways.

[0098] In S2033.2, if the contribution ratio is greater than the first threshold and the consistency coefficient is less than the second threshold, the segmentation anomaly coefficient is calculated based on the anomaly scores of the target image in multiple segmentation dimensions.

[0099] In this embodiment, the segmentation dimensions include at least one of the first to seventh segmentation dimensions. Specifically, the seven segmentation dimensions can be: The first segmentation dimension is determined based on the area of ​​the connected components corresponding to the edited region and the shape score of the connected components. Specifically, this first segmentation dimension is used to determine whether the pixels affected by the editing operation belong to connected components that are too small or have abnormal shapes. Specifically, it can be determined by calculating the cumulative number of pixels within the connected components.

[0100] The second segmentation dimension is determined based on the aspect ratio of the connected components. This second segmentation dimension is specifically used to determine whether the target image is a thin, elongated artifact or a posterior bone tail.

[0101] The third segmentation dimension is determined based on the distance between the editing region and the main bone surface region in the 3D bone model. Specifically, this third segmentation dimension is used to determine whether the region of the target image projected onto the 3D bone model is far from the main bone surface region. The main bone surface region can be the bone surface corresponding to a bone structure with a large area or volume.

[0102] The fourth segmentation dimension is determined based on the distance between the target image and the segmentation boundary corresponding to the 3D bone model. Specifically, this fourth segmentation dimension is used to determine whether pixels in the target image are located at unstable boundary positions.

[0103] The fifth segmentation dimension is determined based on the distance between a pixel in the target image and the first bone surface point, as well as the distance between that pixel and the segmentation line of the last bone surface. Specifically, this fifth segmentation dimension is used to determine whether a pixel in the target image is the first bone surface point or located in a thick segmented region behind the bone. Specifically, the electronic device can iterate over the target image along the vertical axis y using scan lines, for example, calculating the y-coordinate from top to bottom. For a point p=(u, v), it can determine whether there exists a point p'=(u', v') with a v'. <v。

[0104] The sixth segmentation dimension is determined based on the segmentation confidence of the segmented region where the target image is located. Specifically, this sixth segmentation dimension is used to determine whether the segmentation confidence of the region where the target image is located is low.

[0105] The seventh segmentation dimension is determined based on the distance between the position of the target image's back projection in the 3D bone model and the bone surface. Specifically, this seventh segmentation dimension is used to determine whether the back projection of other frames to that position supports the existence of the bone surface.

[0106] In this embodiment, the electronic device can calculate the anomaly scores corresponding to one or more of the above-mentioned segmentation dimensions, and then perform a weighted calculation based on the weight values ​​corresponding to each segmentation dimension and the above-mentioned anomaly scores, thereby calculating the segmentation anomaly coefficient corresponding to the target image. Specifically, the above-mentioned segmentation anomaly coefficient can be expressed as:

[0107] in, The segmentation anomaly coefficient is Ω; Ω represents the affected two-dimensional pixel region in the target image. This represents the anomaly score corresponding to the nth segmentation dimension; The weight is the weight corresponding to the nth segmentation dimension.

[0108] In S2034, if the segmentation anomaly coefficient is greater than or equal to the second anomaly threshold, then the calibration strategy corresponding to the three-dimensional bone model is determined to be the second strategy.

[0109] In this embodiment, if the electronic device recognizes that the target image is not pose abnormal, it can further determine whether there are local anomalies in its two-dimensional segmentation results, that is, whether the aforementioned segmentation anomaly coefficient is greater than or equal to the second prediction threshold. For example, if the user-initiated editing operation is a deletion operation, and if the three-dimensional region deleted by the user is mainly contributed by a few frames, while most neighboring frames do not support the region, it can be determined that the few frames have a risk of false positive segmentation, that is, the region needs to be re-segmented to improve the support of neighboring frames for the region and avoid the segmentation result of the entire region being affected by a few abnormal frames. In this case, the electronic device can use a second strategy to calibrate all segmentation anomaly images with segmentation anomaly coefficients greater than the second anomaly threshold.

[0110] In some possible implementations, the process of executing the second strategy can specifically be as follows: if the calibration strategy is the second strategy, then delete the two-dimensional segmentation region corresponding to the target image and the second three-dimensional data of the two-dimensional segmentation region in the three-dimensional bone model, and update the three-dimensional bone model based on the deleted second three-dimensional data. Specifically, when the electronic device detects that the segmentation anomaly coefficient of the target image is greater than or equal to the second anomaly threshold, i.e. and When the target image is located in region Ω of the 3D bone model, it is determined that there is a local segmentation anomaly. At this point, the electronic device can delete the corresponding 2D segmented region of the target image and also delete the second 3D data of the 2D segmented region in the 3D bone model.

[0111] In S2035, if the segmentation anomaly coefficient is less than the second anomaly threshold, then the quality anomaly coefficient is calculated based on the model parameters of the target image in the three-dimensional bone model.

[0112] In this embodiment, the electronic device can identify imaging quality anomalies when the segmentation anomaly coefficients corresponding to each target image are all less than the second anomaly threshold. The electronic device can determine whether the target image has imaging quality anomalies, ultrasound artifacts, or low-quality frames by judging whether the consistency of the projection of the entire frame or a large area in the target image with multiple adjacent images in three-dimensional space is poor. Specifically, the electronic device can calculate the anomaly scores corresponding to multiple quality dimensions based on the model parameters corresponding to the target image in the three-dimensional bone model. Based on the anomaly scores corresponding to each quality dimension, the quality anomaly coefficient corresponding to the target image can be calculated. The aforementioned model parameters may specifically include the bone surface echo response intensity, bone surface line continuity, bone surface segmentation thickness, number of multi-layer structures, image noise level, proportion of shadow areas or strong artifact areas, the proportion of multiple local areas in the target image that have been deleted by the user or judged as abnormal by the system, and the consistency of the three-dimensional reconstruction results between the target image and adjacent images.

[0113] That is, the aforementioned quality anomaly coefficient can be calculated based on one or more corresponding anomaly scores in the following imaging quality dimensions. Specifically, the aforementioned imaging quality dimensions may include at least one of the following: The first imaging dimension is determined based on the echo response intensity of the target image on the bone surface in the three-dimensional bone model. The second imaging dimension is determined based on the continuous score of the bone surface lines in the three-dimensional bone model of the target image. The third imaging dimension is determined based on the bone surface segmentation thickness of the target image in the three-dimensional bone model. The fourth imaging dimension is determined based on the number of multi-layer structures corresponding to the three-dimensional bone model. The fifth imaging dimension; the fifth imaging dimension is determined based on the image noise of the target image; The sixth imaging dimension; the sixth imaging dimension is determined based on the area of ​​the shadow region of the target image; The seventh imaging dimension is determined based on the number of abnormal pixels in the target image; the number of abnormal pixels is the pixels deleted according to the editing operation and / or the pixels identified by a preset anomaly algorithm.

[0114] In this embodiment, the above-mentioned quality anomaly coefficient can be expressed as:

[0115] in, The quality anomaly coefficient corresponding to the target image; These are the weighting coefficients corresponding to the nth imaging quality dimension. This represents the anomaly score corresponding to the nth imaging quality dimension.

[0116] In S2036, if the quality anomaly coefficient is greater than or equal to the third anomaly threshold, then the calibration strategy corresponding to the three-dimensional bone model is determined to be the third strategy.

[0117] In this embodiment, the electronic device can identify target images with a quality anomaly coefficient greater than or equal to a third threshold as quality anomaly images, and calibrate all quality anomaly images in the target data through a third strategy, thereby reducing the contribution of the aforementioned poor imaging quality images to the three-dimensional bone model and improving the accuracy of the three-dimensional bone model.

[0118] In some possible implementations, the process of executing the third strategy can specifically be as follows: if the calibration strategy is the third strategy, then delete the target image. Specifically, the electronic device deletes the target image when it detects that the quality anomaly coefficient of the target image is greater than or equal to the third anomaly threshold. When the coverage ratio of abnormal areas exceeds a threshold, the target image can be identified as an image with abnormal imaging quality, and the target image can be set to unusable, for example, by deleting the target image or reducing the contribution weight of the target image to the three-dimensional bone model.

[0119] In this embodiment, the electronic device can classify target images with abnormal target data by using pose abnormality coefficient, segmentation abnormality coefficient and quality abnormality coefficient, and then execute different calibration strategies according to different abnormality types.

[0120] Furthermore, the electronic device can identify different anomaly types hierarchically based on their computational difficulty and impact on the 3D bone model. For example, when an anomaly is identified, all images with anomalies in pose are calibrated first, then segmentation anomalies are identified and calibrated, and finally imaging quality anomalies are identified, thus achieving sequential calibration of each anomaly type. For example, Figure 4 A schematic diagram of the calibration process for various anomaly types provided in an embodiment of this application is shown. See also Figure 4 As shown, the above calibration process may specifically include the following steps: In S41, the user's editing operations are received.

[0121] In S42, the target 3D data included in the user's editing operation is determined.

[0122] In S43, the target image corresponding to the target 3D data is determined based on the mapping relationship.

[0123] In S44, it is determined whether the target image is in a pose abnormality, that is, whether the pose abnormality coefficient of the target image is greater than or equal to the first abnormality threshold.

[0124] In S45, if so, the target image is calibrated using the first strategy.

[0125] In S46, if not, determine whether the target image is a segmentation anomaly, that is, whether the segmentation anomaly coefficient of the target image is greater than or equal to the second anomaly threshold.

[0126] In S47, if so, the target image is calibrated using the second strategy.

[0127] In S48, if not, determine whether the target image has an imaging quality abnormality, that is, whether the quality abnormality coefficient of the target image is greater than or equal to the third abnormality threshold.

[0128] In S49, if so, the target image is calibrated with the third strategy.

[0129] In S410, the calibration results are output.

[0130] For example, Figure 5 A schematic diagram illustrating a calibration strategy for different anomaly types provided in an embodiment of this application is shown. See also Figure 5 As shown, the above calibration strategy may include: 1. For abnormal pose: the entire frame of ultrasound image can be deleted or the weight of the 3D bone model can be reduced.

[0131] 2. For segmentation anomalies: Local regions in the corresponding two-dimensional segmented image of the ultrasound image can be deleted or modified.

[0132] 3. For imaging artifacts or abnormal imaging quality: reduce the weight of the 3D bone model or delete the entire ultrasound image.

[0133] 4. For missing bone surface segmentation: The corresponding 2D segmented image can be completed and the source information updated. Specifically, the user's interactive operation is an add operation. After the added region is projected onto the 2D ultrasound image, the projected region has bone surface image features but is not segmented and covered.

[0134] 5. For segmentation offset: Correction can be performed along the scan line direction or the boundary can be calibrated based on neighborhood consistency. Specifically, it can be determined whether there is an offset in the same direction in the bone surface position across multiple adjacent scan lines or multiple ultrasound images.

[0135] Figure 6 This diagram illustrates the specific implementation flowchart of a bone data calibration method provided in the third embodiment of this application after step S204. See also... Figure 6 As shown, relative to Figures 2-3 In any of the embodiments described above, the bone data calibration method provided in this application further includes steps S601 to S602 after step S204, as specifically described below: In S601, for any three-dimensional data in the updated three-dimensional bone model, the support corresponding to the three-dimensional data is determined according to the two-dimensional data source corresponding to the three-dimensional data; the two-dimensional data source is determined according to the pixels corresponding to the three-dimensional data in the updated multiple ultrasound images.

[0136] In S602, the updated three-dimensional bone model is adjusted a second time according to the support corresponding to each of the three-dimensional data to obtain the target bone model.

[0137] In this embodiment, after obtaining the calibration results, the electronic device can acquire an updated 3D bone model. The electronic device can then re-perform 3D fusion on the updated 3D bone model to obtain an updated 3D bone surface. For each 3D data point in the updated 3D bone surface, the support corresponding to each 3D data point is recalculated based on the updated 2D data source set. The target bone model is obtained by removing 3D data with support below a threshold.

[0138] Figure 7 A flowchart illustrating the specific implementation of a bone data calibration method S201 provided in the fourth embodiment of this application is shown. See also Figure 7 As shown, relative to Figures 2-3 In any of the embodiments described above, the bone data calibration method provided in this application includes steps S2011 to S2013, as specifically described below: In S2011, the plurality of ultrasound images are acquired.

[0139] In this embodiment, the electronic device can acquire multiple ultrasound images via an ultrasound tracer. Optionally, the electronic device can also acquire a two-dimensional segmented image corresponding to each ultrasound image.

[0140] For example, the image sequence generated from the above multiple ultrasound images can be represented as follows: ,in Let represent the i-th frame of the ultrasound image. The sequence corresponding to the two-dimensional segmented image can be represented as: ,in Represents the pixel coordinates in the i-th frame of the ultrasound image. Does it belong to the bony surface area? It can be a binary labeled image or a probability map.

[0141] In S2012, for any of the ultrasound images, the three-dimensional projection data of each pixel in the ultrasound image in three-dimensional space is determined according to the calibration matrix and the tracking matrix corresponding to the ultrasound image.

[0142] In this embodiment, the electronic device can acquire the tracking matrix corresponding to each ultrasound image, and the tracking matrix corresponding to each ultrasound image is as follows: ,in Let represent the spatial transformation matrix from the coordinate system of the i-th frame of the ultrasound image to the reference coordinate system or the world coordinate system. Here, Ti = [rx, ry, rz | ti]. ti describes the translation vector of the ultrasound image relative to the target object. Rx, ry, and rz are used to determine the rotation vector of the ultrasound image relative to the target object.

[0143] In this embodiment, the electronic device can acquire the calibration matrix C of the ultrasound probe of the ultrasound tracer. This calibration matrix is ​​used to describe the fixed transformation relationship between the two-dimensional ultrasound image coordinate system and the probe coordinate system or tracking coordinate system. For example, for three-dimensional data p=(x, y, z, 1) in three-dimensional space, and the pixel position of its source ultrasound image is (u, v), the projection relationship between the above three-dimensional data and the source pixel can be expressed as (x, y, z, 1) = C(u, v, 0, 1). The calibration matrix is ​​C = (sx, sy, 0, 1). [ex, ey, 0, T]=[sx ex, sy ey, 0, T).

[0144] Where ex and ey represent the rotation relationship between the image plane coordinate system and the probe tracer coordinate system. T represents the translation relationship between the image plane origin and the probe tracer origin. sx and sy are the origin positions.

[0145] In this embodiment, the electronic device can determine the pixel coordinates of the ultrasound image. Based on the pixel spacing, image origin, and ultrasound image coordinate definition, it is converted into two-dimensional ultrasound image physical coordinates. For pixels belonging to the bone surface candidate region, the calibration matrix C and the tracking matrix are used. Project it back into three-dimensional space:

[0146] in, Represents the pixel coordinates in the i-th frame of the ultrasound image. The corresponding three-dimensional point.

[0147] In one implementation, to reduce the impact of posterior bone artifacts and 2D segmentation thickness on 3D reconstruction, only the first-reaching bone surface point can be retained along each scanning direction. For example, if the scanning direction is the v direction, then for each value in the u direction, the smallest v that satisfies the segmentation condition is selected:

[0148] Then only pixels Or its neighborhood is used as a candidate point on the bone surface for three-dimensional back projection, which yields the three-dimensional projection data corresponding to each two-dimensional pixel.

[0149] In S2013, a three-dimensional bone model of the target object is obtained based on the three-dimensional projection data of all pixels in the three-dimensional space; the three-dimensional bone model records the source information corresponding to each three-dimensional projection data; the source information includes pixels of at least one ultrasound image corresponding to the three-dimensional projection data.

[0150] In this embodiment, the electronic device can fuse all ultrasound images, that is, fuse three-dimensional projection data in three-dimensional space to obtain a three-dimensional bone model corresponding to the target object. The electronic device can store the source information corresponding to each three-dimensional projection data, i.e., obtain the mapping relationship corresponding to each three-dimensional projection data. Specifically, the above mapping relationship can be represented as follows:

[0151] Where u and v represent the coordinates of the source pixel of the 3D data; c is the scan line number; l is the number of the 2D segmented connected region to which the source pixel belongs; w represents the contribution weight of the 2D observation to the 3D element; and i is the frame number of the ultrasound image corresponding to the 3D projection data.

[0152] In this embodiment of the application, the three-dimensional bone model constructed by the electronic device can store the source information of each three-dimensional projection data. When the three-dimensional bone model is edited, the original two-dimensional ultrasound image that caused the formation of the three-dimensional projection data can be directly found, thereby avoiding untraceable post-processing only at the three-dimensional level.

[0153] In this embodiment, Figure 8 This diagram illustrates a structural block diagram of a bone data calibration device according to an embodiment of this application. The bone data calibration device includes units for performing... Figure 2 The corresponding embodiment describes the steps implemented by the bone data calibration device. Please refer to [link / reference] for details. Figure 2 and Figure 2 The relevant descriptions in the corresponding embodiments are shown below. For ease of explanation, only the parts relevant to this embodiment are shown.

[0154] See Figure 8 A bone data calibration device, wherein the ultrasound tracer includes an ultrasound probe, the device comprising: The model acquisition unit 81 is used to acquire a three-dimensional bone model of the target object; the three-dimensional bone model is generated based on multiple ultrasound images of the target object. The model editing unit 82 is configured to, in response to an editing operation on the three-dimensional bone model, determine target data in the three-dimensional bone model corresponding to the editing operation; the target data includes target three-dimensional data and a target image located in the three-dimensional bone model; the target image is an ultrasound image used to construct the target three-dimensional data; The strategy determination unit 83 is used to determine the calibration strategy corresponding to the three-dimensional bone model according to the anomaly type corresponding to the target data if the target data meets the preset anomaly calibration conditions; the anomaly type includes at least one of image quality anomaly, pose anomaly, or segmentation anomaly. The strategy execution unit 84 is used to generate calibration results for the target object based on the editing operation and the calibration strategy; the calibration results include an updated three-dimensional bone model and multiple updated ultrasound images.

[0155] It should be understood that, Figure 8 In the structural block diagram of the device shown, each module is used to perform... Figure 2 , Figure 3 , Figure 5 as well as Figure 6 Each step in any corresponding embodiment, and for Figure 2 , Figure 3 , Figure 5 as well as Figure 6 The steps in the corresponding embodiments have been explained in detail in the above embodiments. Please refer to them for details. Figure 2 , Figure 3 , Figure 5 , Figure 6 The relevant descriptions in the embodiments corresponding to the above figures will not be repeated here.

[0156] Figure 9 This is a structural block diagram of an electronic device provided in another embodiment of this application. For example... Figure 9 The electronic device 900 of this embodiment includes a processor 910, a memory 920, and a computer program 930 stored in the memory 920 and executable by the processor 910, such as a program for a bone data calibration method. When the processor 910 executes the computer program 930, it implements the steps of the various embodiments of the bone data calibration methods described above, for example... Figure 2 S201 to S204 are described above. Alternatively, the processor 910 may execute the computer program 930 to implement the above. Figure 8 The functions of each module in the corresponding embodiments, for example, Figure 8 For details regarding the functions of units 81 to 84, please refer to [link / reference needed]. Figure 8 The relevant descriptions in the corresponding embodiments.

[0157] For example, the computer program 930 may be divided into one or more modules, one or more of which are stored in the memory 920 and executed by the processor 910 to complete this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 930 in the electronic device 900. For example, the computer program 930 may be divided into various unit modules, each with the specific functions described above.

[0158] Electronic device 900 may include, but is not limited to, processor 910 and memory 920. Those skilled in the art will understand that... Figure 9 This is merely an example of electronic device 900 and does not constitute a limitation on electronic device 900. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device may also include input / output devices, network access devices, buses, etc.

[0159] The processor 910 can be a central processing unit, or it can be other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0160] The memory 920 can be an internal storage unit of the electronic device 900, such as a hard disk or RAM of the electronic device 900. The memory 920 can also be an external storage device of the electronic device 900, such as a plug-in hard disk, smart memory card, flash memory card, etc., equipped on the electronic device 900. Furthermore, the memory 920 can include both internal storage units and external storage devices of the electronic device 900.

[0161] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for calibrating bone data, characterized in that, include: Obtain a 3D bone model of the target object; The three-dimensional bone model was generated based on multiple ultrasound images of the target object; In response to an editing operation on the three-dimensional bone model, target data corresponding to the editing operation in the three-dimensional bone model is determined; the target data includes target three-dimensional data located in the three-dimensional bone model and a target image; the target image is an ultrasound image used to construct the target three-dimensional data; If the target data meets the preset anomaly calibration conditions, then the calibration strategy corresponding to the three-dimensional bone model is determined according to the anomaly type corresponding to the target data. Specifically, the target data meets the anomaly calibration conditions as follows: any anomaly coefficient is greater than the threshold corresponding to the anomaly type of the anomaly coefficient. The anomaly type includes at least one of image quality anomaly, pose anomaly, or segmentation anomaly. Based on the editing operation and the calibration strategy, a calibration result for the target object is generated; the calibration result includes an updated three-dimensional bone model and multiple updated ultrasound images. The step of generating the calibration result of the target object based on the editing operation and the calibration strategy includes: If the calibration strategy is the first strategy, then the target image is deleted and the first three-dimensional data corresponding to the target image is removed from the three-dimensional bone model, and the three-dimensional bone model is updated based on the deleted first three-dimensional data; the first strategy is the calibration strategy executed when the anomaly type is pose anomaly; If the calibration strategy is the second strategy, then the two-dimensional segmentation region corresponding to the target image and the second three-dimensional data of the two-dimensional segmentation region in the three-dimensional bone model are deleted, and the three-dimensional bone model is updated based on the deleted second three-dimensional data; the second strategy is the calibration strategy executed when the anomaly type is segmentation anomaly; If the calibration strategy is the third strategy, then the target image is deleted; the third strategy is the calibration strategy executed when the anomaly type is image quality anomaly. If the target data meets the preset anomaly calibration conditions, then based on the anomaly type corresponding to the target data, the calibration strategy corresponding to the three-dimensional bone model is determined, including: Based on the target image and the tracking matrix corresponding to the target image, a pose anomaly coefficient is calculated; the tracking matrix is ​​used to project the target image into a three-dimensional space. If the pose abnormality coefficient is greater than or equal to the first abnormality threshold, then the calibration strategy corresponding to the three-dimensional bone model is determined to be the first strategy. If the pose anomaly coefficient is less than the first anomaly threshold, then the segmentation anomaly coefficient is calculated based on the target image and the neighboring images of the target image; If the segmentation anomaly coefficient is greater than or equal to the second anomaly threshold, then the calibration strategy corresponding to the three-dimensional bone model is determined to be the second strategy. If the segmentation anomaly coefficient is less than the second anomaly threshold, then the quality anomaly coefficient is calculated based on the model parameters of the target image in the three-dimensional bone model; If the quality anomaly coefficient is greater than or equal to the third anomaly threshold, then the calibration strategy corresponding to the three-dimensional bone model is determined to be the third strategy.

2. The calibration method according to claim 1, characterized in that, The step of calculating pose anomaly coefficients based on the target image and the tracking matrix corresponding to the target image includes: For any of the target images, determine the translation jump coefficient and rotation jump coefficient corresponding to the target image; the translation jump coefficient is determined based on the translation vector of the tracking matrix of the target image; the rotation jump coefficient is determined based on the rotation matrix of the tracking matrix of the target image. The pose anomaly coefficient is determined based on the pose score corresponding to the target image, the translation jump coefficient, and the rotation jump coefficient; the pose anomaly coefficient is specifically: Among them, the The pose anomaly coefficient is the coefficient of the position. The translation jump coefficient corresponding to the target image; The rotation jump coefficient corresponding to the target image; The pose score corresponding to the target image; Here, represents the basic anomaly parameters corresponding to the target image; norm(x) is the normalization function. , , and is the weighting coefficient; i is the frame number corresponding to the target image; the basic anomaly parameter is any one of the timestamp jump coefficient, tracking anomaly coefficient, or device quality coefficient.

3. The calibration method according to claim 1, characterized in that, If the pose anomaly coefficient is less than the first anomaly threshold, then a segmentation anomaly coefficient is calculated based on the target image and its neighboring images, including: Based on the editing region where the target image is located, determine the contribution ratio of the target image in the editing region and the consistency coefficient of the editing region in the three-dimensional bone model; If the contribution ratio is greater than a first threshold and the consistency coefficient is less than a second threshold, then the segmentation anomaly coefficient is calculated based on the anomaly scores of the target image in multiple segmentation dimensions; the segmentation dimensions include at least one of the following: The first segmentation dimension is determined based on the area of ​​the connected component corresponding to the edited region and the shape score corresponding to the connected component. The second segmentation dimension is determined based on the aspect ratio of the connected components. The third segmentation dimension is determined based on the distance between the editing area and the main bone surface area in the three-dimensional bone model. The fourth segmentation dimension is determined based on the distance between the target image and the segmentation boundary corresponding to the three-dimensional bone model. The fifth segmentation dimension is determined based on the distance between a pixel in the target image and the first bone surface point, and the distance between that pixel and the segmentation line of the last bone surface. The sixth segmentation dimension; the sixth segmentation dimension is determined based on the segmentation confidence of the segmentation region where the target image is located; The seventh segmentation dimension is determined based on the distance between the position of the back projection of the target image in the three-dimensional bone model and the bone surface.

4. The calibration method according to claim 1, characterized in that, If the segmentation anomaly coefficient is less than the second anomaly threshold, then a quality anomaly coefficient is calculated based on the model parameters of the target image in the three-dimensional bone model, including: The quality anomaly coefficient is calculated based on the anomaly scores of the target image in multiple imaging quality dimensions; the imaging quality dimensions include at least one of the following: The first imaging dimension is determined based on the echo response intensity of the target image on the bone surface in the three-dimensional bone model. The second imaging dimension is determined based on the continuous score of the bone surface lines in the three-dimensional bone model of the target image. The third imaging dimension is determined based on the bone surface segmentation thickness of the target image in the three-dimensional bone model. The fourth imaging dimension is determined based on the number of multi-layer structures corresponding to the three-dimensional bone model. The fifth imaging dimension; the fifth imaging dimension is determined based on the image noise of the target image; The sixth imaging dimension; the sixth imaging dimension is determined based on the area of ​​the shadow region of the target image; The seventh imaging dimension is determined based on the number of abnormal pixels in the target image; the number of abnormal pixels is the pixels deleted according to the editing operation and / or the pixels identified by a preset anomaly algorithm.

5. The calibration method according to any one of claims 1-4, characterized in that, After generating the calibration result of the target object based on the editing operation and the calibration strategy, the method further includes: For any three-dimensional data in the updated three-dimensional bone model, the support corresponding to the three-dimensional data is determined based on the two-dimensional data source corresponding to the three-dimensional data; the two-dimensional data source is determined based on the pixels corresponding to the three-dimensional data in the updated multiple ultrasound images. Based on the support level corresponding to each of the three-dimensional data, the updated three-dimensional bone model is adjusted a second time to obtain the target bone model.

6. The calibration method according to any one of claims 1-4, characterized in that, The process of obtaining the three-dimensional bone model of the target object includes: Acquire the multiple ultrasound images; For any of the ultrasound images, the three-dimensional projection data of each pixel in the ultrasound image in three-dimensional space is determined according to the calibration matrix and the tracking matrix corresponding to the ultrasound image. Based on the three-dimensional projection data of all pixels in the three-dimensional space, a three-dimensional bone model of the target object is obtained; the three-dimensional bone model records the source information corresponding to each three-dimensional projection data; the source information includes the pixels of at least one ultrasound image corresponding to the three-dimensional projection data.

7. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the steps of the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images