Medical data display method, apparatus, device, and storage medium

By aligning the data collected by multiple medical devices, multiple data and data changes can be displayed on the interactive interface, solving the problems of cumbersome data switching and inconvenient display during the diagnosis and treatment process, and improving the efficiency of data viewing.

WO2024230596A1PCT designated stage expired Publication Date: 2024-11-14SHINING 3D TECH CO LTD
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/090864
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-09
Filing Date
2024-04-30
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

When a doctor needs to view the data collected by multiple medical devices during the diagnosis and treatment process, he needs to frequently switch different software platforms, resulting in inefficiency and inability to effectively display the changes in data collected by the same medical device.

Method used

Provide a medical data display method, by obtaining data collected by multiple device sources at different time nodes, aligning the data at the same time node in space, and aligning the data at different time nodes of the same device source at different time nodes, realizing The interactive interface displays the data changes of multiple data under a certain time node and a certain device source at different time nodes.

Benefits of technology

It improves data presentation efficiency, facilitates users to view and analyze multi-source data, and solves the problems of cumbersome data switching and inconvenient display of changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024090864_14112024_PF_FP_ABST
    Figure CN2024090864_14112024_PF_FP_ABST
Patent Text Reader

Abstract

A medical data display method, an apparatus, a device, and a storage medium. The method comprises: after acquiring multiple paths of medical data collected by multiple device sources in respect of the same target subject at different time nodes, aligning in space multiple paths of medical data collected by different device sources at the same time node, so as to unify same into one coordinate system; and aligning in space multiple paths of medical data collected by the same device source at different time nodes, so as to unify same into one coordinate system. Thus, the present invention can display on an interaction interface multiple paths of medical data collected by different device sources at a certain time node, and the change conditions of medical data collected by a certain device source at different time nodes, thereby improving data presentation efficiency and facilitating users in checking data.
Need to check novelty before this filing date? Find Prior Art

Description

Medical data display method, device, equipment and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on May 9, 2023, with application number 202310520863.0 and invention name “Medical Data Display Method, Device, Equipment and Storage Medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present disclosure relates to the field of data processing technology, and in particular to a medical data display method, device, equipment and storage medium. Background Art

[0004] During the diagnosis and treatment of patients, it is often necessary to collect data from the same part of the patient using multiple medical devices. This allows for diagnosis based on the multiple medical data collected by these devices. Furthermore, doctors also need to review how the same type of medical data from the same patient changes over time.

[0005] Currently, medical data collected by different medical devices need to be viewed through different software platforms. When doctors view medical data collected by multiple medical devices, they need to constantly switch between different software platforms, which is very cumbersome and greatly affects work efficiency. In addition, the current medical data viewing software cannot display the changes in medical data collected by the same medical device, which is not conducive to doctors' analysis of changes in medical data.

[0006] Summary of the Invention

[0007] The present disclosure provides a medical data display method, apparatus, device and storage medium.

[0008] According to a first aspect of an embodiment of the present disclosure, a medical data display method is provided, the method comprising: obtaining multiple medical data collected by multiple device sources at different time nodes for the same target object; spatially aligning the multiple medical data collected by different device sources at the same time node, and spatially aligning the multiple medical data collected by the same device source at different time nodes; when a user-triggered instruction to display medical data at a target time node is obtained, displaying the aligned multiple medical data collected by different device sources at the target time node on an interactive interface; when a user-triggered instruction to display medical data collected by a target device source is obtained, displaying changes in the medical data collected by the target device source at different time nodes on an interactive interface.

[0009] According to a second aspect of an embodiment of the present disclosure, a medical data display device is provided, the device comprising: an acquisition module configured to acquire multiple medical data collected by multiple device sources at different time nodes for the same target object; an alignment module configured to spatially align multiple medical data collected by different device sources at the same time node, and spatially align multiple medical data collected by the same device source at different time nodes; a display module configured to, when a user-triggered instruction to display medical data at a target time node is obtained, display the aligned multiple medical data collected by different device sources at the target time node on an interactive interface; and, when a user-triggered instruction to display medical data collected by a target device source is obtained, display changes in medical data collected by the target device source at different time nodes on an interactive interface.

[0010] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a processor, a memory, and computer instructions stored in the memory for execution by the processor. When the processor executes the computer instructions, the method mentioned in the first aspect above can be implemented.

[0011] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed, the method mentioned in the first aspect is implemented.

[0012] In an embodiment of the present disclosure, after obtaining multiple medical data collected by multiple device sources for the same target object at different time nodes, the multiple medical data collected by different device sources at the same time node can be spatially aligned and unified into one coordinate system, and the multiple medical data collected by the same device source at different time nodes can be spatially aligned and unified into one coordinate system, so that the multiple medical data collected by different device sources at a certain time node can be displayed on an interactive interface, as well as the changes in medical data collected by a certain device source at different time nodes can be displayed.

[0013] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0015] FIG1 is a schematic diagram of an oral diagnosis and treatment scenario in which multiple equipment sources are used to collect medical data according to an embodiment of the present disclosure.

[0016] FIG2 is a flowchart of a medical data display method according to an embodiment of the present disclosure.

[0017] FIG3 is a schematic diagram of three-dimensional data alignment according to an embodiment of the present disclosure.

[0018] FIG4 is a schematic diagram of a user interaction interface according to an embodiment of the present disclosure.

[0019] FIG5 is a flow chart of a medical data display device according to an embodiment of the present disclosure.

[0020] FIG6 is a schematic diagram of the logical structure of a device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0022] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms "a", "the" and "the" used in this disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items. In addition, the term "at least one" herein means any combination of at least two of any one or more of a plurality of.

[0023] It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining."

[0024] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present disclosure and to make the above-mentioned purposes, features and advantages of the embodiments of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure are further described in detail below with reference to the accompanying drawings.

[0025] During the diagnosis and treatment of patients, it is often necessary to collect data from the same part of the patient using multiple medical devices. This allows for diagnosis based on the multiple medical data collected by these devices. Furthermore, doctors also need to review how the same type of medical data from the same patient changes over time.

[0026] Taking the diagnosis and treatment of oral diseases as an example, the process of oral disease diagnosis and treatment usually involves the collection and analysis of medical data from multiple sources over time. As shown in Figure 1, these medical data come from different equipment sources (for example, oral scanners, facial scanners, CBCT (Cone-Beam Computed Tomography), CT (Computed Tomography), etc.), and the types of data are also very diverse, including two-dimensional data, three-dimensional data, pathological data, etc. For example, the data involved in the diagnosis and treatment of oral diseases include two-dimensional imaging data such as cephalograms, panoramic films, small dental films, high-definition facial photos, and high-definition dental photos; three-dimensional data such as three-dimensional dental scan data, three-dimensional facial scan data, skull CBCT data, and dental CBCT data; pathological data such as cell slice data and blood test reports; and different types of medical data such as mandibular movement data.

[0027] Currently, doctors need to switch to different software platforms to view medical data collected by different devices. For example, 3D oral scan data collected by an oral scanner must be viewed using the software that comes with the oral scanner, and 3D facial scan data collected by a facial scanner must be viewed using the software that comes with the facial scanner. CBCT data of the skull and teeth must be viewed using the software that comes with the CBCT device, and pathology data requires separate software. In most scenarios, these medical data are interconnected, requiring doctors to compare and analyze them simultaneously. This requires doctors to constantly switch between multiple software platforms to view different medical data.

[0028] In addition, for oral disease diagnosis and treatment scenarios, doctors usually need to view how certain medical data changes over time, such as how the shape of a tooth changes with age. Current data viewing software cannot achieve this function, which makes it very inconvenient for doctors to perform data analysis and greatly affects work efficiency.

[0029] It can be seen that for scenarios such as oral disease diagnosis and treatment that involve data analysis across time and multiple device sources, there is an urgent need for a solution that can facilitate users to perform multi-source data analysis and comparison.

[0030] In view of this, the embodiment of the present disclosure provides a medical data display method. After obtaining multiple medical data collected by multiple device sources for the same target object at different time nodes, the multiple medical data collected by different device sources at the same time node can be spatially aligned and unified into the same coordinate system, and the multiple medical data collected by the same device source at different time nodes can be spatially aligned and unified into the same coordinate system. This makes it possible to display the multiple medical data collected by different device sources at a certain time node on an interactive interface, as well as display the changes in medical data collected by a certain device source at different time nodes, thereby improving data presentation efficiency and facilitating user viewing of data.

[0031] The medical data display method provided by the embodiments of the present disclosure can be implemented using self-developed software code with data viewing functionality. For example, in some scenarios, the software code can be presented in the form of data viewing software or a web client with data viewing functionality. Users can import multiple channels of medical data into the data viewing software or web client, which can then process and display the imported medical data.

[0032] Of course, in some scenarios, the software code can also be presented in the form of a medical data management system, which can communicate with multiple medical device sources and obtain medical data from multiple medical device sources. Users can manage and view medical data through the medical data management system.

[0033] The medical data mentioned in the embodiments of the present disclosure may be any type of data required during the diagnosis and treatment process. Taking the oral diagnosis and treatment scenario as an example, the medical data may include cephalograms, panoramic films, small dental films, high-definition facial photos, high-definition dental photos, 3D dental scan data, 3D facial scan data, CBCT skull data, CBCT dental data, cell slice data, blood test reports, mandibular movement data, and the like.

[0034] As shown in FIG2 , it is a schematic diagram of the medical data display method in an embodiment of the present disclosure.

[0035] In step S201, multiple medical data collected by multiple device sources at different time points for the same target object can be obtained. For example, taking the execution subject of this method as data viewing software, the user can import multiple medical data collected by multiple device sources at different time points for the same patient into the data viewing software. Or taking the execution subject of this method as a medical data management system, the medical data management system can be connected to multiple device sources for communication. After the multiple device sources have collected the patient's medical data, they can be automatically transmitted to the medical data management system, and the medical data management system can classify the acquired data according to the user dimension. When it is necessary to display data, the user can select the medical data to be displayed in the data management system, and then process and display it based on the medical data selected by the user.

[0036] In step S202, after acquiring multiple channels of medical data, the multiple channels of medical data collected by different device sources at the same time node can be spatially aligned. Considering that multiple channels of medical data collected by different device sources are usually viewed and analyzed together, and medical data collected by different device sources usually have different coordinate systems, after acquiring multiple channels of medical data, the multiple channels of medical data can be first divided according to the time dimension. For example, medical data collected on the same day can be divided into a group, so that multiple channels of data collected by different device sources at different time nodes can be obtained. Then, the multiple channels of medical data collected by different device sources at the same time node can be spatially aligned and unified into the same coordinate system.

[0037] Considering that medical data collected by different equipment sources for the same target object often have overlapping areas, these overlapping areas can be used to match and align the two data sources. For example, in an oral diagnosis and treatment scenario, an oral scanner is usually used to scan the user's mouth to obtain 3D scan data of the user's mouth. At the same time, CBCT data of the user's mouth is also collected using CBCT. Since both sets of 3D data include the user's teeth, the two sets of 3D data can be matched based on the teeth and unified into the same coordinate system.

[0038] In step S203, considering that users usually need to view the changes of data of the same dimension at different time nodes, after obtaining multiple channels of medical data, the multiple channels of medical data can be divided according to the device dimension, and the medical data collected by the same device source can be divided into a group. Then, the multiple channels of medical data collected by the same device source at different time nodes can be spatially aligned.

[0039] After aligning the multiple channels of medical data, if a user-triggered data display command is detected via the interactive interface, the aligned medical data can be displayed based on the data display command. For example, the user can select which device source data to display, or which time point data to display, on the interactive interface. In step S204, if a user-triggered command to display medical data at a target time point is received, the aligned multiple channels of medical data collected by different device sources at the target time point are displayed on the interactive interface.

[0040] In step S206, if a user-triggered instruction to display the medical data collected by the target device source is obtained, changes in the medical data collected by the target device source at different time points are displayed on the interactive interface.

[0041] For example, suppose multiple medical data sources include data X1, X2, and X3 collected by device source A at time nodes 1, 2, and 3, respectively; data Y1, Y2, and Y3 collected by device source B at time nodes 1, 2, and 3, respectively; and data Z1, Z2, and Z3 collected by device source C at time nodes 1, 2, and 3, respectively. The data X1, Y1, and Z1 collected by different device sources at time node 1 can be aligned. The user can choose to display the aligned X1, Y1, and Z1 at time node 1, or only any two of the aligned X1, Y1, and Z1. Of course, it is also possible to align X1, X2, and X3 collected by device source A at time nodes 1, 2, and 3 to show the changes in the data collected by the device source at different time nodes.

[0042] In some embodiments, considering that medical data collected by different device sources at the same time for the same target object often overlap, when displaying the aligned medical data collected by different device sources at the same time, the overlapping portions of the aligned multiple medical data can be fused before the fused multiple medical data is displayed on the interactive interface. By fusion of the overlapping regions before display, the overlapping portions of the data collected by different device sources can be avoided from being cluttered.

[0043] For example, in the case of oral disease diagnosis and treatment, suppose the data collected at the same time includes oral 3D scan data, facial 3D scan data, and oral CBCT data. Since there is overlap in all three data sets, for example, there are several overlapping teeth in the oral 3D scan data and the facial 3D scan data, and there are also overlapping teeth in the oral 3D scan data and the oral CBCT data. Therefore, it is possible to first align these data and unify them into a single coordinate system, then merge the overlapping areas into a single data set for display. This can avoid errors in the three data sets, resulting in a cluttered final data presentation and affecting the display quality.

[0044] In some embodiments, to better demonstrate the changes in medical data collected by the same device source at different time points, the changes in medical data collected by the same device source at different time points can be displayed through animation. Because the acquisition process is discrete, animation frames corresponding to time points between two adjacent time points can be generated through interpolation animation. For example, morphing technology can be used to generate continuous animation frames to continuously present the data changes between adjacent time points.

[0045] In some embodiments, considering that users need to view the changes of medical data over time at a finer granularity, semantic segmentation processing can also be performed on the medical data collected by the same device source at various time nodes to obtain multiple semantic entities at each time node. Then, the semantic entities are aligned in space, that is, a corresponding relationship between the semantic entities is established. When a user-triggered instruction to display the target semantic entity is obtained, the changes of the target semantic entity at different time nodes can be displayed on the interactive interface.

[0046] For example, in the scenario of oral disease diagnosis and treatment, the three-dimensional data of the user's mouth is usually collected. Since doctors usually need to view the changes of a single tooth over time, the three-dimensional data of the user's mouth can be semantically segmented first to obtain the three-dimensional data of a single tooth. Then align the teeth at different time nodes, that is, establish a corresponding relationship between the teeth with the same number. When the user wants to view the changes of a certain tooth over time, the corresponding display instruction can be triggered. After the instruction is detected, the changes of the tooth over time can be displayed on the interactive interface. For example, the changes of the tooth can be displayed in the form of an animation.

[0047] Of course, in some scenarios, before semantic segmentation of 3D data, texture mapping can be performed on the 3D model represented by the 3D data using a 2D image. After obtaining a 3D model with texture information, semantic segmentation can be performed again, resulting in a single semantic entity that includes texture information. When displaying the changes of a single semantic entity over time, the changes of the semantic entity with texture information can also be displayed.

[0048] In some embodiments, considering scenarios similar to oral disease diagnosis and treatment, there are many types of data to be analyzed, such as multiple channels of three-dimensional data, multiple channels of two-dimensional image data, pathological data, and so on. Therefore, when spatially aligning multiple channels of medical data collected by different equipment sources at the same time node, there may be multiple situations, such as spatially aligning two channels of three-dimensional data, spatially aligning two channels of two-dimensional images, aligning two-dimensional images with three-dimensional data, aligning one channel of pathological data with a two-dimensional image or three-dimensional data, and so on. Among them, aligning a two-dimensional image with three-dimensional data refers to using a two-dimensional image to perform texture mapping processing on a three-dimensional model represented by three-dimensional data, that is, mapping the texture of the two-dimensional image to the three-dimensional model represented by three-dimensional data. Aligning pathological data with a two-dimensional image refers to determining which image area in the two-dimensional image represents the case data of the part, and aligning pathological data with three-dimensional data refers to determining which area of ​​the three-dimensional model represented by the three-dimensional data represents the pathological data.

[0049] In some embodiments, the multi-channel medical data is multi-channel medical data in an oral diagnosis and treatment scenario. As shown in FIG3 , aligning the three-dimensional data with the three-dimensional data includes one or more of the following: spatially aligning the three-dimensional scan data of the user's oral cavity with the three-dimensional scan data of the user's face, spatially aligning the three-dimensional scan data of the user's oral cavity with the CBCT data of the user's oral cavity, and spatially aligning the three-dimensional scan data of the user's face with the CBCT data of the user's skull. In an oral diagnosis and treatment scenario, multiple channels of three-dimensional data are typically collected, including the user's oral cavity three-dimensional scan data, the user's facial three-dimensional scan data, the user's oral CBCT data, and the user's skull CBCT data. By integrating these multiple channels of data, the internal structure of the user's oral cavity can be understood, facilitating the formulation of a treatment plan for the patient. Therefore, these multiple channels of data need to be aligned and displayed.

[0050] In some embodiments, the multi-channel medical data is multi-channel medical data in an oral diagnosis and treatment scenario, and aligning the two-dimensional image with the three-dimensional data (i.e., texture mapping the two-dimensional image data to a three-dimensional model represented by the three-dimensional data) includes one or more of the following situations: mapping the texture of the two-dimensional image of the user's face to a three-dimensional model represented by the three-dimensional scan data of the user's face, and mapping the texture of the two-dimensional image of the user's mouth to a three-dimensional model represented by the three-dimensional scan data of the user's mouth. In an oral diagnosis and treatment scenario, the user's mouth is usually scanned in three dimensions to obtain a three-dimensional model of the mouth. At the same time, the user's face is also scanned in three dimensions to obtain a three-dimensional model of the user's face. Since the three-dimensional model only includes spatial information and does not contain texture information, it is also necessary to collect two-dimensional images of the mouth and face, and texture map the two-dimensional image to the corresponding three-dimensional model to obtain a three-dimensional model of the mouth or face with texture information.

[0051] In some embodiments, the 2D images in multiple channels of medical data may include at least two 2D images captured under different lighting environments. Due to the inconsistent capture environments of the at least two 2D images, the overall brightness of the at least two 2D images may be inconsistent. Directly applying texture mapping from these two 2D images to the corresponding 3D model may result in uneven brightness across different parts of the resulting 3D model, impacting the final rendering. Therefore, in this case, the at least two 2D images can be preprocessed to unify their brightness before the texture mapping of the preprocessed at least two 2D images to the corresponding 3D model is performed.

[0052] In some embodiments, the at least two two-dimensional images include a two-dimensional image of the user's mouth and a two-dimensional image of the user's face. For example, in an oral medical diagnosis scenario, three-dimensional data is usually collected for the user's mouth and the user's face to obtain respective three-dimensional models. At the same time, two-dimensional images can also be collected for the user's mouth and the user's face respectively, and texture mapped to the corresponding three-dimensional models to obtain a three-dimensional model with texture information. Because the light inside the mouth is usually darker than the light on the face, the overall brightness of the two-dimensional image of the user's mouth is usually darker than that of the two-dimensional image of the user's face. If the two-dimensional images are directly texture mapped to the corresponding three-dimensional model, and then the three-dimensional model of the mouth and the three-dimensional model of the face are aligned and fused for display, the brightness of different parts of the final fused three-dimensional model will be uneven, affecting the overall effect. Therefore, the two-dimensional images can be pre-processed before texture mapping to unify the brightness of the two-dimensional images. For example, the brightness of the darker image can be improved overall.

[0053] In some embodiments, the multi-channel medical data may include at least two 2D images captured by cameras with different resolutions. Due to the different resolutions of the at least two 2D images, after texture mapping the at least two 2D images onto the corresponding 3D model, the resulting overall model may have inconsistent texture information resolutions in different areas. For example, some areas may be sharp while others may be blurry, affecting the overall visual effect. To avoid this issue, the at least two 2D images may be preprocessed to unify their resolutions before texture mapping them onto the corresponding 3D model.

[0054] In some embodiments, the at least two two-dimensional images include a two-dimensional image of a designated part captured by a single-lens reflex camera and a two-dimensional image of a designated part captured by a three-dimensional scanning device, wherein the designated part includes the user's face and / or the user's mouth.

[0055] For example, during dental treatment, a 3D scanning device is often used to collect 3D data of a user's oral cavity and / or face. The 3D scanning device can use two cameras to capture 2D images of the user's oral cavity and / or face from different perspectives. These 2D images are then used to perform 3D reconstruction to produce a 3D model. These 2D images can also be used to perform texture mapping on the 3D model, resulting in a 3D model with texture information. Furthermore, a single-lens reflex camera can be used to capture high-definition images of the user's oral cavity and / or face. For example, a high-definition image of the face is typically captured. Because the resolution of high-definition facial images captured by a single-lens reflex camera is often higher than the resolution of oral images captured by a 3D scanning device, if the two 2D images are texture mapped separately onto the corresponding 3D model, the resulting model may have inconsistent texture resolutions in different areas, impacting the overall quality. Therefore, the two 3D images can be preprocessed to unify their resolutions. For example, the low-resolution image can be upsampled to increase its resolution. Texture mapping can then be performed on the 3D model using the preprocessed 2D images.

[0056] In some embodiments, it is taken into account that users often have different needs when displaying multiple medical data. For example, sometimes it is necessary to display a collection of multiple medical data collected by multiple device sources at a certain time node, sometimes it is necessary to display only a single data collected by a certain device source at a certain time node, and sometimes it is necessary to display the changes in multiple data collected by a certain device source at multiple time nodes. In order to meet the different needs of users and to respond to user display instructions quickly and conveniently. After spatially aligning the multiple medical data collected by different device sources at the same time node, and spatially aligning the multiple medical data collected by the same device source at different time nodes, an expression model can be generated based on the aligned data. The expression model can include multiple data levels, different data levels include medical data of different dimensions, and data at different data levels can be associated. After generating the expression model, the aligned medical data can be displayed based on the expression model.

[0057] In some scenarios, after aligning multiple medical data, other processing can be performed on the data based on actual needs. For example, in some embodiments, if the multiple medical data includes 3D data, semantic segmentation can be performed on the 3D data to obtain semantic segmentation results, and then a representation model can be generated based on the semantic segmentation results and the data alignment results.

[0058] Taking the oral disease diagnosis and treatment scenario as an example, after obtaining a 3D oral model, further semantic segmentation can be performed on the model to identify individual teeth. Alternatively, semantic segmentation can be performed on the oral CBCT data to identify the upper and lower ossicles. Based on the results of these various processing steps, an expression model can be generated. In other words, the expression model represents the results of various processing steps, such as alignment, semantic segmentation, and texture mapping, on this multi-channel medical data. Once these results are represented using the expression model, data of different dimensions can be displayed based on the user's display needs and the expression model.

[0059] In some embodiments, the multi-channel medical data is medical data in an oral disease diagnosis and treatment scenario. When dividing the data hierarchy of the expression model, the data hierarchy can be divided into the following data hierarchies:

[0060] (1) Spatial 3D grid layer, wherein the spatial 3D grid layer includes grid data obtained by rasterizing the three-dimensional data of the user's head, that is, the layer includes multiple 3D grids and the spatial coordinates corresponding to each 3D grid. When the three-dimensional data of the head is rasterized, it can be divided evenly (the sizes of the minimum 3D grids obtained by division are consistent) or unevenly (the sizes of the minimum 3D grids obtained by division are inconsistent). For example, in some scenarios, the size of the minimum 3D grid can be adaptively adjusted. For example, for areas with relatively dense data, the grid can be divided at a higher resolution, that is, the size of the minimum 3D grid obtained by division can be smaller. For areas with relatively sparse data, the grid can be divided at a higher resolution, that is, the size of the minimum 3D grid obtained by division can be larger. In this way, the position of each medical data in the data-dense area can be accurately represented, and the amount of calculation can be reduced.

[0061] (2) The geometry layer includes the geometric morphology data of the user's oral cavity, such as the shape of the teeth, skull, jaw, and face. The geometric morphology can include information such as the shape and surface texture of various parts of the oral cavity. The spatial position of the geometric morphology data is represented by the grid coordinates in the spatial 3D grid layer. For example, certain 3D grids in the spatial 3D grid layer can be associated with the three-dimensional model of the teeth. The location of the associated 3D grid is the location of the teeth in the entire skull area.

[0062] (3) Attribute layer, which includes attribute data of the user's oral cavity, wherein the spatial position of the attribute data is represented by the grid coordinates in the spatial 3D grid layer; for example, the attribute data (such as pathological data) can be associated with the grid in the 3D grid layer, and the position of the associated 3D grid is the position of the part to which the attribute data belongs in the head area.

[0063] (4) Semantic element layer, which includes geometric shape data and attribute data corresponding to each semantic entity in the user's mouth, wherein the spatial position of each semantic entity is represented by the grid coordinates in the spatial 3D grid layer. Each semantic entity can be obtained by semantically segmenting the collected three-dimensional data or two-dimensional data. For example, the three-dimensional scanning data of the oral cavity can be semantically segmented into individual tooth entities. Each tooth entity can be represented based on the geometric shape data and attribute data of the oral cavity. Each semantic entity can be associated with a grid in the 3D grid layer, and the position of the associated 3D grid is the position of the semantic entity in the skull area.

[0064] Of course, the embodiment of the present disclosure divides multiple medical data into the above four data levels based on the general needs of users when viewing data. In actual applications, the division can also be performed in other ways and is not limited to the four data levels listed above.

[0065] In some embodiments, the geometric morphology data may include one or more of the following: 3D scan data of the user's mouth, 3D scan data of the user's face, CBCT data of the user's mouth, CBCT data of the user's skull, texture information of the user's mouth, and texture information of the user's face. The texture information of the user's mouth is aligned with the 3D model corresponding to the 3D scan data of the user's mouth, and the texture information of the user's face is aligned with the 3D model corresponding to the 3D scan data of the user's face. The 3D scan data and texture information can be displayed in different layers. When displaying the geometric morphology data, the 3D model can be displayed without or with texture information based on the user's selection.

[0066] In some embodiments, the attribute data may include one or more of the following: two-dimensional image data, such as an image of the user's mouth, an image of the user's face, a cephalogram, etc., pathological data, and physiological slice data.

[0067] In some embodiments, the semantic entity may include one or more of the following: a single tooth, a maxillary bone, a mandibular bone, etc.

[0068] In some embodiments, after the above-mentioned expression model is generated based on the results of various processing (alignment, semantic segmentation, etc.) of multiple medical data, the data of some or all data layers in the expression model can be displayed based on the display instructions triggered by the user. For example, in some scenarios, the user only wants to view the geometric shape data, that is, only the data of the geometric layer can be displayed. If the user only wants to view the attribute data, then only the data of the attribute layer can be displayed. In some embodiments, in order to display multiple medical data in a more detailed manner, the medical data of each data layer can be divided into multiple layers, and the user can choose to display the medical data of one or more layers. For example, for the semantic element layer, since this layer includes the geometric shape data and attribute data of multiple semantic entities, the geometric shape data of each semantic entity can be divided into one layer, and the attribute data of each semantic entity can also be divided into one layer. When the user only selects the layer corresponding to the geometric shape data of a semantic entity, only the geometric shape of the semantic entity is displayed. When the user selects the layer corresponding to the geometric shape data and the layer corresponding to the attribute data of a semantic entity, the data of both layers of the semantic entity can be displayed simultaneously. For example, you can display only the 3D model of a certain tooth, or display the 3D model of a certain tooth with texture information.

[0069] Of course, the layer settings can be specifically set based on actual display needs. By showing or hiding layers, it is convenient for users to view medical data of different dimensions.

[0070] In some embodiments, as shown in FIG4 , in order to facilitate user operations, multiple data levels of the expression model and layer identifiers of multiple layers under each data level can be displayed on the interactive interface. The user can choose to display or hide these data levels or the layers under each data level, and then the medical data corresponding to the layer selected by the user is displayed on the interactive interface based on the user's layer selection operation. In some embodiments, for the layer not selected by the user, the medical data corresponding to the layer can be made transparent or semi-transparent before being displayed. For example, if the user wants to view the information of a certain tooth, the corresponding layer of the tooth can be selected. At this time, if the layers corresponding to other teeth are set to transparent, the interactive interface will only display this tooth. If the layers corresponding to other teeth are set to semi-transparent, the interactive interface will also display other teeth, but they will be displayed in a darker form. This display method makes it easier for users to understand the position of the current tooth in the entire mouth.

[0071] It is not difficult to understand that the solutions described in the above embodiments can be combined when there is no conflict, and they are not listed one by one in the embodiments of this disclosure.

[0072] Correspondingly, the embodiment of the present disclosure also provides a medical data display device, as shown in Figure 5, the device 50 includes: an acquisition module 51, used to acquire multiple medical data collected by multiple device sources for the same target object at different time nodes; an alignment module 52, used to spatially align multiple medical data collected by different device sources at the same time node, and spatially align multiple medical data collected by the same device source at different time nodes; a display module 53, used to display the aligned multiple medical data collected by different device sources at the target time node on an interactive interface when a user-triggered instruction to display medical data at the target time node is obtained; and when a user-triggered instruction to display medical data collected by the target device source is obtained, display the changes in the medical data collected by the target device source at different time nodes on an interactive interface.

[0073] The specific steps of the task processing method executed by the above-mentioned device can be referred to the description in the above-mentioned method embodiment, which will not be repeated here.

[0074] Furthermore, an embodiment of the present disclosure also provides a device, as shown in Figure 6, which includes a processor 61, a memory 62, and computer instructions stored in the memory 62 for execution by the processor 61. When the processor 61 executes the computer instructions, it implements any one of the methods in the above embodiments.

[0075] An embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which implements the method described in any of the aforementioned embodiments when the program is executed by a processor.

[0076] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0077] Through the description of the above implementation methods, it can be seen that those skilled in the art can clearly understand that the embodiments of the present disclosure can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the technical solution of the embodiments of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments of the present disclosure.

[0078] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0079] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the functions of each module can be implemented in the same one or more software and / or hardware when implementing the embodiment of the present disclosure. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the embodiment. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0080] The above is only a specific implementation of the embodiment of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiment of the present disclosure. These improvements and modifications should also be regarded as the scope of protection of the embodiment of the present disclosure. Industrial Applicability

[0081] The medical data display method provided by the embodiment of the present disclosure can realize the display of multiple medical data collected by different device sources at a certain time node on an interactive interface, as well as the display of changes in medical data collected by a certain device source at different time nodes, thereby improving data presentation efficiency, facilitating user viewing of data, and having strong industrial practicality.

Claims

1. A medical data display method, the method include: Acquire multiple medical data collected by multiple equipment sources at different time points for the same target object; Spatially align multiple medical data collected by different equipment sources at the same time node, and spatially align multiple medical data collected by the same equipment source at different time nodes; When a user-triggered instruction to display medical data at a target time node is obtained, the aligned multiple medical data collected by different device sources at the target time node are displayed on the interactive interface; When a user-triggered instruction to display the medical data collected by the target device source is obtained, changes in the medical data collected by the target device source at different time nodes are displayed on the interactive interface.

2. The method according to claim 1, wherein the multiple medical data collected by different equipment sources at the target time node after alignment are displayed on the interactive interface. include: The overlapping parts of the aligned multiple medical data are fused, and the fused multiple medical data are displayed on the interactive interface.

3. The method according to claim 2, wherein the interactive interface displays the changes in the medical data collected by the target device source at different time points. include: The changes in the medical data collected by the target device source at different time nodes are displayed in the form of animation, wherein the animation frames corresponding to the time points between two adjacent time nodes are generated by interpolation animation.

4. The method according to claim 1, wherein the multi-channel medical data includes three-dimensional data of the target object collected by a designated device source at different time points, and the method further include: For the three-dimensional data at each time node, semantic segmentation is performed on the three-dimensional data to obtain multiple semantic entities; Align multiple semantic entities at different time nodes in space; When a display instruction of a target semantic entity triggered by a user is obtained, changes of the target semantic entity at different time nodes are displayed on the interactive interface.

5. According to the method according to any one of claims 1 to 4, the spatial alignment of multiple medical data collected by different device sources at the same time node includes one or more of the following situations: Spatially align at least two channels of three-dimensional data collected by different device sources at the same time node; Spatially align at least two two-dimensional images collected by different device sources at the same time node; Mapping at least one two-dimensional image texture collected by different device sources at the same time node to a three-dimensional model represented by at least one three-dimensional data; At least one pathological data collected by different equipment sources at the same time node is aligned with at least one two-dimensional image or three-dimensional data.

6. According to the method of claim 5, the multiple medical data are medical data in an oral diagnosis and treatment scenario, and the spatial alignment of at least two three-dimensional data includes one or more of the following situations: spatially aligning the three-dimensional scan data of the user's oral cavity and the three-dimensional scan data of the user's face; ; spatially aligning the three-dimensional scan data of the user's oral cavity with the CBCT data of the user's oral cavity; The 3D scan data of the user's face and the CBCT data of the user's skull are spatially aligned.

7. According to the method described in claim 5 or 6, the multiple medical data are medical data in an oral diagnosis and treatment scenario, and the mapping of at least one two-dimensional image texture to a three-dimensional model represented by at least one three-dimensional data includes one or more of the following situations: mapping the two-dimensional image texture of the user's face to a three-dimensional model represented by three-dimensional scanning data of the user's face; mapping the two-dimensional image texture of the user's mouth to a three-dimensional model represented by three-dimensional scanning data of the user's mouth.

8. The method according to any one of claims 5 to 7, wherein the multiple medical data includes at least two two-dimensional images acquired under different lighting environments, and before the at least two two-dimensional images are texture mapped to the three-dimensional model represented by at least one three-dimensional data, the method further comprises: include: The at least two two-dimensional images are preprocessed to unify the brightness of the at least two two-dimensional images.

9. According to the method of claim 5, the at least two two-dimensional images include a two-dimensional image of the user's mouth and a two-dimensional image of the user's face.

10. The method according to claim 5, wherein the multiple medical data comprises at least two two-dimensional images acquired by photographing devices with different resolutions, and before the at least two two-dimensional images are texture mapped to the three-dimensional model represented by at least one three-dimensional data, the method further comprises: include: The at least two two-dimensional images are preprocessed to unify the resolutions of the at least two two-dimensional images.

11. According to the method of claim 10, the at least two two-dimensional images include a two-dimensional image of a designated part taken by a single-lens reflex camera, and a two-dimensional image of the designated part collected by a three-dimensional scanning device, and the designated part includes a user's face and / or a user's mouth.

12. The method according to claim 1 further comprises: spatially aligning multiple medical data collected by different equipment sources at the same time node, and / or spatially aligning multiple medical data collected by the same equipment source at different time nodes. include: generating an expression model based on the aligned data, wherein the expression model includes multiple data levels, and different data levels include medical data of different dimensions; The aligned medical data is displayed based on the expression model.

13. The method according to claim 12, wherein the multi-channel medical data further comprises three-dimensional data of the target object, and the method further comprises include: Performing semantic segmentation processing on the three-dimensional data to obtain a semantic segmentation result; The step of generating an expression model based on the aligned data comprises: An expression model is generated based on the aligned data and the semantic segmentation result.

14. The method according to claim 12, wherein the multi-channel medical data is medical data in an oral diagnosis and treatment scenario, and the expression model includes the following data levels: A spatial 3D grid layer, wherein the spatial 3D grid layer includes grid data obtained by rasterizing three-dimensional data of the user's head; A geometric layer, the geometric layer includes geometric shape data of the user's mouth, wherein: The spatial position of the geometric shape data is represented by grid coordinates in the spatial 3D grid layer; an attribute layer, the attribute layer including attribute data of the user's oral cavity, wherein the spatial position of the attribute data is represented by grid coordinates in the spatial 3D grid layer; A semantic element layer, wherein the semantic element layer includes geometric shape data and attribute data corresponding to each semantic entity in the user's mouth, wherein the spatial position of each semantic entity is represented by a grid coordinate in the spatial 3D grid layer.

15. According to the method of claim 14, the geometric morphology data includes one or more of the following: three-dimensional scanning data of the user's mouth, three-dimensional scanning data of the user's face, CBCT data of the user's mouth, CBCT data of the user's skull, CBCT data of the user's jaw, texture information of the user's mouth, and texture information of the user's face.

16. The method according to claim 14 or 15, wherein the attribute data comprises one or more of the following: two-dimensional image data, pathological data, and physiological slice data.

17. The method according to any one of claims 14 to 16, wherein the semantic entity comprises one or more of the following: a single tooth, a maxillary bone, and a mandibular bone.

18. The method according to claim 12, wherein the aligned data is displayed based on the expression model. include: Medical data of some or all of the multiple data levels of the expression model are displayed based on a display instruction triggered by the user.

19. According to the method of claim 12, the medical data of each data level can be displayed through multiple layers, and the user can choose to display the medical data of one or more layers.

20. The method according to claim 19, wherein the aligned data is displayed based on the expression model. include: In response to the user's layer selection operation on the interactive interface, medical data corresponding to the layer selected by the user is displayed on the interactive interface, wherein the interactive interface displays multiple data levels of the expression model and layer identifiers of multiple layers under each data level.

21. The method according to claim 20, further comprising: include: For the layers not selected by the user, the medical data corresponding to the layers are made transparent or semi-transparent before being displayed.

22. A medical data display device, the device include: An acquisition module is configured to acquire multiple medical data collected by multiple device sources at different time points for the same target object; An alignment module is configured to spatially align multiple medical data collected by different device sources at the same time node, and spatially align multiple medical data collected by the same device source at different time nodes; A display module is configured to display the aligned multiple medical data collected by different device sources at the target time node on the interactive interface when an instruction to display the medical data at the target time node is obtained triggered by the user; And when a user-triggered instruction to display the medical data collected by the target device source is obtained, the changes in the medical data collected by the target device source at different time nodes are displayed on the interactive interface.

23. An electronic device, comprising a processor, a memory, and computer instructions stored in the memory and executable by the processor, wherein the processor implements the method according to any one of claims 1 to 21 when executing the computer instructions.

24. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 21.

Citation Information

Patent Citations

  • Head-based augmented display method, system and equipment and storage medium

    CN113823000A

  • Tooth model generation method and device, electronic equipment and storage medium

    CN115457198A

  • Orthodontic treatment monitoring method, device and equipment and storage medium

    CN115732094A

  • Orthodontic treatment monitoring method, device and equipment and storage medium

    CN115910388A

  • Digital model comparison method, device, equipment and medium

    CN116128938A