Medical data display method and device, equipment and storage medium
By aligning and displaying the data collected by multiple medical devices, the problem of inefficient data viewing during the diagnosis and treatment process is solved, efficient comparison and analysis of multi-source data is achieved, and work efficiency is improved.
Patent Information
- Application Number
- CN202310520863.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-09
- Publication Date
- 2024-11-12
AI Technical Summary
When a doctor needs to view the data collected by multiple medical devices during the diagnosis and treatment process, he needs to frequently switch different software platforms, resulting in inefficiency and inability to effectively display the changes in data collected by the same medical device.
Provide a medical data display method, by obtaining data collected by multiple device sources at different time nodes, performing spatial alignment, and unifying them to the same coordinate system, realizing the display of multiple data and a device under a time node in the interactive interface. Data changes in the source at different time nodes.
It improves data presentation efficiency, facilitates users to view and analyze multi-source data, and simplifies the data comparison and analysis process of doctors during diagnosis and treatment.
Smart Images

Figure CN118942602A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing technology, and in particular to a medical data display method, device, equipment and storage medium. Background Art
[0002] In the process of diagnosing and treating patients, it is usually necessary to collect data from the same part of the patient through multiple medical devices, so as to diagnose the patient based on multiple medical data collected by multiple medical devices. At the same time, for the same type of medical data of the same patient, doctors also need to view the changes of the medical data over time.
[0003] At present, medical data collected by different medical devices need to be viewed through different software platforms. When doctors view medical data collected by multiple medical devices, they need to constantly switch between different software platforms, which is very cumbersome and greatly affects work efficiency. In addition, the current medical data viewing software is unable to display the changes in medical data collected by the same medical device, which is not conducive to doctors' analysis of changes in medical data. Summary of the invention
[0004] The present invention provides a medical data display method, device, equipment and storage medium.
[0005] According to a first aspect of an embodiment of the present disclosure, a medical data display method is provided, the method comprising:
[0006] Acquire multiple medical data collected by multiple equipment sources at different time points for the same target object;
[0007] Spatially align multiple medical data collected by different equipment sources at the same time node, and spatially align multiple medical data collected by the same equipment source at different time nodes;
[0008] When a user-triggered instruction to display medical data at a target time node is obtained, the aligned multiple medical data collected by different device sources at the target time node are displayed on the interactive interface;
[0009] When a user-triggered instruction to display the medical data collected by the target device source is obtained, changes in the medical data collected by the target device source at different time nodes are displayed on the interactive interface.
[0010] According to a second aspect of an embodiment of the present disclosure, a medical data display device is provided, the device comprising:
[0011] An acquisition module, used to acquire multiple medical data collected by multiple device sources at different time points for the same target object;
[0012] An alignment module, used to spatially align multiple medical data collected by different device sources at the same time node, and to spatially align multiple medical data collected by the same device source at different time nodes;
[0013] A display module is used to display multiple medical data collected by different device sources at the target time node after alignment on an interactive interface when a user-triggered instruction to display medical data at the target time node is obtained; and to display changes in medical data collected by the target device source at different time nodes on an interactive interface when a user-triggered instruction to display medical data at the target time node is obtained.
[0014] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a processor, a memory, and computer instructions stored in the memory and executable by the processor, wherein when the processor executes the computer instructions, the method mentioned in the first aspect above can be implemented.
[0015] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed, the method mentioned in the first aspect is implemented.
[0016] In the disclosed embodiment, after obtaining multiple medical data collected by multiple device sources at different time nodes for the same target object, the multiple medical data collected by different device sources at the same time node can be spatially aligned and unified into one coordinate system, and the multiple medical data collected by the same device source at different time nodes can be spatially aligned and unified into one coordinate system. This allows the multiple medical data collected by different device sources at a certain time node to be displayed on an interactive interface, as well as the changes in medical data collected by a certain device source at different time nodes, thereby improving data presentation efficiency and facilitating users to view data.
[0017] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.
[0019] Figure 1 This is a schematic diagram of an oral treatment scenario in which multiple device sources are used to collect medical data in an embodiment of the present disclosure.
[0020] Figure 2 This is a flowchart of a medical data display method according to an embodiment of the present disclosure.
[0021] Figure 3 It is a schematic diagram of three-dimensional data alignment according to an embodiment of the present disclosure.
[0022] Figure 4 It is a schematic diagram of a user interaction interface of an embodiment of the present disclosure.
[0023] Figure 5 It is a flow chart of a medical data display device according to an embodiment of the present disclosure.
[0024] Figure 6 It is a schematic diagram of the logical structure of a device according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0025] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0026] The terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items. In addition, the term "at least one" herein means any combination of at least two of any one or more of a plurality of.
[0027] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0028] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present disclosure and to make the above-mentioned purposes, features and advantages of the embodiments of the present disclosure more obvious and understandable, the technical solutions in the embodiments of the present disclosure are further described in detail below in conjunction with the accompanying drawings.
[0029] In the process of diagnosing and treating patients, it is usually necessary to collect data from the same part of the patient through multiple medical devices, so as to diagnose the patient based on multiple medical data collected by multiple medical devices. At the same time, for the same type of medical data of the same patient, doctors also need to view the changes of the medical data over time.
[0030] Taking the diagnosis and treatment of oral diseases as an example, the diagnosis and treatment of oral diseases usually involves the collection and analysis of medical data from multiple sources over time, such as Figure 1 As shown, these medical data come from different equipment sources (such as oral scanners, facial scanners, CBCT, CT, etc.), and the types of data are also very diverse, including two-dimensional data, three-dimensional data, pathological data, etc. For example, the data involved in the diagnosis and treatment of oral diseases include two-dimensional imaging data such as cephalometric lateral films, panoramic films, small dental films, high-definition facial photos, and high-definition dental photos; three-dimensional data such as three-dimensional dental scanning data, three-dimensional facial scanning data, skull CBCT data, and dental CBCT data; pathological data such as cell section data and blood test reports; and different types of medical data such as mandibular movement data.
[0031] Currently, when doctors view medical data collected by different devices, they need to switch to different software platforms to view them. For example, the oral 3D scanning data collected by the oral scanner needs to be viewed through the software that comes with the oral scanner, and the facial 3D scanning data collected by the facial scanner needs to be viewed through the software that comes with the facial scanner. The skull CBCT data and tooth CBCT data need to be viewed through the software that comes with the CBCT device, and the pathology data needs to be viewed through another software. In most scenarios, these medical data are related to each other, and doctors need to compare and analyze these medical data at the same time, which causes doctors to constantly switch back and forth between multiple software platforms to view different medical data.
[0032] In addition, for oral disease diagnosis and treatment scenarios, doctors usually need to view how certain medical data changes over time, such as how the shape of a tooth changes with age. Current data viewing software is unable to achieve this function, making it very inconvenient for doctors to perform data analysis, greatly affecting work efficiency.
[0033] It can be seen that for scenarios such as oral disease diagnosis and treatment that involve data analysis across time and multiple device sources, there is an urgent need for a solution that can facilitate users to perform multi-source data analysis and comparison.
[0034] In view of this, the embodiments of the present disclosure provide a medical data display method. After obtaining multiple medical data collected by multiple device sources for the same target object at different time nodes, the multiple medical data collected by different device sources at the same time node can be spatially aligned and unified into the same coordinate system, and the multiple medical data collected by the same device source at different time nodes can be spatially aligned and unified into the same coordinate system. This allows the multiple medical data collected by different device sources at a certain time node to be displayed on an interactive interface, as well as the changes in medical data collected by a certain device source at different time nodes, thereby improving data presentation efficiency and facilitating users to view data.
[0035] The medical data display method provided by the embodiment of the present disclosure can be implemented by a self-developed software code with a data viewing function. For example, in some scenarios, the software code can be presented in the form of a data viewing software, or can also be presented in the form of a web client with a data viewing function, and the user can import multiple medical data into the data viewing software or web client by himself, and the data viewing software or web client can process and display the imported medical data.
[0036] Of course, in some scenarios, the software code can also be presented in the form of a medical data management system, which can communicate with multiple medical device sources and obtain medical data from multiple medical device sources. Users can manage and view medical data through the medical data management system.
[0037] The medical data mentioned in the embodiments of the present disclosure may be various types of data required in the diagnosis and treatment process. Taking the oral diagnosis and treatment scenario as an example, the medical data may be cephalometric lateral radiographs, panoramic radiographs, small dental radiographs, high-definition facial photos, high-definition dental photos, three-dimensional dental scan data, three-dimensional facial scan data, skull CBCT data, dental CBCT data, cell slice data, blood test reports, mandibular movement data, etc.
[0038] like Figure 2 Shown is a schematic diagram of the medical data display method in an embodiment of the present disclosure.
[0039] In step S201, multiple medical data collected by multiple device sources at different time points for the same target object can be obtained. For example, taking the execution subject of the method as data viewing software, the user can import multiple medical data collected by multiple device sources at different time points for the same patient in the data viewing software. Or taking the execution subject of the method as a medical data management system, the medical data management system can be connected to multiple device sources for communication. After multiple device sources have collected the patient's medical data, they can be automatically transmitted to the medical data management system, and the medical data management system can classify the acquired data according to the user dimension. When it is necessary to display data, the user can select the medical data to be displayed in the data management system, and then process and display it based on the medical data selected by the user.
[0040] In step S202, after acquiring multiple channels of medical data, the multiple channels of medical data collected by different device sources at the same time node can be spatially aligned. Considering that multiple channels of medical data collected by different device sources are usually put together for viewing and analysis, and the medical data collected by different device sources usually have different coordinate systems, after acquiring multiple channels of medical data, the multiple channels of medical data can be first divided according to the time dimension, for example, the medical data collected on the same day can be divided into a group, so that multiple channels of data collected by different device sources at each time node can be obtained. Then, the multiple channels of medical data collected by different device sources at the same time node can be spatially aligned and unified into the same coordinate system.
[0041] Among them, considering that there are usually overlapping areas in the medical data collected by different device sources for the same target object, these overlapping areas can be used to achieve matching and alignment of the two data. For example, taking the oral diagnosis and treatment scenario as an example, an oral scanner is usually used to scan the user's mouth to obtain 3D scanning data of the user's mouth. At the same time, CBCT data of the user's mouth is also collected using CBCT. Since both sets of 3D data include the user's teeth, the two sets of 3D data can be matched based on the teeth and unified into the same coordinate system.
[0042] In step S203, considering that users usually need to view the changes of data of the same dimension at different time nodes, after acquiring multiple channels of medical data, the multiple channels of medical data can be divided according to the device dimension, and the medical data collected by the same device source can be divided into a group. Then, the multiple channels of medical data collected by the same device source at different time nodes can be spatially aligned.
[0043] After the alignment of multiple medical data, if a data display instruction triggered by the user through the interactive interface is detected, the aligned medical data can be displayed based on the data display instruction. For example, the user can select which device source data to display on the interactive interface, or which time node data to display, etc. In step S204, if a user-triggered instruction to display medical data at the target time node is obtained, the multiple medical data collected by different device sources at the aligned target time node are displayed on the interactive interface.
[0044] In step S206, if a user-triggered instruction to display the medical data collected by the target device source is obtained, the changes in the medical data collected by the target device source at different time nodes are displayed on the interactive interface.
[0045] For example, assume that multiple medical data include data X1, X2, and X3 collected by device source A at time nodes 1, 2, and 3, respectively, and data Y1, Y2, and Y3 collected by device source B at time nodes 1, 2, and 3, respectively, and data Z1, Z2, and Z3 collected by device source C at time nodes 1, 2, and 3, respectively. The data X1, Y1, and Z1 collected by different device sources at time node 1 can be aligned, and the user can choose to display the aligned X1, Y1, and Z1 at time node 1, or can choose to display only any two of the aligned X1, Y1, and Z1. Of course, the X1, X2, and X3 collected by device source A at three time nodes, namely, time node 1, time node 2, and time node 3, can also be aligned to display the changes in the data collected by the device source at different time nodes.
[0046] In some embodiments, considering that medical data collected by different device sources at the same time node for the same target object usually have overlapping parts, therefore, when displaying the medical data collected by different device sources at the same time node after alignment, the overlapping parts of the aligned multiple medical data can be fused first, and then the fused multiple medical data can be displayed on the interactive interface. By fusing the overlapping areas before displaying, the overlapping parts of the data collected by different device sources can be avoided to be cluttered.
[0047] For example, taking oral disease diagnosis and treatment as an example, assuming that the data collected at the same time node includes oral 3D scanning data, facial 3D scanning data, and oral CBCT data, there are overlapping parts in the three sets of data. For example, there are several overlapping teeth in the oral 3D scanning data and the facial 3D scanning data, and there are also overlapping teeth in the oral 3D scanning data and the oral CBCT data. Therefore, these data can be aligned and unified into a coordinate system, and then the overlapping areas can be merged into one data and then displayed. In this way, it can be avoided that the final presented data is relatively messy due to certain errors in the three-way data, which affects the display effect.
[0048] In some embodiments, in order to better display the changes of medical data collected by the same device source at different time nodes, the changes of medical data collected by the same device source at different time nodes can be displayed in the form of animation. Since the collection process is discrete, the animation frames corresponding to the time points between two adjacent time nodes can be generated by interpolation animation. For example, continuous animation frames can be generated by Morphing technology to continuously present the data changes between adjacent time nodes.
[0049] In some embodiments, considering that users need to view the changes of medical data over time at a finer granularity, semantic segmentation processing can also be performed on the medical data collected by the same device source at various time nodes to obtain multiple semantic entities at various time nodes. Then, the semantic entities are aligned in space, that is, a corresponding relationship between the semantic entities is established. When a user-triggered instruction to display the target semantic entity is obtained, the changes of the target semantic entity at different time nodes can be displayed on the interactive interface.
[0050] For example, in the oral disease diagnosis and treatment scenario, the three-dimensional data of the user's mouth is usually collected. Since doctors usually need to view the changes of a single tooth over time, the three-dimensional data of the user's mouth can be semantically segmented first to obtain the three-dimensional data of a single tooth. Then align the teeth at different time nodes, that is, establish a corresponding relationship between the teeth with the same number. When the user wants to view the changes of a certain tooth over time, the corresponding display command can be triggered. After the command is detected, the changes of the tooth over time can be displayed on the interactive interface. For example, the changes of the tooth can be displayed in the form of an animation.
[0051] Of course, in some scenarios, before semantic segmentation of 3D data, the 3D model represented by the 3D data can be texture mapped using a 2D image to obtain a 3D model with texture information, and then semantic segmentation can be performed to obtain a single semantic entity including texture information. When showing the change of a single semantic entity over time, the change of the semantic entity with texture information can also be shown.
[0052] In some embodiments, considering scenarios similar to oral disease diagnosis and treatment, there are many types of data to be analyzed, such as multiple channels of three-dimensional data, multiple channels of two-dimensional image data, pathological data, etc. Therefore, when spatially aligning multiple channels of medical data collected by different device sources at the same time node, there may be multiple situations, such as spatially aligning two channels of three-dimensional data, spatially aligning two channels of two-dimensional images, aligning two-dimensional images with three-dimensional data, aligning one channel of pathological data with two-dimensional images or three-dimensional data, etc. Among them, aligning a two-dimensional image with three-dimensional data refers to using a two-dimensional image to perform texture mapping on a three-dimensional model represented by three-dimensional data, that is, mapping the texture of the two-dimensional image to the three-dimensional model represented by three-dimensional data. Aligning pathological data with a two-dimensional image refers to determining which image area in the two-dimensional image represents the case data of the part represented by the pathological data, and aligning pathological data with three-dimensional data refers to determining which area of the three-dimensional model represented by the three-dimensional data represents the pathological data.
[0053] In some embodiments, the multi-channel medical data is multi-channel medical data in an oral diagnosis and treatment scenario, such as Figure 3 As shown, aligning 3D data with 3D data includes one or more of the following situations: aligning the 3D scan data of the user's mouth with the 3D scan data of the user's face in space, aligning the 3D scan data of the user's mouth with the CBCT data of the user's mouth in space, and aligning the 3D scan data of the user's face with the CBCT data of the user's skull in space. In the oral diagnosis and treatment scenario, multiple 3D data such as the user's oral 3D scan data, the user's facial 3D scan data, the user's oral CBCT data, and the user's skull CBCT data are usually collected. By integrating these multiple data, the internal structure of the user's mouth can be understood, which is convenient for formulating treatment plans for patients. Therefore, these multiple data need to be aligned and displayed.
[0054] In some embodiments, the multi-channel medical data is multi-channel medical data in an oral diagnosis and treatment scenario, and aligning the two-dimensional image with the three-dimensional data (i.e., mapping the texture of the two-dimensional image data to the three-dimensional model represented by the three-dimensional data) includes one or more of the following situations: mapping the texture of the two-dimensional image of the user's face to the three-dimensional model represented by the three-dimensional scan data of the user's face, and mapping the texture of the two-dimensional image of the user's mouth to the three-dimensional model represented by the three-dimensional scan data of the user's mouth. In an oral diagnosis and treatment scenario, the user's mouth is usually scanned in three dimensions to obtain a three-dimensional model of the mouth. At the same time, the user's face is also scanned in three dimensions to obtain a three-dimensional model of the user's face. Since the three-dimensional model only includes spatial information and does not contain texture information, it is also necessary to collect two-dimensional images of the mouth and face, and map the texture of the two-dimensional image to the corresponding three-dimensional model to obtain a three-dimensional model of the mouth or face with texture information.
[0055] In some embodiments, considering that the two-dimensional images in the multi-channel medical data may include at least two channels of two-dimensional images collected under different lighting environments, the at least two channels of two-dimensional images are collected in inconsistent environments, resulting in inconsistent overall brightness of the at least two channels of two-dimensional images. If the textures of the two channels of two-dimensional images are directly mapped to the corresponding three-dimensional model, it may cause uneven brightness of different parts of the final overall three-dimensional model, affecting the final presentation effect. Therefore, in view of this situation, the at least two channels of two-dimensional images can be preprocessed first to unify the brightness of the at least two channels of two-dimensional images, and then the preprocessed at least two channels of two-dimensional images are texture mapped to the corresponding three-dimensional model.
[0056] In some embodiments, the at least two two-dimensional images include a two-dimensional image of the user's mouth and a two-dimensional image of the user's face. For example, in an oral medical diagnosis scenario, three-dimensional data is usually collected for the user's mouth and the user's face to obtain respective three-dimensional models. At the same time, two-dimensional images can also be collected for the user's mouth and the user's face respectively, and texture mapped to the corresponding three-dimensional model to obtain a three-dimensional model with texture information. Since the light inside the mouth is usually darker than the light on the face, the overall brightness of the two-dimensional image of the user's mouth is usually darker than that of the two-dimensional image of the user's face. If the textures of the two-dimensional images are directly mapped to the corresponding three-dimensional model, and then the three-dimensional model of the mouth and the three-dimensional model of the face are aligned and fused for display, the brightness of different parts of the final fused three-dimensional model will be uneven, affecting the overall effect. Therefore, the two-dimensional images can be preprocessed before texture mapping to unify the brightness of the two-dimensional images. For example, for an image with a darker brightness, its brightness can be improved as a whole.
[0057] In some embodiments, considering that the multi-channel medical data may include at least two two-dimensional images captured by camera devices with different resolutions, due to the different resolutions of the at least two two-dimensional images, after the textures of the at least two two-dimensional images are mapped to the corresponding three-dimensional model, the texture information resolutions of different parts of the final generated overall model are inconsistent, for example, some places are clear and some places are blurred, which will affect the overall visual effect. In order to avoid the above problem, before the textures of the at least two two-dimensional images are mapped to the corresponding three-dimensional model, the at least two two-dimensional images can be preprocessed to unify the resolutions of the at least two two-dimensional images.
[0058] In some embodiments, the at least two two-dimensional images include a two-dimensional image of a designated part captured by a SLR camera and a two-dimensional image of a designated part captured by a three-dimensional scanning device, wherein the designated part includes a user's face and / or a user's mouth.
[0059] For example, in the process of oral diagnosis and treatment, a three-dimensional scanning device is usually used to collect three-dimensional data of the user's oral cavity and / or the user's face, wherein the three-dimensional scanning device can use two cameras to collect two-dimensional images of the user's oral cavity and / or the user's face from different perspectives, and then use these two-dimensional images to perform three-dimensional reconstruction to obtain a three-dimensional model. At the same time, these two-dimensional images can also be used to perform texture mapping on the three-dimensional model to obtain a three-dimensional model with texture information. In addition, a single-lens reflex camera can also be used to take high-definition images of the user's oral cavity and / or the user's face. For example, a high-definition image is usually taken for the user's face. Since the resolution of the high-definition facial image taken by the single-lens reflex camera is often higher than the resolution of the oral cavity image collected by the three-dimensional scanning device, if the two-dimensional images are texture mapped to the corresponding three-dimensional model respectively, the texture information resolution of different parts of the final generated model may be inconsistent, affecting the overall effect. Therefore, the two-way three-dimensional images can be pre-processed first to unify the resolution of the two, for example, the low-resolution image is up-sampled to improve its resolution, and then the pre-processed two-dimensional images are used to perform texture mapping on the three-dimensional model.
[0060] In some embodiments, it is considered that users often have different needs when displaying multiple medical data. For example, sometimes it is necessary to display a collection of multiple medical data collected by multiple device sources at a certain time node, sometimes it is only necessary to display a single data collected by a certain device source at a certain time node, and sometimes it is necessary to display the changes of multiple data collected by a certain device source at multiple time nodes. In order to meet the different needs of users and to respond to the user's display instructions conveniently and quickly. After aligning the multiple medical data collected by different device sources at the same time node in space, and aligning the multiple medical data collected by the same device source at different time nodes in space, an expression model can be generated based on the aligned data. Among them, the expression model can include multiple data levels, different data levels include medical data of different dimensions, and data of different data levels can be associated. After the expression model is generated, the aligned medical data can be displayed based on the expression model.
[0061] In some scenarios, after the alignment of multiple medical data, other processing can be performed on the medical data based on actual needs. For example, in some embodiments, the multiple medical data include three-dimensional data, and the three-dimensional data can be firstly processed by semantic segmentation to obtain semantic segmentation results, and then an expression model is generated based on the semantic segmentation results and the data alignment results.
[0062] Taking the oral disease diagnosis and treatment scenario as an example, after obtaining the 3D model of the oral cavity, the 3D model of the oral cavity can be further processed by semantic segmentation to obtain individual teeth. Or the CBCT data of the oral cavity can be further processed by semantic segmentation to obtain the upper and lower ossicles. Then, an expression model can be generated based on the processing results of various processes. In other words, the expression model is a way to characterize the processing results of alignment, semantic segmentation, texture mapping and other processing of these multi-channel medical data. After using the expression model to characterize these processing results, data of different dimensions can be displayed based on the user's display needs and the expression model.
[0063] In some embodiments, the multi-channel medical data is medical data in an oral disease diagnosis and treatment scenario. When dividing the expression model into data levels, the data levels can be divided into the following data levels:
[0064] (1) A spatial 3D grid layer, wherein the spatial 3D grid layer includes grid data obtained by rasterizing the three-dimensional data of the user's head, that is, the layer includes multiple 3D grids and the spatial coordinates corresponding to each 3D grid. When the three-dimensional data of the head is rasterized, it can be divided evenly (the sizes of the smallest 3D grids obtained by division are consistent) or unevenly (the sizes of the smallest 3D grids obtained by division are inconsistent). For example, in some scenarios, the size of the smallest 3D grid can be adaptively adjusted. For example, for areas with dense data, the grid can be divided at a higher resolution, that is, the size of the smallest 3D grid obtained by division can be smaller. For areas with sparse data, the grid can be divided at a higher resolution, that is, the size of the smallest 3D grid obtained by division can be larger. In this way, it can be ensured that the position of each medical data in the data-dense area can be accurately represented, and the amount of calculation can be reduced.
[0065] (2) Geometry layer, where the geometry layer includes the geometric shape data of the user's oral cavity, such as the shape of teeth, skull, jaw, face, etc. The geometry may include information such as the shape and surface texture of each part of the oral cavity, and the spatial position of the geometry data is represented by the grid coordinates in the spatial 3D grid layer. For example, some 3D grids in the spatial 3D grid layer may be associated with the three-dimensional model of teeth, and the position of the associated 3D grid is the position of the teeth in the entire skull area.
[0066] (3) Attribute layer, the attribute layer includes attribute data of the user's oral cavity, wherein the spatial position of the attribute data is represented by grid coordinates in the spatial 3D grid layer; for example, the attribute data (e.g., pathological data) can be associated with the grid in the 3D grid layer, and the position of the associated 3D grid is the position of the part to which the attribute data belongs in the head area.
[0067] (4) Semantic element layer, the semantic element layer includes geometric shape data and attribute data corresponding to each semantic entity in the user's mouth, wherein the spatial position of each semantic entity is represented by the grid coordinates in the spatial 3D grid layer. Each semantic entity can be obtained by semantically segmenting the collected three-dimensional data or two-dimensional data. For example, the three-dimensional scanning data of the oral cavity can be semantically segmented into each tooth entity. Each tooth entity can be represented based on the geometric shape data and attribute data of the oral cavity. Each semantic entity can be associated with a grid in the 3D grid layer, and the position of the associated 3D grid is the position of the semantic entity in the skull area.
[0068] Of course, the embodiment of the present disclosure divides multiple medical data into the above four data levels based on the general needs of users when viewing data. In actual applications, the division can also be performed in other ways and is not limited to the above four data levels.
[0069] In some embodiments, the geometric morphology data may include one or more of the following: three-dimensional scanning data of the user's mouth, three-dimensional scanning data of the user's face, CBCT data of the user's mouth, CBCT data of the user's skull, texture information of the user's mouth, and texture information of the user's face. The texture information of the user's mouth is aligned with the three-dimensional model corresponding to the three-dimensional scanning data of the user's mouth, and the texture information of the user's face is aligned with the three-dimensional model corresponding to the three-dimensional scanning data of the user's face. The three-dimensional scanning data and texture information can be displayed through different layers. When displaying the geometric morphology data, the three-dimensional model without texture information can be displayed based on the user's choice, or the three-dimensional model with texture information can be displayed.
[0070] In some embodiments, the attribute data may include one or more of the following: two-dimensional image data, such as an image of the user's mouth, an image of the user's face, a cephalogram, etc., pathological data, physiological slice data.
[0071] In some embodiments, the semantic entity may include one or more of the following: a single tooth, an upper jaw, a lower jaw, etc.
[0072] In some embodiments, after the above expression model is generated based on the processing results of various processing (alignment, semantic segmentation, etc.) of multiple medical data, the data of some or all data levels in the expression model can be displayed based on the display instructions triggered by the user. For example, in some scenarios, the user only wants to view the geometric shape data, that is, only the data of the geometric layer can be displayed, and the user only wants to view the attribute data, then only the data of the attribute layer can be displayed. In some embodiments, in order to display multiple medical data in a more detailed manner, the medical data of each data layer can be divided into multiple layers, and the user can choose to display the medical data of one or more layers. For example, for the semantic element layer, since the layer includes the geometric shape data and attribute data of multiple semantic entities, the geometric shape data of each semantic entity can be divided into one layer, and the attribute data of each semantic entity can also be divided into one layer. When the user only selects the layer corresponding to the geometric shape data of a certain semantic entity, only the geometric shape of the semantic entity is displayed. When the user selects the layer corresponding to the geometric shape data of a certain semantic entity and the layer corresponding to the attribute data, the data of the two layers of the semantic entity can be displayed at the same time. For example, you can display only the 3D model of a certain tooth, or display the 3D model of a certain tooth with texture information.
[0073] Of course, the layer settings can be specifically set based on actual display needs. By showing or hiding the layers, it is convenient for users to view medical data of different dimensions.
[0074] In some embodiments, Figure 4 As shown, in order to facilitate user operation, multiple data levels of the expression model and layer identifiers of multiple layers under each data level can be displayed in the interactive interface. The user can choose to display or hide these data levels or the layers under each data level, and then display the medical data corresponding to the layer selected by the user in the interactive interface based on the user's layer selection operation. In some embodiments, for the layer not selected by the user, the medical data corresponding to the layer can be made transparent or semi-transparent before being displayed. For example, if the user wants to view the information of a certain tooth, the corresponding layer of the tooth can be selected. At this time, if the layers corresponding to other teeth are set to a transparent state, the interactive interface will only display this tooth. If the layers corresponding to other teeth are set to a semi-transparent state, the interactive interface will also display other teeth, but they will be displayed in a darker form. This display method makes it easy for users to understand the position of the current tooth in the entire mouth.
[0075] It is not difficult to understand that the solutions described in the above embodiments can be combined when there is no conflict, and they are not listed one by one in the embodiments of the present disclosure.
[0076] Accordingly, the present disclosure also provides a medical data display device, such as Figure 5 As shown, the device 50 comprises:
[0077] An acquisition module 51 is used to acquire multiple medical data collected by multiple device sources at different time points for the same target object;
[0078] An alignment module 52, used to spatially align multiple channels of medical data collected by different device sources at the same time node, and spatially align multiple channels of medical data collected by the same device source at different time nodes;
[0079] The display module 53 is used to display the aligned multiple medical data collected by different device sources at the target time node on the interactive interface when a user-triggered instruction to display the medical data at the target time node is obtained; and to display the changes of the medical data collected by the target device source at different time nodes on the interactive interface when a user-triggered instruction to display the medical data at the target time node is obtained.
[0080] Among them, the specific steps of the above-mentioned device executing the task processing method can refer to the description in the above-mentioned method embodiment, and will not be repeated here.
[0081] Furthermore, the present disclosure also provides a device, such as Figure 6 As shown, the device includes a processor 61, a memory 62, and computer instructions stored in the memory 62 and executable by the processor 61. When the processor 61 executes the computer instructions, the method described in any one of the above embodiments is implemented.
[0082] The present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in any of the above embodiments is implemented.
[0083] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0084] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the embodiments of the present disclosure can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the embodiments of the present disclosure is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present disclosure or some parts of the embodiments.
[0085] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.
[0086] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the functions of each module can be implemented in the same one or more software and / or hardware when implementing the embodiment of the present disclosure. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the embodiment. A person of ordinary skill in the art can understand and implement it without paying creative labor.
[0087] The above is only a specific implementation of the embodiment of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiment of the present disclosure. These improvements and modifications should also be regarded as the protection scope of the embodiment of the present disclosure.
Claims
1. A medical data display method, It is characterized in that The method comprises: Acquire multiple medical data collected by multiple equipment sources at different time points for the same target object; Spatially align multiple medical data collected by different equipment sources at the same time node, and spatially align multiple medical data collected by the same equipment source at different time nodes; When a user-triggered instruction to display medical data at a target time node is obtained, the aligned multiple medical data collected by different device sources at the target time node are displayed on the interactive interface; When a user-triggered instruction to display the medical data collected by the target device source is obtained, changes in the medical data collected by the target device source at different time nodes are displayed on the interactive interface.
2. The method according to claim 1, It is characterized in that The displaying of the aligned multi-channel medical data collected by different equipment sources at the target time node on the interactive interface includes: The overlapping parts of the aligned multiple medical data are fused, and the fused multiple medical data are displayed on the interactive interface.
3. The method according to claim 2, It is characterized in that The displaying of changes in the medical data collected by the target device source at different time points on the interactive interface includes: The changes in the medical data collected by the target device source at different time nodes are displayed in the form of animation, wherein the animation frames corresponding to the time points between two adjacent time nodes are generated by interpolation animation.
4. The method according to claim 1, It is characterized in that The multi-channel medical data includes three-dimensional data of the target object collected by a designated device source at different time points, and the method further includes: For the three-dimensional data at each time node, semantic segmentation is performed on the three-dimensional data to obtain multiple semantic entities; Align multiple semantic entities at different time nodes in space; When a display instruction of a target semantic entity triggered by a user is obtained, changes of the target semantic entity at different time nodes are displayed on the interactive interface.
5. The method according to any one of claims 1 to 4, It is characterized in that The spatial alignment of multiple medical data collected by different equipment sources at the same time node includes one or more of the following situations: Spatially align at least two channels of three-dimensional data collected by different device sources at the same time node; Spatially align at least two two-dimensional images collected by different device sources at the same time node; Mapping at least one two-dimensional image texture collected by different device sources at the same time node to a three-dimensional model represented by at least one three-dimensional data; At least one pathological data collected by different equipment sources at the same time node is aligned with at least one two-dimensional image or three-dimensional data.
6. The method according to claim 5, It is characterized in that The multiple medical data are medical data in an oral diagnosis and treatment scenario, and the spatial alignment of at least two three-dimensional data includes one or more of the following situations: spatially aligning three-dimensional scanning data of the user's oral cavity and three-dimensional scanning data of the user's face; spatially aligning the three-dimensional scan data of the user's oral cavity with the CBCT data of the user's oral cavity; Spatially aligning the 3D scan data of the user's face with the CBCT data of the user's skull; and / or The multiple channels of medical data are medical data in an oral diagnosis and treatment scenario, and the step of mapping at least one channel of two-dimensional image texture to a three-dimensional model represented by at least one channel of three-dimensional data includes one or more of the following situations: mapping the two-dimensional image texture of the user's face to a three-dimensional model represented by three-dimensional scan data of the user's face; mapping the two-dimensional image texture of the user's mouth to a three-dimensional model represented by three-dimensional scan data of the user's mouth; and / or The multi-channel medical data includes at least two two-dimensional images collected under different lighting environments. Before the at least two two-dimensional images are texture mapped to a three-dimensional model represented by at least one three-dimensional data, the method also includes: preprocessing the at least two two-dimensional images to unify the brightness of the at least two two-dimensional images.
7. The method according to claim 5, It is characterized in that The at least two two-dimensional images include a two-dimensional image of the user's mouth and a two-dimensional image of the user's face.
8. The method according to claim 5, It is characterized in that The multiple medical data include at least two two-dimensional images acquired by camera devices with different resolutions. Before the at least two two-dimensional images are texture mapped to a three-dimensional model represented by at least one three-dimensional data, the method further includes: The at least two two-dimensional images are preprocessed to unify the resolutions of the at least two two-dimensional images.
9. The method according to claim 8, It is characterized in that The at least two two-dimensional images include a two-dimensional image of a designated part taken by a single-lens reflex camera and a two-dimensional image of the designated part collected by a three-dimensional scanning device, and the designated part includes a user's face and / or a user's mouth.
10. The method according to claim 1, It is characterized in that After spatially aligning multiple channels of medical data acquired by different device sources at the same time node, and / or spatially aligning multiple channels of medical data acquired by the same device source at different time nodes, the method further includes: generating an expression model based on the aligned data, wherein the expression model includes multiple data levels, and different data levels include medical data of different dimensions; The aligned medical data is displayed based on the expression model.
11. The method according to claim 10, It is characterized in that The multi-channel medical data also includes three-dimensional data of the target object, and the method further includes: Performing semantic segmentation processing on the three-dimensional data to obtain a semantic segmentation result; The step of generating an expression model based on the aligned data comprises: An expression model is generated based on the aligned data and the semantic segmentation result.
12. The method according to claim 10, It is characterized in that The multi-channel medical data is medical data in the oral diagnosis and treatment scenario, and the expression model includes the following data levels: A spatial 3D grid layer, wherein the spatial 3D grid layer includes grid data obtained by rasterizing three-dimensional data of the user's head; A geometry layer, the geometry layer comprising geometry data of the user's oral cavity, wherein the spatial position of the geometry data is represented by grid coordinates in the spatial 3D grid layer; an attribute layer, the attribute layer including attribute data of the user's oral cavity, wherein the spatial position of the attribute data is represented by grid coordinates in the spatial 3D grid layer; A semantic element layer, wherein the semantic element layer includes geometric shape data and attribute data corresponding to each semantic entity in the user's mouth, wherein the spatial position of each semantic entity is represented by a grid coordinate in the spatial 3D grid layer.
13. The method according to claim 12, It is characterized in that The geometric shape data includes one or more of the following: three-dimensional scanning data of the user's oral cavity, three-dimensional scanning data of the user's face, CBCT data of the user's oral cavity, CBCT data of the user's skull, CBCT data of the user's jaw, texture information of the user's oral cavity, texture information of the user's face; and / or The attribute data includes one or more of the following: two-dimensional image data, pathological data, physiological slice data; and / or The semantic entity includes one or more of the following: a single tooth, an upper jaw, and a lower jaw.
14. The method according to claim 10, It is characterized in that Displaying the aligned data based on the expression model includes: Medical data of some or all of the multiple data levels of the expression model are displayed based on a display instruction triggered by the user.
15. The method according to claim 10, It is characterized in that The medical data of each data level can be displayed through multiple layers, and the user can choose to display the medical data of one or more layers.
16. The method according to claim 15, It is characterized in that Displaying the aligned data based on the expression model includes: In response to the user's layer selection operation on the interactive interface, medical data corresponding to the layer selected by the user is displayed on the interactive interface, wherein the interactive interface displays multiple data levels of the expression model and layer identifiers of multiple layers under each data level.
17. The method according to claim 16, It is characterized in that The method further includes: for a layer not selected by the user, the medical data corresponding to the layer is made transparent or semi-transparent before being displayed.
18. A medical data display device, It is characterized in that The device comprises: An acquisition module, used to acquire multiple medical data collected by multiple device sources at different time points for the same target object; An alignment module, used to spatially align multiple medical data collected by different device sources at the same time node, and to spatially align multiple medical data collected by the same device source at different time nodes; A display module is used to display multiple medical data collected by different device sources at the target time node after alignment on an interactive interface when a user-triggered instruction to display medical data at the target time node is obtained; and to display changes in medical data collected by the target device source at different time nodes on an interactive interface when a user-triggered instruction to display medical data at the target time node is obtained.
19. An electronic device, It is characterized in that The electronic device includes a processor, a memory, and computer instructions stored in the memory and executable by the processor. When the processor executes the computer instructions, the method according to any one of claims 1 to 17 is implemented.
20. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 17 is implemented.
Citation Information
Cited By
Medical data display method, apparatus, device, and storage medium
EP4712091A1
