Medical image display method, medical image processing method, and image display system

Abstract

The application relates to a medical image display method, a medical image processing method, an image display system, a computer device, a storage medium and a computer program product. The method comprises the following steps: acquiring an initial medical image collected by an augmented reality device; performing image enhancement processing on the initial medical image to obtain a to-be-processed medical image; identifying the to-be-processed medical image by using at least one identification model obtained by pre-training, to obtain a target and a target state in the to-be-processed medical image; fusing the target and the target state obtained by each identification model; and mapping the fused target and the target state to a virtual space corresponding to the augmented reality device for display. The image enhancement can avoid image blurring, the identification accuracy can be improved for the states of un-identified or un-labeled patient wounds and medical instruments, and the display is more accurate.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, and in particular to a medical image display method, a medical image processing method, an image display system, a computer device, a storage medium, and a computer program product. Background Technology

[0002] Augmented Reality (AR) is a technology that calculates the position and angle of camera images in real time and adds corresponding images. It was first proposed in 1990. It uses various techniques to overlay computer-generated virtual objects or non-geometric information about real objects onto real-world scenes, thereby enhancing the real world.

[0003] Currently, AR technology has been applied to medical surgery, and image processing mainly uses deep learning methods and traditional image processing to achieve human perception.

[0004] However, when wearing AR devices, the AR glasses need to merge reality and virtuality, resulting in a blurry display of real objects and reduced accuracy. Summary of the Invention

[0005] Therefore, it is necessary to provide a medical image display method, medical image processing method, image display system, computer device, computer-readable storage medium, and computer program product that can improve display accuracy in response to the above-mentioned technical problems.

[0006] In a first aspect, this application provides a method for displaying medical images, the method comprising:

[0007] Acquire initial medical images captured by augmented reality devices;

[0008] The initial medical image is subjected to image enhancement processing to obtain the medical image to be processed;

[0009] The medical image to be processed is identified by at least one recognition model obtained through pre-training, so as to obtain the target and target state in the medical image to be processed;

[0010] The target and target state obtained from each of the recognition models are fused together;

[0011] The merged target and its state are mapped and displayed in the virtual space corresponding to the augmented reality device.

[0012] In one embodiment, the image enhancement processing of the initial medical image to obtain the medical image to be processed includes:

[0013] Convert the initial medical image into a grayscale image;

[0014] Calculate the gradient histogram and target grayscale value of the grayscale image;

[0015] Based on the gradient histogram and the grayscale target value, the brightness coefficient to be processed corresponding to the initial medical image is obtained; the corresponding color integral image to be processed is calculated based on the initial medical image.

[0016] The initial medical image is enhanced based on the color integral image to be processed and the brightness coefficient to be processed to obtain the medical image to be processed.

[0017] In one embodiment, the step of performing image enhancement processing on the initial medical image based on the color integral image to be processed and the brightness coefficient to be processed to obtain the medical image to be processed includes:

[0018] The color integral image to be processed is matched with the standard color integral image to determine the standard luminance coefficient corresponding to the luminance coefficient to be processed.

[0019] The pixel values ​​in the initial medical image are updated using the standard brightness coefficient to obtain the medical image to be processed.

[0020] In one embodiment, fusing the target and target state obtained from each of the recognition models includes:

[0021] Obtain the weights corresponding to each of the aforementioned recognition models;

[0022] The targets and target states obtained by each of the recognition models are fused based on the weights corresponding to the recognition models.

[0023] In one embodiment, before fusing the targets and target states obtained by each recognition model based on the weights corresponding to the recognition models, the method further includes:

[0024] Obtain the processing time and target recognition accuracy corresponding to the first recognition model and the second recognition model;

[0025] When the difference between the processing time of the first recognition model and the processing time of the second recognition model is greater than a threshold, the target and target state corresponding to the second recognition model are used as the fused target and target state.

[0026] When the difference between the processing time of the first recognition model and the processing time of the second recognition model is less than or equal to the threshold, and the recognition accuracy of the first recognition model is greater than the recognition accuracy of the second recognition model, the target and target state obtained by each recognition model are further fused based on the weights corresponding to the recognition models.

[0027] When the difference between the processing time of the first recognition model and the processing time of the second recognition model is less than or equal to the threshold, and the recognition accuracy of the first recognition model is less than or equal to the recognition accuracy of the second recognition model, the target and target state corresponding to the second recognition model are used as the fused target and target state.

[0028] In one embodiment, mapping the fused target and target state to the virtual space corresponding to the augmented reality device for display includes:

[0029] Based on the conversion relationship of multiple acquisition devices of the augmented reality device, the fused target and target state are mapped onto the three-dimensional space of the augmented reality device;

[0030] The target and its state in the three-dimensional space are mapped to the virtual space corresponding to the augmented reality device for display.

[0031] Secondly, this application also provides a medical image processing method, the medical image processing method comprising:

[0032] The medical image display method in any of the above embodiments is used to obtain the target and the target state.

[0033] Match the target and its state with a preset scenario;

[0034] An alarm message is generated based on the matching results.

[0035] In one embodiment, matching the target and the target state with a preset scene includes at least one of the following:

[0036] The acquired operating devices and their states are matched with the standard operating devices and their states in the preset scenario.

[0037] The acquired tissue parts and their states are matched with standard tissue parts and their states in a preset scenario; and

[0038] The operation process is determined based on the obtained target and the target state, and the operation process is matched with the standard process in the preset scenario.

[0039] Thirdly, this application also provides an image display system, the system comprising:

[0040] Augmented reality devices are used to acquire initial medical images and display information processed by a display processor.

[0041] A processor is configured to execute the steps of the method described in any of the above embodiments to obtain display information of the virtual space corresponding to the enhanced display device.

[0042] Fourthly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any one of the embodiments of the claims.

[0043] Fifthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the embodiments.

[0044] The aforementioned medical image display method, medical image processing method, image display system, computer equipment, storage medium, and computer program product, after acquiring the initial medical image collected by the augmented reality device, first perform image enhancement processing on the initial medical image to obtain the medical image to be processed. This image enhancement can avoid image blurring. Then, at least one recognition model is obtained through pre-training to recognize the medical image to be processed, thereby obtaining the target and target state in the medical image to be processed. This can improve the recognition accuracy for unrecognized or unlabeled patient wounds and medical device states, thus making the display more accurate. Attached Figure Description

[0045] Figure 1 This is a system diagram of an image display system in one embodiment;

[0046] Figure 2 This is a hardware schematic diagram of an image display system in one embodiment;

[0047] Figure 3 This is a functional schematic diagram of an augmented reality device in one embodiment;

[0048] Figure 4 This is a functional diagram of a terminal or server in one embodiment;

[0049] Figure 5 This is a flowchart illustrating a medical image display method in one embodiment;

[0050] Figure 6 This is a flowchart illustrating the image enhancement steps in one embodiment;

[0051] Figure 7 This is a schematic diagram illustrating the processing steps of two recognition models in one embodiment;

[0052] Figure 8 This is a flowchart illustrating the training steps of the first recognition model in one embodiment.

[0053] Figure 9 This is a flowchart illustrating the training steps of the second recognition model in one embodiment;

[0054] Figure 10 This is a flowchart illustrating the fusion steps in one embodiment;

[0055] Figure 11 A flowchart of the display steps of an enhanced display device in one embodiment;

[0056] Figure 12 This is a flowchart illustrating a medical image processing method in one embodiment;

[0057] Figure 13 This is a flowchart of the alarm information generation steps in one embodiment;

[0058] Figure 14 This is a startup prompt image from one embodiment;

[0059] Figure 15 A procedure selection prompt diagram in one embodiment;

[0060] Figure 16 This is a schematic diagram of target recognition in one embodiment;

[0061] Figure 17 This is a schematic diagram of a reminder in one embodiment;

[0062] Figure 18 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] Combination Figure 1 As shown, Figure 1 This is a system diagram of an image display system in one embodiment. In this embodiment, the image display system includes an augmented reality device 100 and a processor 200. The processor 200 can be integrated into the augmented reality device 100, or it can be integrated into a separate terminal or server and communicate with the augmented reality device. This application does not impose specific limitations on the processor, as long as it can perform the corresponding functions.

[0065] Specifically, in combination Figure 2 As shown, Figure 2This is a hardware schematic diagram of an image display system in one embodiment. In this embodiment, the processor is installed in a terminal or server, and the augmented reality device communicates with the terminal or server through a data interface. The augmented reality device acquires initial medical images during the operation process and sends them to the terminal or server through the data interface, so that the processor of the terminal or server processes the initial medical images.

[0066] Specifically, the augmented reality device can be AR glasses, and in other embodiments, it can be other augmented reality devices. (Combined with...) Figure 3 As shown, Figure 3 This is a functional schematic diagram of an augmented reality (AR) device in one embodiment. In this embodiment, the AR device's processor includes an image acquisition module for acquiring initial medical images during the operation process. The initial medical images are acquired by the AR device through the image acquisition module. The initial medical images are obtained during the operation, such as a scan during surgery, and may include images of medical equipment and the patient's wound. The AR device sends the images to a terminal or server through a data interface module, so that the processor of the terminal or server can process the initial medical images and receive the processing results of the initial medical images from the terminal or server through the data interface. The AR device's model generation module obtains the corresponding model from the local model library module based on the feedback processing results, and generates a target medical image based on the processing results and the model through a model registration module. The target medical image is then holographically projected onto the medical equipment and the patient's wound through a holographic projection module.

[0067] Specifically, in combination Figure 4 As shown, Figure 4 This is a functional diagram of a terminal or server in one embodiment, wherein the terminal or server receives an initial medical image sent by an augmented reality device through a data interface. A preprocessing module in the processor of the terminal or server performs image augmentation processing on the initial medical image to obtain a medical image to be processed; a model processing module in the processor of the terminal or server identifies the medical image to be processed using at least one pre-trained recognition model to obtain the target and target state in the medical image to be processed, such as medical equipment, patient's mouth, or objects in the environment and their corresponding target states; an adaptive fusion module in the processor of the terminal or server fuses the targets and target states obtained from each recognition model; the data interface in the processor of the terminal or server sends the fused targets and target states to the augmented reality device, and the processor of the augmented reality device maps them to the corresponding virtual space of the augmented reality device for display, where the real content may include surgical reminders, alarms, and navigation, so that doctors can perform surgery more intelligently and safely.

[0068] At the start of the operation, the terminal or server communicates with the augmented reality device and determines whether the connection between the terminal or server and the augmented reality device is successful. That is, the terminal or server and the augmented reality device are turned on, the corresponding IP addresses are set, and communication is carried out through network protocols. If a connection failure occurs, i.e. a blocked state, different IP addresses are adjusted to continue communication.

[0069] If the terminal or server and the augmented reality device successfully establish a communication connection, the terminal or server issues instructions according to the specified data format. After receiving the acquisition command, the augmented reality device opens its camera and stores the data at the corresponding address. After the augmented reality device has acquired the image, it sends the data back to the terminal or server via the network protocol, thus completing the entire terminal or server image acquisition process.

[0070] After acquiring the initial medical image from the augmented reality device, the initial medical image is first enhanced to obtain the medical image to be processed. This image enhancement can avoid image blurring. Then, at least one recognition model is obtained through pre-training to recognize the medical image to be processed, and to obtain the target and target state in the medical image to be processed. This can improve the recognition accuracy for unrecognized or unlabeled patient wounds and medical instruments, and thus make the display more accurate.

[0071] In one embodiment, such as Figure 5 As shown, a medical image display method is provided, which is applied to... Figure 1 Taking the processor in the example, the explanation includes the following steps:

[0072] S502: Acquire initial medical images captured by the augmented reality device.

[0073] Specifically, the initial medical image is obtained during an operation, such as a scan during surgery; the initial medical image can include images of medical equipment and the patient's wound. During the operation, the initial medical image is acquired using the image acquisition device of the augmented reality device, for example, through the camera of the augmented reality device. Optionally, the augmented reality device may include multiple cameras, thus acquiring initial medical images from multiple cameras. This allows for parallel processing of multiple initial medical images in subsequent processing, or the multiple initial medical images can be pre-fused and then processed only on the fused initial medical image.

[0074] S504: Perform image enhancement processing on the initial medical image to obtain the medical image to be processed.

[0075] Specifically, image enhancement processing can refer to enhancing the brightness of an image, that is, by identifying weak light and backlight in the initial medical image and intelligently supplementing the lighting, so as to obtain a medical image to be processed with a brightness that meets the requirements.

[0076] S506: Recognize the medical image to be processed using at least one recognition model obtained through pre-training, and obtain the target and target state in the medical image to be processed.

[0077] Specifically, the recognition model can be pre-trained and may include deep learning models as well as traditional recognition models. In other embodiments, the recognition model may also include models of other algorithms, which are not specifically limited here.

[0078] Deep learning models can infer the state and location of wounds during surgery, as well as the identification and use of instruments, by modifying deep learning network branches and adding attention mechanisms at different scales, making the surgical process safer and more intelligent. Traditional recognition models can improve processing efficiency by applying prior knowledge, for example, by using feature vectors and classifiers to process medical images to obtain the target and its state.

[0079] The target can refer to a target in the medical image to be processed, including but not limited to various devices, medical instruments, tissue sites, or wounds in the operational scene. The various devices can include surgical robot systems, the number of trolleys, and operating tables. The target state can include corresponding preset states, which can be pre-defined. For example, the state of a medical instrument includes unused, in use, and used. The wound state includes unsutured and sutured, etc. Those skilled in the art can set the target state of each object as needed.

[0080] In the above embodiments, the distance and angle between the surgical instruments and the patient and the AR image device, as well as the brightness, can lead to unstable recognition. Therefore, the initial medical image and network structure are reasonably optimized according to the actual situation to improve recognition accuracy and stability.

[0081] S508: Fuse the target and target state obtained from each recognition model.

[0082] Specifically, to improve accuracy, this embodiment fuses the target and target state obtained from each recognition model. This combines the recognition results of multiple models, thus increasing accuracy. For example, threshold weights are assigned to the target and target state obtained from each recognition model to obtain the fused target and target state.

[0083] S510: Maps the merged target and its state to the virtual space corresponding to the augmented reality device for display.

[0084] Specifically, the processor maps the fused target and target state to the virtual space corresponding to the augmented reality device for display. For example, it first maps the target and target state to the three-dimensional space of the augmented reality device, and then maps the target and target state in the three-dimensional space to the virtual space corresponding to the augmented reality device for display.

[0085] The aforementioned medical image display method, after acquiring the initial medical image collected by the augmented reality device, first performs image enhancement processing on the initial medical image to obtain the medical image to be processed. This image enhancement can avoid image blurring. Then, at least one recognition model is obtained through pre-training to recognize the medical image to be processed, and obtain the target and target state in the medical image to be processed. This can improve the recognition accuracy for unrecognized or unlabeled patient wounds and medical devices, and thus make the display more accurate.

[0086] In one embodiment, see Figure 6 As shown, Figure 6 This is a flowchart illustrating an image enhancement step in one embodiment. The image enhancement step, which involves performing image enhancement processing on an initial medical image to obtain a medical image to be processed, includes: converting the initial medical image to a grayscale image; calculating the gradient histogram and grayscale target value of the grayscale image; obtaining the brightness coefficient to be processed corresponding to the initial medical image based on the gradient histogram and grayscale target value; calculating the corresponding color integral image to be processed based on the initial medical image; and performing image enhancement processing on the initial medical image based on the color integral image to be processed and the brightness coefficient to be processed to obtain the medical image to be processed.

[0087] Specifically, the initial medical image is a color image. Firstly, grayscale information is extracted from the color image to remove the influence of color on the overall brightness information, in order to calculate the brightness coefficient to be processed. Secondly, a color integral image is calculated using the initial color medical image. These two steps can be processed in parallel to improve efficiency. Finally, based on the brightness coefficient to be processed and the color integral image, the brightness of the initial medical image is adjusted to obtain the final medical image.

[0088] Optionally, the brightness coefficient to be processed corresponding to the initial medical image is calculated based on the grayscale image, including: calculating the gradient histogram of the grayscale image and the grayscale target value; and obtaining the brightness coefficient to be processed corresponding to the initial medical image based on the gradient histogram and the grayscale target value.

[0089] After obtaining the grayscale image, the gradient histogram of the grayscale image is calculated, which represents the brightness difference between adjacent pixels. To facilitate scanning each pixel in the grayscale image in a certain order, and to calculate the target grayscale value during the scanning process, this target grayscale value can be the grayscale median. For example, when traversing each pixel in the grayscale image, the grayscale values ​​of each pixel can be sorted to obtain the grayscale median. Finally, the brightness coefficient of the initial medical image is determined by combining the gradient histogram distribution with the grayscale median value, thereby identifying bright and weak light. Because dark and weak light are generally darker, the gradient texture is smaller, and the grayscale median is also smaller, thus identifying the overall brightness of the image. If the gradient is small and the grayscale median value is also small, the brightness coefficient is low, and vice versa. That is to say, if the gradient is less than a certain value, it means that the initial medical image is either generally bright or generally dark. Therefore, the grayscale median value is used to determine whether the initial medical image is bright or dark, thereby obtaining the brightness coefficient.

[0090] For convenience, a table relating brightness coefficient, gradient, and grayscale median can be pre-set. After calculating the gradient and grayscale median, the corresponding brightness coefficient can be obtained by looking up the table.

[0091] Optionally, the color integral image can be obtained by first scanning the entire initial medical image using the RGB three-channel pixel values ​​and counting the RGB three-channel pixel values ​​to obtain a color histogram. Then, through a normalization operation, each gray level of the color histogram is divided by the maximum gray level range of 255, and the values ​​are accumulated according to each gray level to obtain the color integral image.

[0092] Optionally, image enhancement processing is performed on the initial medical image based on the color integral image to be processed and the brightness coefficient to be processed to obtain the medical image to be processed, including: matching the color integral image to be processed with a standard color integral image to determine the standard brightness coefficient corresponding to the brightness coefficient to be processed; updating the pixel values ​​in the initial medical image through the standard brightness coefficient to obtain the medical image to be processed.

[0093] Specifically, when performing reverse reasoning on the three-channel pixel mapping table of the integral image, the pixel mapping table is updated in conjunction with the brightness coefficient, thereby correcting the grayscale value in the overall image and ultimately improving the brightness.

[0094] The standard color integral image is a pre-stored integral image with uniform brightness that meets the requirements. The color integral image to be processed is matched with the standard color integral image, and the standard brightness coefficient corresponding to the successfully matched standard color integral image is obtained. In this way, the pixel values ​​in the initial medical image are updated by the standard brightness coefficient to obtain the medical image to be processed, so as to improve the brightness.

[0095] In the above embodiments, the initial medical image is enhanced to obtain the medical image to be processed, which improves the brightness of the image and avoids image blurring, thereby laying the foundation for subsequent recognition accuracy.

[0096] In one embodiment, combined with Figure 7 As shown, the medical image to be processed is identified by at least one recognition model obtained through pre-training to obtain the target and target state in the medical image to be processed, including: identifying the medical image to be processed by a first recognition model obtained through pre-training to obtain the target and target state in the medical image to be processed corresponding to the first recognition model, wherein the first recognition model is a deep learning model including an attention mechanism; and identifying the medical image to be processed by a second recognition model obtained through pre-training to obtain the target and target state in the medical image to be processed corresponding to the second recognition model, wherein the second recognition model is a recognition model generated by feature vectors and a classifier.

[0097] In this embodiment, the recognition model includes two types, but there is no specific limit to the number of recognition models. The first recognition model is a deep learning model that includes an attention mechanism, which can reduce the influence of the distance and angle between the surgical instruments and the patient and the AR image device. The second recognition model is a recognition model generated by feature vectors and classifiers, which can improve processing efficiency.

[0098] Specifically, in combination Figure 8 As shown, the training method for the first recognition model can include: receiving and labeling sample datasets, such as receiving sample images of instruments and wounds, labeling the sample images, and then preprocessing the labeled sample images. For example, the sample images can be divided into data of normal scenes and data of complex scenes, and then the data of complex scenes can be further subdivided. Initially, unrecognized and unlabeled images from the initial virtual display are labeled. A large amount of data on instruments and wounds during surgery is collected. Due to video encoding issues and special circumstances during the collection process, such as shadows caused by excessive movement, inconsistent image clarity and blurriness due to distance issues with AR glasses, inconsistent field of view, smoke, reflections, and excessive bleeding covering surgical instruments, etc., the sample images are divided into data of normal scenes and data of complex scenes, and then the data of complex scenes is further subdivided. Then, a neural network model is built and trained to obtain the trained first recognition model.

[0099] When training the first recognition model, such as a neural network recognition model, we first perform coarse classification on common scenes, and then perform fine classification on complex scenes. This is similar to letting the neural network model learn common knowledge first and then learn obscure knowledge.

[0100] Correspondingly, the pre-trained first recognition model identifies the medical image to be processed, obtaining the target and target state in the medical image. This includes: sequentially identifying the medical image to be processed using at least two layers of the pre-trained first recognition model, obtaining the target and target state in the medical image. The input to the next network structure in each of the at least two network layers is the target output of the adjacent previous network structure, and the outputs of each network layer that are not input to the next network structure are the target and target state in the medical image. The two network layers correspond to different classifications, for example, the first network layer performs a coarse classification of conventional scenes to obtain normal scenes and complex scenes, while the second network layer performs a fine classification of complex scenes. In other embodiments, other network layers can be added to achieve more detailed classification, with the outputs of each network layer that are not input to the next network structure being the target and target state in the medical image.

[0101] In surgical scene observations, routine situations constitute a relatively large proportion. Optionally, when building the neural network, different scale convolutional boxes are added to a mature basic model framework and layered and fused together, i.e., an attention mechanism. This counteracts the influence of different instrument and wound image scales compared to repeatedly learning from images of different sizes. Then, multi-process, multi-GPU training is used to accelerate model output. Finally, the neural network model is used for inference to identify the target and target state, such as identifying patient wounds and surgical instruments.

[0102] In the above embodiments, the problem of sample imbalance during model training is solved, and the accuracy of identification and classification of patient wounds and instrument conditions during surgery is improved.

[0103] In one embodiment, a pre-trained second recognition model is used to identify the medical image to be processed, and to obtain the target and target state in the medical image to be processed. This includes: extracting image information from the medical image to be processed; extracting feature vectors from the image information to obtain an initial feature vector; mapping the initial feature vector to obtain a target feature vector; and obtaining the target and target state in the medical image to be processed based on the target feature vector using a pre-trained classifier.

[0104] Specifically, the second recognition model is a traditional model that primarily trains on the dimension mapping matrix corresponding to the target feature vector to improve classification accuracy. In practical applications, it combines... Figure 9 As shown, the training method of the second recognition model includes: receiving and labeling sample datasets, such as receiving sample images of instruments and wounds, and labeling the sample images. The labeling method can be referred to the labeling method of the sample images of the first recognition model, and will not be repeated here.

[0105] The sample images are then preprocessed to extract image information, such as the RGB three-channel matrix information, with dimensions consistent with the image's width and height. Next, a search box is set up, the entire image is traversed, and regional image patches are extracted. Then, the HOG operator feature operation is performed to map the three-dimensional matrix information into a one-dimensional vector, thus using a one-dimensional vector to represent the image patch.

[0106] Similarly, the standard image is processed in the same way as the sample image. Then, the one-dimensional vector corresponding to the sample image is compared with the one-dimensional vector corresponding to the standard image. The recognition result is obtained by the distance between the two vectors. For example, if the distance between the two vectors is less than a threshold, the category of the corresponding standard image is obtained as the category of the sample image.

[0107] The distance between two vectors can be calculated using an SVM approach. This involves mapping a one-dimensional vector to a higher dimension using a dimension mapping matrix, comparing the distances between the mapped higher-dimensional vectors, and finally updating the dimension mapping matrix based on the category of the obtained sample image and the category of the annotation. This process continues until the category of the sample image and the category of the annotation are the same, thus obtaining the dimension mapping matrix.

[0108] In practical applications, this involves extracting image information from the medical image to be processed; extracting feature vectors from the image information to obtain initial feature vectors; mapping the initial feature vectors to obtain target feature vectors; and using a pre-trained classifier based on the target feature vectors to obtain the target and its state in the medical image to be processed. Specifically, mapping the initial feature vectors to obtain the target feature vectors includes: using a pre-trained dimension mapping matrix to map the initial feature vectors to obtain the target feature vectors.

[0109] In the above embodiments, the medical images to be processed are also processed using a traditional recognition model generated by feature vectors and classifiers, which can improve processing efficiency, and the accuracy is also guaranteed by combining the results of multiple recognition models.

[0110] In one embodiment, the target and target state obtained by each recognition model are fused, including: obtaining the weights corresponding to each recognition model; and fusing the target and target state obtained by each recognition model based on the weights corresponding to the recognition models.

[0111] The weights of each model can be pre-generated. For example, after training, the confidence of each recognition model can be calculated using test samples. The confidence of each recognition model can be normalized and used as the weight of each recognition model. In actual processing, the target and target state obtained by each recognition model can be fused based on the weight of the recognition model. For example, the results of each recognition model and the weight can be weighted to obtain the fused target and target state.

[0112] Specifically, in combination Figure 10 As shown, before fusing the targets and target states obtained by each recognition model based on the weights corresponding to the recognition models, the process further includes: obtaining the processing time and target recognition accuracy of the first and second recognition models; when the difference between the processing time of the first and second recognition models is greater than a threshold, the target and target state corresponding to the second recognition model are used as the fused target and target state; when the difference between the processing time of the first and second recognition models is less than or equal to the threshold, and the recognition accuracy of the first recognition model is greater than the recognition accuracy of the second recognition model, the targets and target states obtained by each recognition model are further fused based on the weights corresponding to the recognition models; when the difference between the processing time of the first and second recognition models is less than or equal to the threshold, and the recognition accuracy of the first recognition model is less than or equal to the recognition accuracy of the second recognition model, the target and target state corresponding to the second recognition model are used as the fused target and target state.

[0113] Specifically, the processor first obtains the processing time, target recognition accuracy, and weights corresponding to the first and second recognition models. Then, it calculates whether the difference between the processing times of the first and second recognition models is greater than a threshold. If the difference between the processing times of the first and second recognition models is greater than the threshold, the target and target state corresponding to the second recognition model are used as the fused target and target state.

[0114] When the difference between the processing time of the first recognition model and the processing time of the second recognition model is less than or equal to the threshold, it is further determined whether the target recognition accuracy of the first recognition model is greater than that of the second recognition model. When the recognition accuracy of the first recognition model is greater than that of the second recognition model, the targets and target states obtained by each recognition model are fused based on the weights corresponding to the recognition models. When the recognition accuracy of the first recognition model is less than or equal to that of the second recognition model, the target and target state corresponding to the second recognition model are used as the fused target and target state.

[0115] In the above embodiments, the results of each recognition model are fused by combining the processing time of each recognition model, the recognition accuracy of the target, and the weight, which improves processing efficiency while ensuring accuracy.

[0116] In one embodiment, mapping the fused target and target state to the virtual space corresponding to the augmented reality device for display includes: mapping the fused target and target state to the three-dimensional space of the augmented reality device according to the conversion relationship of multiple acquisition devices of the augmented reality device; and mapping the target and target state in the three-dimensional space to the virtual space corresponding to the augmented reality device for display.

[0117] Specifically, in combination Figure 11 As shown, Figure 11 This is a flowchart of the display steps of an augmented reality device in one embodiment. In this embodiment, the recognition fusion result is initially obtained because the augmented reality device has been calibrated at the factory with multi-camera and multi-sensor conversion relationships, i.e., a mapping matrix. In this way, the recognition results in the two-dimensional image, i.e., coordinates, category, and state, are mapped to the AR space, i.e., the three-dimensional image, through this mapping matrix.

[0118] Then, a consistency transformation is performed between the virtual space scanned by AR and the 3D image, that is, the recognition results are integrated into the AR virtual space. Finally, based on the recognition results, namely coordinates and recognized tiles, registration and display are performed. The coordinates are the virtual display position, the category is the corresponding display model, and the status is the severity of the wound or the coordination status of surgical instruments with the surgical procedure, thereby providing reminders and alarms. For example, if massive bleeding is detected, an alarm will be triggered. The display models are pre-stored in a model library, and different display models are used to display different target categories.

[0119] In one embodiment, such as Figure 12 As shown, a medical image processing method is provided, which can be applied to... Figure 1 Taking the processor in the example, the explanation includes the following steps:

[0120] S1202: Obtain the target and target state based on the medical image display method in any of the above embodiments.

[0121] Specifically, the objectives and the methods for obtaining them can be found above and will not be repeated here.

[0122] S1204: Match the target and its state with a preset scenario.

[0123] S1206: Generate alarm information based on the matching results.

[0124] Specifically, the preset scenario can be a scenario corresponding to the standard operation set according to different operation stages. For example, at the beginning of the operation, the preset scenario includes each standard operation device and the status of the standard operation device. During the operation, the preset scenario is a scenario of standard tissue parts and the status of standard tissue parts or a scenario of standard process.

[0125] Optionally, matching the target and target state with a preset scenario includes at least one of the following: matching the acquired operating device and operating device state with the standard operating device and standard operating device state in the preset scenario; matching the acquired tissue part and tissue part state with the standard tissue part and standard tissue part state in the preset scenario; and determining the operation process based on the acquired target and target state, and matching the operation process with the standard process in the preset scenario.

[0126] Specifically, in combination Figure 13 As shown, Figure 13 This is a flowchart illustrating the alarm information generation steps in one embodiment. In this embodiment, the doctor first puts on AR glasses and enters the surgical scene inspection stage, needing to observe the overall operating room scene from multiple angles. Secondly, multi-target recognition is performed on the real-time video of the scene. Targets can include the surgical robot system, the number of trolleys, various equipment, and the operating table. The multi-target recognition method can be the medical image display method described above. The identified targets are then compared with the robot system type, trolley number, equipment shape, and placement method preset in the pre-operative plan for the type of surgery to determine if an alarm of "surgical scene and surgical instruments mismatch" will be triggered. If so, different levels or types of alarms are generated based on the comparison results. Otherwise, the doctor, wearing AR glasses, approaches the operating table and carefully observes the patient's pre-operative preparation from multiple angles. Based on the initial medical images collected by the AR glasses, the doctor identifies exposed tissue areas or wound conditions and compares them with the preset surgical procedures in the pre-operative plan to determine if an alarm of "surgical procedure type and wound location mismatch" will be triggered. If so, different levels or types of alarms are generated. Otherwise, once the pre-operation environmental check is completed and the doctor begins surgery, the processor automatically activates the "surgical procedure monitoring" program. If there are any errors, omissions, or additions in the steps during the surgery, an alarm of the "surgical procedure error" type will be triggered.

[0127] In the above embodiments, by identifying the target and its state before and during the operation, different types of alarms are triggered to assist doctors in safely completing the surgery. Under the premise of low computational cost, the accuracy of the surgery is improved, laying the foundation for intelligent surgery in the future.

[0128] To enable those skilled in the art to fully understand this application, please refer to... Figures 14 to 17 As shown, before the surgery began, the doctor wore AR glasses and saw the following image in the AR glasses: Figure 14 The glasses allow users to simultaneously see both the real world and a virtual world projected onto the glasses. First, the system platform must be activated; the software interface will then appear on the glasses, which can be accessed with a simple tap.

[0129] The next screen will redirect you to the procedure selection interface. Since the specific surgery required is determined before the patient enters the operating room, you must first select the procedure. This process is also related to... Figure 15 As shown, you can simply tap with your finger, and you can also swipe up or down the list of procedures.

[0130] At the start or during surgery, AR glasses can capture two-dimensional images, i.e., initial medical images, which are then processed by a processor to identify the target and its state, for example... Figure 16 The instrument and wound were identified to prepare for subsequent localization.

[0131] Optionally, on each screen, a surgical reminder menu can be displayed above the doctor's field of vision without obstructing the surgical area. For example... Figure 17 In the diagram, the upper left shows the previous surgical procedure, the middle shows the ongoing surgical procedure, and the right shows the next surgical procedure. This helps to remind the doctor and facilitates the entire surgical operation.

[0132] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0133] Based on the same inventive concept, this application also provides a medical image display device and a medical image processing device for implementing the medical image display method and medical image processing method described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the medical image display device and medical image processing device provided below can be found in the limitations of the medical image display method and medical image processing method described above, and will not be repeated here.

[0134] In one embodiment, a medical image display device is provided, comprising:

[0135] The initial medical image acquisition module is used to acquire the initial medical images captured by the augmented reality device;

[0136] The image enhancement module is used to enhance the initial medical image to obtain the medical image to be processed;

[0137] The model processing module is used to identify the medical image to be processed using at least one recognition model obtained through pre-training, and to obtain the target and target state in the medical image to be processed.

[0138] The fusion module is used to fuse the targets and target states obtained from each recognition model;

[0139] The display module is used to map the fused target and its state to the virtual space corresponding to the augmented reality device for display.

[0140] In one embodiment, the image enhancement module is further configured to convert the initial medical image into a grayscale image; calculate the gradient histogram and grayscale target value of the grayscale image; obtain the brightness coefficient to be processed corresponding to the initial medical image based on the gradient histogram and grayscale target value; calculate the corresponding color integral image to be processed based on the initial medical image; and perform image enhancement processing on the initial medical image based on the color integral image to be processed and the brightness coefficient to be processed to obtain the medical image to be processed.

[0141] In one embodiment, the image enhancement module is further configured to calculate the gradient histogram of the grayscale image and the grayscale target value; and to obtain the brightness coefficient to be processed corresponding to the initial medical image based on the gradient histogram and the grayscale target value.

[0142] In one embodiment, the image enhancement module is further configured to match the color integral image to be processed with a standard color integral image to determine the standard brightness coefficient corresponding to the brightness coefficient to be processed; and update the pixel values ​​in the initial medical image using the standard brightness coefficient to obtain the medical image to be processed.

[0143] In one embodiment, the above-mentioned model processing module is further configured to recognize the medical image to be processed using a pre-trained first recognition model to obtain the target and target state in the medical image to be processed corresponding to the first recognition model, wherein the first recognition model is a deep learning model including an attention mechanism; and to recognize the medical image to be processed using a pre-trained second recognition model to obtain the target and target state in the medical image to be processed corresponding to the second recognition model, wherein the second recognition model is a recognition model generated by feature vectors and a classifier.

[0144] In one embodiment, the above-mentioned model processing module is further configured to sequentially identify the medical image to be processed through at least two layers of the network structure of the first recognition model obtained in pre-training, so as to obtain the target and target state in the medical image to be processed, wherein the input of the next network structure in the at least two layers of the network structure is the target output of the adjacent previous network structure, and the output of each layer of the network structure that is not input to the next network structure is the target and target state in the medical image to be processed.

[0145] In one embodiment, the above-mentioned model processing module is further used to extract image information of the medical image to be processed; extract feature vectors from the image information to obtain an initial feature vector; perform dimensional mapping on the initial feature vector to obtain a target feature vector; and obtain the target and target state in the medical image to be processed based on the target feature vector using a pre-trained classifier.

[0146] In one embodiment, the model processing module is further configured to perform dimensional mapping on the initial feature vector to obtain the target feature vector using a pre-trained dimensional mapping matrix.

[0147] In one embodiment, the fusion module is further configured to obtain the weights corresponding to each recognition model; and to fuse the target and target state obtained by each recognition model based on the weights corresponding to the recognition models.

[0148] In one embodiment, the fusion module is further configured to obtain the processing time and target recognition accuracy corresponding to the first recognition model and the second recognition model; when the difference between the processing time of the first recognition model and the processing time of the second recognition model is greater than a threshold, the target and target state corresponding to the second recognition model are used as the fused target and target state; when the difference between the processing time of the first recognition model and the processing time of the second recognition model is less than or equal to the threshold, and the recognition accuracy of the first recognition model is greater than the recognition accuracy of the second recognition model, the targets and target states obtained by each recognition model are further fused based on the weights corresponding to the recognition models; when the difference between the processing time of the first recognition model and the processing time of the second recognition model is less than or equal to the threshold, and the recognition accuracy of the first recognition model is less than or equal to the recognition accuracy of the second recognition model, the target and target state corresponding to the second recognition model are used as the fused target and target state.

[0149] In one embodiment, the display module is further configured to map the fused target and target state to the three-dimensional space of the augmented reality device according to the conversion relationship of multiple acquisition devices of the augmented reality device; and to map the target and target state in the three-dimensional space to the virtual space corresponding to the augmented reality device for display.

[0150] In one embodiment, a medical image processing apparatus is provided, comprising:

[0151] The identification module is used to acquire the target and the target status based on the medical image display device in any of the above embodiments;

[0152] The matching module is used to match the target and its state with a preset scene;

[0153] The alarm module is used to generate alarm information based on the matching results.

[0154] In one embodiment, the matching module matches the target and target state with a preset scenario in at least one of the following ways: matching the acquired operating device and operating device state with the standard operating device and standard operating device state in the preset scenario; matching the acquired tissue part and tissue part state with the standard tissue part and standard tissue part state in the preset scenario; and determining the operation process based on the acquired target and target state, and matching the operation process with the standard process in the preset scenario.

[0155] The modules in the aforementioned medical image display device and medical image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0156] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 18As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a medical image display method and a medical image processing method. The display unit of the computer device is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0157] Those skilled in the art will understand that Figure 18 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0158] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0159] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0160] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0161] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0162] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for displaying medical images, characterized in that, The method includes: Acquire initial medical images captured by augmented reality devices; The initial medical image is subjected to image enhancement processing to obtain the medical image to be processed; The medical image to be processed is identified by at least one recognition model obtained through pre-training, and the target and target state in the medical image to be processed are obtained corresponding to each recognition model. The process of fusing the targets and target states obtained by each of the recognition models includes: obtaining the weights corresponding to each of the recognition models; and fusing the targets and target states obtained by each of the recognition models based on the weights corresponding to the recognition models. The merged target and its state are mapped to the virtual space corresponding to the augmented reality device for display. Before fusing the targets and target states obtained by each recognition model based on the weights corresponding to the recognition models, the method further includes: The processing time and target recognition accuracy of the first and second recognition models are obtained. The first recognition model is a deep learning model that includes an attention mechanism to reduce the influence of the distance and angle between the surgical instruments and the patient and the AR image device. The second recognition model is a recognition model generated by feature vectors and a classifier to improve processing efficiency. When the difference between the processing time of the first recognition model and the processing time of the second recognition model is greater than a threshold, the target and target state corresponding to the second recognition model are used as the fused target and target state. When the difference between the processing time of the first recognition model and the processing time of the second recognition model is less than or equal to the threshold, and the recognition accuracy of the first recognition model is greater than the recognition accuracy of the second recognition model, the target and target state obtained by each recognition model are further fused based on the weights corresponding to the recognition models. When the difference between the processing time of the first recognition model and the processing time of the second recognition model is less than or equal to the threshold, and the recognition accuracy of the first recognition model is less than or equal to the recognition accuracy of the second recognition model, the target and target state corresponding to the second recognition model are used as the fused target and target state.

2. The method according to claim 1, characterized in that, The process of enhancing the initial medical image to obtain the medical image to be processed includes: The initial medical image is converted into a grayscale image; the gradient histogram and target grayscale value of the grayscale image are calculated. Based on the gradient histogram and the grayscale target value, the brightness coefficient to be processed corresponding to the initial medical image is obtained; The corresponding color integral image to be processed is calculated based on the initial medical image; The initial medical image is enhanced based on the color integral image to be processed and the brightness coefficient to be processed to obtain the medical image to be processed.

3. The method according to claim 2, characterized in that, The step of performing image enhancement processing on the initial medical image based on the color integral image to be processed and the brightness coefficient to be processed to obtain the medical image to be processed includes: The color integral image to be processed is matched with the standard color integral image to determine the standard luminance coefficient corresponding to the luminance coefficient to be processed. The pixel values ​​in the initial medical image are updated using the standard brightness coefficient to obtain the medical image to be processed.

4. The method according to any one of claims 1 to 3, characterized in that, The step of identifying the medical image to be processed using at least one pre-trained recognition model to obtain the target and target state in the medical image to be processed corresponding to each recognition model includes: The medical image to be processed is identified by a first recognition model obtained through pre-training, and the target and target state in the medical image to be processed are obtained. The medical image to be processed is identified by a second recognition model obtained through pre-training, thereby obtaining the target and target state in the medical image to be processed that correspond to the second recognition model.

5. The method according to claim 1, characterized in that, The step of mapping the fused target and target state to the virtual space corresponding to the augmented reality device for display includes: Based on the conversion relationship of multiple acquisition devices of the augmented reality device, the fused target and target state are mapped onto the three-dimensional space of the augmented reality device; The target and its state in the three-dimensional space are mapped to the virtual space corresponding to the augmented reality device for display.

6. A medical image processing method, characterized in that, The medical image processing method includes: The medical image display method according to any one of claims 1 to 5 is used to obtain the target and the target state; Match the target and its state with a preset scenario; An alarm message is generated based on the matching results.

7. The medical image processing method according to claim 6, characterized in that, The step of matching the target and the target state with a preset scene includes at least one of the following: The acquired operating devices and their states are matched with the standard operating devices and their states in the preset scenario. The obtained tissue parts and their states are matched with the standard tissue parts and their states in the preset scene. and The operation process is determined based on the obtained target and the target state, and the operation process is matched with the standard process in the preset scenario.

8. An image display system, characterized in that, The system includes: Augmented reality devices are used to acquire initial medical images and display information processed by a display processor. A processor is configured to perform the steps of the method according to any one of claims 1 to 5 or 6 to 7 to obtain display information of the virtual space corresponding to the augmented reality device.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5 or 6 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5 or 6 to 7.

Images