Medical Image Display Method, Device, Electronic Device, and Storage Medium

Abstract

The present invention discloses a medical image display method, device, electronic device and storage medium, which acquire target position information input by a user and a target medical image video corresponding to the target position information; determine target scan information matching the target position information; obtain a target medical image corresponding to the target scan information in the target medical image video according to a preset mapping relationship, and display the target medical image; by acquiring the scan information of the medical image video, the mapping relationship is obtained through the association and matching between the scan information and the medical image video. In this way, the scan information and the video frame corresponding to the scan information are associated together through the mapping relationship. When the user inputs the target position information, the video frame matching the target scan information corresponding to the target position information input by the user is provided to the user, so that the video image of the area to be observed can be quickly retrieved, the display efficiency of the medical image video is improved, and the time cost is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a method, device, electronic device and storage medium for medical image display. Background Art

[0002] Ultrasound scanning is a relatively common imaging method in AI medicine. Generally, during ultrasound scanning, a human operator uses an ultrasound device to scan a target area of the human body. After multiple scans, multiple scanned video images are obtained. Usually, when a user views a medical image video, the videos are played one by one. In the case of multiple video images, when encountering the same area or the same lesion, the user needs to repeatedly flip through multiple video images for further observation and diagnosis. This method requires spending a lot of time to search for the video image of the area or lesion to be viewed from multiple video images, resulting in a high time cost for displaying multiple medical image videos. Summary of the Invention

[0003] Embodiments of the present invention provide a method, device, electronic device and storage medium for medical image display to improve the problem of high time cost in displaying multiple existing medical image videos.

[0004] On the one hand, embodiments of the present invention provide a method for medical image display, the method comprising:

[0005] Obtaining target position information input by a user based on a displayed 3D model view and a target medical image video corresponding to the target position information;

[0006] Determining target scan information matching the target position information;

[0007] According to a preset mapping relationship, obtaining a target medical image corresponding to the target scan information in the target medical image video, and displaying the target medical image; the mapping relationship is obtained by acquiring scan information of the medical image video and associatively matching the scan information with the medical image video.

[0008] On the other hand, embodiments of the present invention provide a device for medical image display, the device comprising:

[0009] An interaction module, configured to obtain target position information input by a user based on a displayed 3D model view and a target medical image video corresponding to the target position information;

[0010] An obtaining module, configured to determine target scan information matching the target position information;

[0011] A display module, configured to obtain a target medical image corresponding to the target scan information in the target medical image video according to a preset mapping relationship, and display the target medical image; the mapping relationship is obtained by acquiring the scan information of the medical image video and associatively matching the scan information with the medical image video.

[0012] On the other hand, an embodiment of the present invention provides an electronic device, including a memory and a processor; the memory stores an application program, and the processor is configured to run the application program in the memory to execute the operations in the medical image display method described above.

[0013] On the other hand, an embodiment of the present invention provides a storage medium, which stores multiple instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the medical image display method.

[0014] In an embodiment of the present invention, the target position information input by the user based on the displayed 3D model view and the target medical image video corresponding to the target position information are obtained; the target scan information matching the target position information is determined; according to the preset mapping relationship, the target medical image corresponding to the target scan information is obtained in the target medical image video, and the target medical image is displayed; the mapping relationship is obtained by acquiring the scan information of the medical image video and associatively matching the scan information with the medical image video. In this way, the scan information and the video frames corresponding to the scan information are associated together through the mapping relationship. When the user inputs the target position information through operations such as dragging and clicking, the video frames matching the target scan information corresponding to the target position information input by the user in multiple medical image videos are provided to the user at the same time, so that the video images of the area to be observed can be quickly retrieved, the display efficiency of multiple medical image videos is improved, and the time cost is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative efforts.

[0016] Figure 1 FIG. is a schematic diagram of an application scenario of the medical image display method provided by an embodiment of the present invention;

[0017] Figure 2 FIG. is a schematic flowchart of an embodiment of the medical image display method provided by an embodiment of the present invention;

[0018] Figure 3It is a flowchart of an embodiment for establishing a mapping relationship based on scanning points in the medical image display method provided by an embodiment of the present invention;

[0019] Figure 4 It is another flowchart of an embodiment for establishing a mapping relationship based on scanning points in the medical image display method provided by an embodiment of the present invention;

[0020] Figure 5 It is a flowchart of an embodiment for establishing a mapping relationship based on feature points provided by an embodiment of the present invention;

[0021] Figure 6 It is a schematic structural diagram of a medical image display device provided by an embodiment of the present invention;

[0022] Figure 7 It is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0023] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present invention.

[0024] An embodiment of the present invention provides a medical image display method, device, electronic device, and storage medium. According to the embodiment of the medical image display method provided by the embodiment of the present invention, it should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps described or shown can be executed in an order different from that here.

[0025] As Figure 1 shown, Figure 1 It is a schematic diagram of an application scenario of the medical image display method provided by an embodiment of the present invention. The shown application scenario includes a database 102, a server 101, and a terminal 103.

[0026] The server 101 can be an independent server or a server network or server cluster composed of servers, such as a computer, a network host, a single network server, a set of multiple network servers, or a cloud server composed of multiple servers. Among them, the cloud server is composed of a large number of computers or network servers based on cloud computing.

[0027] The server 101 can be provided with one or more functional units. In some embodiments of the present invention, as Figure 1As shown, the server 101 may include an interface unit, a query unit, and a processing unit.

[0028] Among them, the interface unit is used to communicate with the terminal 103, obtain the target location information sent by the terminal 103, and return the target medical image to the terminal 103.

[0029] The query unit is used to access the database 102 to obtain medical image videos.

[0030] The processing unit is used to determine the target scan information that matches the target location information according to the target medical image video corresponding to the target location information, and obtain the target medical image corresponding to the target scan information in the target medical image video according to the preset mapping relationship, and display the target medical image.

[0031] The above database 102 may be a Redis database (full name: Remote Dictionary Server), an SQL (full name: Structured Query Language) database, or other types of databases. The database 102 is used to store medical image videos, preset mapping relationships, etc.

[0032] The terminal 103 may have a network connection function. For example, the terminal 103 is an electronic device with a display function, including but not limited to a computer, a tablet computer, a personal computer, a smart display, etc. In some embodiments of the present invention, the terminal 103 is connected to the server 101 through a communication network. In some embodiments of the present invention, the communication network may be a wired network or a wireless network. In some embodiments of the present invention, the above wired network or wireless network uses standard communication technologies and / or protocols. The network may be the Internet or any network, including but not limited to a wide area network, a metropolitan area network, a local area network, a 3rd Generation Partnership Project (3GPP), a Long Term Evolution (LTE), a Worldwide Interoperability for Microwave Access (WiMAX) mobile communication, or a computer network communication based on the TCP / IP protocol suite (TCP / IP Protocol Suite TCP / IP), User Datagram Protocol (UDP).

[0033] As Figure 2 shown, Figure 2is a schematic flowchart of an embodiment of the medical image display method provided by an embodiment of the present invention. The medical image display method shown includes steps 201 to 203:

[0034] 201. Obtain the target position information input by the user based on the displayed 3D model view and the target medical image video corresponding to the target position information.

[0035] Among them, the target position information may be target coordinate information, or the target position information may also be target coordinate information and target viewing angle information. Among them, the target viewing angle information may be the display angle of the 3D model in the 3D model view.

[0036] The 3D model view refers to the view of the 3D model of the interested part of the display object, where the object may be a patient receiving medical treatment, a tester, an object to be scanned, etc.; the interested part may be an organ, tissue or component of the object in the interested area, and the interested area may be a pre-marked area to be scanned. In some embodiments of the present invention, after obtaining the medical image video of the interested area of the object, three-dimensional reconstruction may be performed according to the medical image video to obtain the 3D model of the interested part of the object.

[0037] The medical image may be a CT image, an MRI image, a DSA image, an ultrasound image, etc. The medical image video may be a video image obtained by scanning the interested part of the object with a scanning device. The scanning device includes, but is not limited to, a computed tomography (CT) device, a magnetic resonance imaging (MRI) device, a digital subtraction angiography (DSA) device, an ultrasound image scanning device, etc.

[0038] In some embodiments of the present invention, step 201 includes: responding to the object identifier input by the user, obtaining the 3D model view corresponding to the object identifier and the medical image video corresponding to the 3D model, displaying the 3D model view, obtaining the target position information input by the user based on the 3D model view, and obtaining the target medical image video corresponding to the target position information from the medical image video corresponding to the 3D model.

[0039] In some embodiments of the present invention, there are multiple ways to obtain the target medical image video corresponding to the target position information. Exemplarily, they include:

[0040] (1) When the target position information is target coordinate information, the coordinate information corresponding to each frame of medical image in the medical image video corresponding to the 3D model can be obtained, and the target coordinate information is compared with each coordinate information; the coordinate information in each coordinate information that matches the target coordinate information is determined, and the medical image corresponding to the coordinate information that matches the target coordinate information in each coordinate information in the medical image video corresponding to the 3D model is extracted; the target medical image video corresponding to the target position information is obtained based on the extracted medical images.

[0041] In some embodiments of the present invention, when there is one frame of image among the extracted medical images, the extracted medical image is used as a reference frame image, a preset number of preceding frame images of the reference frame image are extracted from the medical image video corresponding to the 3D model, and a preset number of subsequent frame images of the reference frame image are extracted from the medical image video corresponding to the 3D model. The extracted preceding frame images, reference frame image, and subsequent frame images are combined to obtain continuous frame images, and the continuous frame images are set as the target medical image video corresponding to the target position information.

[0042] In some embodiments of the present invention, when there are multiple frames of images among the extracted medical images, the smallest frame image and the largest frame image among the multiple frames of images are obtained, and a continuous multiple frames of images between the smallest frame image and the largest frame image are extracted from the medical image video corresponding to the 3D model. The extracted continuous multiple frames of images are set as the target medical image video corresponding to the target position information.

[0043] (2) When the target position information is target perspective information, the 3D model in the 3D model view can be projected onto the target perspective corresponding to the target perspective information to obtain the target area information corresponding to the 3D model under the target perspective information; the area information on each frame of medical image in the medical image video corresponding to the 3D model is obtained; the area information on each frame of medical image is respectively compared with the target area information, and based on the comparison result, the medical images with area information matching the target area information are extracted from the medical image video corresponding to the 3D model, and the target medical image video corresponding to the target position information is obtained based on the extracted medical images. Among them, the area information can be the local area of an organ, tissue, or component of an object shown in the medical image in the region of interest. In some embodiments of the present invention, when the target position information is perspective information, the method for obtaining the target medical image video corresponding to the target position information based on the extracted medical images is the same as the method described above when the target position information is target coordinate information, and will not be elaborated here.

[0044] (3) When the target position information is target perspective information, the perspective of the medical image video corresponding to the 3D model can be obtained, the target perspective of the target perspective information can be obtained, and the target perspective can be compared with the perspective of the medical image video; if the target perspective is consistent with the perspective of the medical image video, the medical image video corresponding to the 3D model is set as the target medical image video corresponding to the target position information; if the target perspective is inconsistent with the perspective of the medical image video, the medical image video is adjusted according to the 3D model and the target perspective to obtain a medical image video with adjusted perspective, and the medical image video with adjusted perspective is set as the target medical image video corresponding to the target position information. In some embodiments of the present invention, the 3D model can be mapped to the target perspective to obtain the target area of the 3D model under the target perspective, the medical image containing the target area in the medical image video corresponding to the 3D model is obtained, and the selected medical image is rotationally processed according to the target perspective information to obtain the rotationally processed selected medical image, and the rotationally processed selected medical image is set as the target medical image video corresponding to the target position information.

[0045] In some embodiments of the present invention, after the selected medical image is rotationally processed according to the target perspective information to obtain the rotationally processed selected medical image, if the input target position information only includes target perspective information, the rotationally processed selected medical image is set as the target medical image video corresponding to the target position information; if the input target position information further includes at least one of the target coordinate information, the rotationally processed selected medical image is set as the candidate medical image video, and the target medical image video corresponding to the target coordinate information is extracted from the candidate medical image video in the above manner.

[0046] (4) When the target position information includes coordinate information and perspective information, the 3D model in the 3D model view can be projected under the perspective information to obtain the target area information corresponding to the 3D model under the perspective information; the area information on each frame of the medical image in the medical image video corresponding to the 3D model is obtained; the area information on each frame of the medical image is respectively compared with the target area information, and the first medical image with the area information matching the target area information is extracted from the medical image video corresponding to the 3D model according to the comparison result; the candidate target medical image video corresponding to the perspective information is obtained according to the extracted first medical image; the coordinate information corresponding to each frame of the medical image of the candidate target medical image video is compared with the target coordinate information, and the second medical image with the coordinate information matching the target coordinate information is extracted from the candidate target medical image video; the target medical image video corresponding to the target position information is obtained according to the extracted second medical image.

[0047] (5) When the target position information includes target coordinate information and viewing angle information, the candidate target medical image video corresponding to the target coordinate information can also be extracted from the medical image video corresponding to the 3D model in the aforementioned manner; the target medical image video corresponding to the viewing angle information is extracted from the candidate target medical image video corresponding to the target coordinate information.

[0048] (6) When the target position information includes target coordinate information and target viewing angle information, the first candidate target medical image video corresponding to the target coordinate information can also be extracted from the medical image video corresponding to the 3D model in the aforementioned manner, and the second candidate target medical image video corresponding to the target viewing angle information is extracted from the medical image video corresponding to the 3D model; the union image sequence or intersection image sequence of the first candidate target medical image video and the second candidate target medical image video is set as the target medical image video corresponding to the target position information.

[0049] It should be noted that the above acquisition of the target medical image video corresponding to the target position information is only for exemplary illustration and does not constitute a limitation on the medical image display method provided by the embodiments of the present invention.

[0050] 202. Determine the target scan information that matches the target position information.

[0051] The scan information includes but is not limited to scan points and feature points. Among them, the scan point refers to the scan point on the interested part when scanning the interested part of the scanned object, and the feature point refers to the marked feature point on the interested part when scanning the interested part of the scanned object. The target scan information that matches the target position information refers to the target scan point or target feature point in the scan points or feature points of the medical image video whose error corresponding to the target position information input based on the 3D model view is less than or equal to the preset error threshold.

[0052] In some embodiments of the present invention, step 202 includes: comparing the target position information with the scan information corresponding to the medical image video, determining the error between the target position information and each scan information, and setting the scan information with an error less than or equal to the preset error threshold as the target scan information.

[0053] In some embodiments of the present invention, when the target position information is target coordinate information, the coordinate distance between the target coordinate information and the coordinate information of the scan points or feature points corresponding to the medical image video is calculated to obtain the error between the target coordinate information and each scan point or feature point, and the scan points or feature points with an error less than or equal to the preset error threshold are set as the target scan information.

[0054] In some embodiments of the present invention, when the target position information is perspective information, that is, when there is no target coordinate information in the target position information, each frame of medical image in the target medical image video corresponding to the target position information can be set as the target medical image, and the target medical image can be displayed frame by frame.

[0055] In some embodiments of the present invention, when the target position information is perspective information, that is, when there is no target coordinate information in the target position information, the coordinate distance between the coordinate information of the center point of the target area information corresponding to the 3D model under this perspective information and the coordinate information of the scanning point or feature point corresponding to the medical image video can be calculated according to the target area information corresponding to the perspective information under this perspective information, so as to obtain the error between this perspective information and each scanning point or feature point, and the scanning point or feature point with an error less than or equal to the preset error threshold can be set as the target scanning information.

[0056] In some embodiments of the present invention, when the target position information includes perspective information and target coordinate information, the target scanning information matching the target position information can be obtained according to the target scanning information determination method when the above target position information is target coordinate information.

[0057] 203. According to the preset mapping relationship, obtain the target medical image corresponding to the target scanning information in the target medical image video, and display the target medical image.

[0058] The mapping relationship is obtained by acquiring the scanning information of the medical image video and associating and matching according to the scanning information. This mapping relationship is used to represent the corresponding relationship between each frame of medical image in the medical image video and the scanning information. In some embodiments of the present invention, each scanning information corresponds to at least one frame of medical image.

[0059] In some embodiments of the present invention, step 203 includes: according to the preset mapping relationship, obtain the scanning information corresponding to each frame of medical image in the target medical image video, compare the scanning information corresponding to each frame of medical image in the target medical image video with the target scanning information, select the medical image in the target medical image video whose scanning information is consistent with the target scanning information, set the selected medical image whose scanning information is consistent with the target scanning information as the target medical image, and display the target medical image. For example, when the scanning information is a scanning point, according to the preset mapping relationship, obtain the scanning point corresponding to each frame of medical image in the target medical image video, and compare the coordinate information or scanning point identification information of the scanning point corresponding to each frame of medical image in the target medical image video with the coordinate information or scanning point identification of the target scanning point, and select the medical image in the target medical image video whose coordinate information or scanning point identification information of the scanning point is consistent with the coordinate information or scanning point identification of the target scanning point.

[0060] In some embodiments of the present invention, step 203 includes: determining the target number of frames of the medical image corresponding to the target scan information according to the target scan information and a preset mapping relationship; selecting from the target medical image video the medical images that are consistent with the target number of frames, setting the selected medical images that are consistent with the target number of frames as the target medical images, and displaying the target medical images.

[0061] In some embodiments of the present invention, after displaying the target medical image, in response to the annotation information input by the user based on the view of the target medical image, the input annotation information is displayed on the view of the target medical image, where the annotation information includes but is not limited to text, characters, letters, graphs, curves, and straight lines.

[0062] In some embodiments of the present invention, after displaying the target medical image, it is also possible to respond to the second perspective information input by the user based on the view of the target medical image, and rotate the angle of the target medical image displayed in the view of the target medical image according to the second perspective information.

[0063] In some embodiments of the present invention, after displaying the target medical image, it is also possible to respond to the new perspective information input by the user based on the 3D model view, obtain a new target medical image video corresponding to the perspective information from the medical image video corresponding to the 3D model according to the perspective information, extract a new target medical image corresponding to the target scan information from the new target medical image video according to the mapping relationship, and display the new target medical image.

[0064] In the embodiments of the present invention, the scan information is associated with the video frames corresponding to the scan information through a mapping relationship. When the user inputs target position information through operations such as dragging and clicking, the video frames matching the target scan information corresponding to the target position information input by the user in multiple medical image videos are provided to the user at the same time, so that the video images of the area to be observed can be quickly retrieved, the display efficiency of multiple medical image videos is improved, and the time cost is reduced.

[0065] In some embodiments of the present invention, after collecting the medical image video of the object's region of interest through a scanning device, three-dimensional reconstruction is performed according to the image information of each frame of the medical image in the medical image video to obtain a 3D model of the object's region of interest, the 3D model is associated and stored with the identification information of the object's region of interest, and the 3D model of the object's region of interest is associated and stored with the medical image video of the object's region of interest. It should be noted that the embodiments of the present invention do not limit the specific manner of three-dimensional reconstruction.

[0066] In some embodiments of the present invention, after obtaining the 3D model of the region of interest of an object, in response to the user inputting the identification information corresponding to the region of interest, the 3D model associated with the identification information is acquired and the 3D model is displayed on the 3D model view.

[0067] In some embodiments of the present invention, when the 3D model is displayed on the 3D model view, in response to the rotation instruction, view scaling instruction and / or view dragging instruction input by the user, at least one of the operations of rotating, scaling the size and dragging the model of the 3D model displayed on the 3D model view is performed.

[0068] In some embodiments of the present invention, when the 3D model is displayed on the 3D model view, in response to the user inputting an instruction, the target position information input by the user based on the displayed 3D model view is acquired, and the target medical image video corresponding to the target position information is searched from the medical image video corresponding to the 3D model, where the target position information includes coordinate information and viewing angle information.

[0069] In some embodiments of the present invention, the target medical image video corresponding to the target position information can be searched from the medical image video corresponding to the 3D model according to the target medical image video acquisition method in step 201.

[0070] In some embodiments of the present invention, the target medical image video corresponding to the target position information can be selected from the medical image video corresponding to the 3D model by a similarity evaluation method. Specifically, the method for selecting the target medical image video includes steps a1 to a4:

[0071] Step a1, acquiring the target position information input by the user based on the displayed 3D model view.

[0072] Step a2, selecting at least one frame of medical image that matches the target coordinate information in the target position information from the pre-stored medical image videos, and setting the selected at least one frame of medical image and a preset number of consecutive frame medical images adjacent to the at least one frame of medical image as the first candidate medical image video corresponding to the target coordinate information.

[0073] In some embodiments of the present invention, the coordinate information of the image region in the region of interest on each frame of the medical image in the pre-stored medical image video corresponding to the 3D model can be compared with the target coordinate information, and the medical images with the coordinate distance difference between the coordinate information and the target coordinate information less than or equal to the preset coordinate distance difference threshold are selected from the medical image video corresponding to the 3D model. According to the position of the selected medical image in the medical image video, a preset number of consecutive frames of medical images adjacent to the selected medical image are selected from the medical image video, and the selected medical image and the preset number of consecutive frames of medical images adjacent to the selected medical image are set as the first candidate medical image video corresponding to the target coordinate information. Among them, the preset number of consecutive frames of medical images adjacent to the selected medical image can be the previous consecutive frames of medical images of the preset number of the selected medical images, or the subsequent consecutive frames of medical images of the preset number of the selected medical images, or the previous frame medical image and the subsequent frame medical image of the selected medical image, and the number of images of the previous frame medical image and the subsequent frame medical image is equal to the preset number.

[0074] Step a3: Obtain the target region information corresponding to the 3D model in the 3D model view according to the target view information in the target position information, select at least one frame of medical image with the target region information from the pre-stored medical image video, and set the selected at least one frame of medical image and the preset number of consecutive frames of medical images adjacent to the at least one frame of medical image as the second candidate medical image video corresponding to the target view information.

[0075] In some embodiments of the present invention, in the medical image video of the region of interest of the object collected by the scanning device, the image information of each frame of the medical image is determined, where the image information includes the three-dimensional coordinates of the image region in each frame of the medical image, and the three-dimensional coordinates are used to determine the position of the image region on each frame of the medical image in the region of interest of the object.

[0076] In some embodiments of the present invention, step a3 includes: according to the target view information in the target position information, project the 3D model in the 3D model view into the view corresponding to the target view information to obtain the target region information corresponding to the 3D model in the 3D model view, compare the three-dimensional coordinates of the image region in each frame of the medical image with the three-dimensional coordinates corresponding to the target region information, select at least one frame of medical image with the three-dimensional coordinates of the image region the same as or close to the three-dimensional coordinates of the target region information from the medical image video, and set the selected at least one frame of medical image and the preset number of consecutive frames of medical images adjacent to the at least one frame of medical image as the second candidate medical image video corresponding to the target view information.

[0077] In some embodiments of the present invention, at least one medical image in which the three-dimensional coordinates of the image region are the same as or close to the three-dimensional coordinates of the target region information refers to a medical image in which the three-dimensional coordinate difference between the three-dimensional coordinates of the image region of the medical image and the three-dimensional coordinates of the target region information is less than or equal to a preset three-dimensional coordinate difference threshold.

[0078] In some embodiments of the present invention, at least one medical image with target region information can also be selected from a pre-stored medical image video, and the at least one selected medical image and a preset number of consecutive medical images adjacent to the at least one medical image are rotationally processed according to the target perspective information to obtain a second candidate medical image video corresponding to the target perspective information.

[0079] Step a4, determine a target medical image video corresponding to the target position information according to the first candidate medical image video and the second candidate medical image video.

[0080] In some embodiments of the present invention, according to the method in step 201, the intersection of the first candidate medical image video and the second candidate medical image video can be determined according to each frame of the first candidate medical image of the first candidate medical image video and each frame of the second candidate medical image of the second candidate medical image video, and the intersection is set as the target medical image video corresponding to the target position information.

[0081] In some embodiments of the present invention, according to the method in step 202, the union of the first candidate medical image video and the second candidate medical image video can be determined according to each frame of the first candidate medical image of the first candidate medical image video and each frame of the second candidate medical image of the second candidate medical image video, and the union is set as the target medical image video corresponding to the target position information.

[0082] In some embodiments of the present invention, when the input target position information is target perspective information and target coordinate information, the perspective of the medical image video corresponding to the 3D model can also be switched according to the target perspective information, and a target medical image video matching the target coordinate information is obtained from the medical image video after perspective switching. Specifically, the method for determining the target medical image video includes steps b1 to b4:

[0083] Step b1, obtain the target perspective information and target coordinate information in the target position information input by the user based on the displayed 3D model view.

[0084] Step b2, obtain a second candidate medical image video matching the target perspective information from the pre-stored medical image video.

[0085] In some embodiments of the present invention, similar to step 201, the target perspective to be switched can be determined according to the target perspective information, and the target perspective is compared with the perspective of the medical image video; if the target perspective is the same as the perspective of the medical image video, the medical image video corresponding to the 3D model is set as the candidate medical image video matching the target perspective information; if the target perspective is different from the perspective of the medical image video, the medical image video is adjusted according to the 3D model and the target perspective to obtain the medical image video after perspective adjustment, and the medical image video after perspective adjustment is set as the candidate medical image video matching the target perspective information. Among them, adjusting the medical image video according to the 3D model and the target perspective includes image screening and rotation processing. For example, the 3D model is mapped to the target perspective to obtain the target part of the 3D model from the target perspective, the medical images in the medical image video corresponding to the 3D model where the target part exists are obtained, and the selected medical images are rotated according to the angle information to obtain the medical image video after perspective adjustment.

[0086] In some embodiments of the present invention, the target perspective to be switched can be determined according to the target perspective information, and the target perspective is compared with the perspective of the medical image video; if the target perspective is the same as the perspective of the medical image video, the medical image video corresponding to the 3D model is set as the candidate medical image video matching the target perspective information; if the target perspective is different from the perspective of the medical image video, the pre-stored mapping data of medical images and perspectives is queried, and the converted medical image video that is the same as the target perspective is selected from the mapping data of medical images and perspectives, and the selected converted medical image video that is the same as the target perspective is set as the candidate medical image video matching the target perspective information. Among them, the mapping data of medical images and perspectives stores multiple perspectives and the converted medical image video corresponding to each perspective. Among them, the converted medical image video is obtained by setting different preset perspectives and performing image screening and image rotation on the medical images in the medical image video corresponding to the 3D model according to each perspective.

[0087] Step b3: Select at least one medical image that matches the target coordinate information from the second candidate medical image video, and a preset number of consecutive frame medical images adjacent to the at least one medical image.

[0088] Step b4: Set the at least one selected medical image and the preset number of consecutive frame medical images adjacent to the at least one medical image as the target medical image video corresponding to the target position information.

[0089] In some embodiments of the present invention, at least one medical image that matches the target coordinate information can be selected from the candidate medical image video according to the method in step a2.

[0090] In some embodiments of the present invention, after obtaining the target medical imaging video corresponding to the target position information, the target scanning information matching the target position information may be determined according to step 202.

[0091] In some embodiments of the present invention, after determining the target scanning information matching the target position information and the target medical imaging video corresponding to the target position information, according to the preset mapping relationship, the target medical image corresponding to the target scanning information may be obtained from the target medical imaging video, and the target medical image may be displayed. The scanning information includes scanning points and feature points.

[0092] In some embodiments of the present invention, when the scanning information is a scanning point, in order to enable the user to quickly view the video image of the area to be observed, reduce the fatigue of the user, and improve the diagnostic efficiency, after scanning the region of interest of the object to obtain a medical imaging video, the present invention embodiment may determine the scanning path according to the medical imaging video, and determine the medical image corresponding to each scanning point on the scanning path, and associate each scanning point on the scanning path with the medical image corresponding to the scanning point to obtain a preset mapping relationship. Specifically, as Figure 3 shown, Figure 3 is a flowchart of an embodiment of establishing a mapping relationship in the medical image display method provided by the embodiment of the present invention. The method for establishing the mapping relationship shown includes 301 to 303:

[0093] 301, identify the obtained medical imaging video through a preset recognition model to obtain the recognition information of the medical imaging video and the medical image where the recognition information is located.

[0094] Among them, the preset recognition model can be a recognition model based on machine learning, such as a recognition model based on logistic regression, a recognition model based on random forest, a recognition model based on dictionary, etc.; the preset recognition model can also be a recognition model based on neural network, such as a recognition model based on Convolutional Neural Networks (CNN), a recognition model based on De-Convolutional Networks (DN), a Deep Neural Networks (DNN), a recognition model based on Deep Convolutional Inverse Graphics Networks (DCIGN), a recognition model based on Region-based Convolutional Networks (RCNN), a recognition model based on Faster Region-based Convolutional Networks (Faster RCNN), and a recognition model based on Bidirectional Encoder Representations from Transformers (BERT).

[0095] The recognition information is used to represent the identification of key points, the identification of key regions or the identification of target regions in the medical image video. The identification is used to determine the names of the key points, key regions or target regions. In some embodiments of the present invention, the identification can be at least one of letters, numbers, characters and words. Among them, the key points can be pre-marked points in the region of interest of the object, and the key points can also be key points in the region of interest of the object. For example, when the region of interest of the object is a blood vessel, the key points can be the bifurcation point of the main blood vessel, the inflection point, etc. When the region of interest of the object is a mechanical product, the key points can be the connection points of the components in the mechanical product.

[0096] In some embodiments of the present invention, step 301 includes: inputting each frame of medical image in the obtained medical image video into the preset recognition model for recognition, determining the names of the key points, key regions or target regions in the medical image video, and marking the medical images with key points, key regions or target regions; setting the names of the key points, key regions or target regions in the medical image video as the recognition information of the medical image video, and setting the marked medical images as the medical images where the recognition information is located.

[0097] 302. Obtain the scanning path of the medical image video according to the medical image where the recognition information is located.

[0098] The scanning path includes a plurality of scanning points arranged in the scanning order of the region of interest of the object, and each scanning point corresponds to a frame of medical image.

[0099] In some embodiments of the present invention, the frame position of the medical image where the identification information is located in the medical image video can be obtained through the frame order of the medical images in the medical image video and the medical image where the identification information is located. The scanning order between the key points, key regions or target regions of the region of interest of the object is determined according to the frame position of the medical image where the identification information is located in the medical image video. According to the scanning order, the scanning path of the medical image video is obtained.

[0100] In some embodiments of the present invention, image differencing can also be performed on every two adjacent medical images in the medical image video to obtain a sequence of position changes of the region of interest of the object in the medical image video. According to the medical image where the identification information is located and the frame order of the medical images in the medical image video, the scanning order between the key points, key regions or target regions of the region of interest of the object is determined. According to this sequence of position changes and the scanning order, the pre-stored first scanning data is queried to obtain a scanning path that matches the sequence of position changes and the scanning order. The pre-stored first scanning data stores a plurality of scanning paths of the region of interest of the object, as well as the sequence of position changes of the region of interest of the object on each scanning path and the scanning order between the key points, key regions or target regions of the region of interest of the object.

[0101] In some embodiments of the present invention, the frame position of the medical image where the identification information is located in the medical image video can also be obtained through the frame order of the medical images in the medical image video and the medical image where the identification information is located. The scanning order between the key points, key regions or target regions of the region of interest of the object is determined according to the frame position of the medical image where the identification information is located in the medical image video. The pre-stored second scanning data of the region of interest of the object is queried to obtain a scanning path that matches the scanning order. Among them, the first scanning data stores a variety of scanning paths of the region of interest of the object, as well as the scanning order between the key points, key regions or target regions of the region of interest of the object corresponding to each scanning path.

[0102] In some embodiments of the present invention, in step 302, the scanning path of the medical image video can be obtained through the frame order of the medical image video and the medical image where the identification information is located. Specifically, the method for determining the scanning path includes:

[0103] (1) According to the frame order of the medical images in the medical image video and the medical image corresponding to the identification information, the acquisition order of the identification information of the medical image video is obtained.

[0104] (2) Obtain the scanning order of the scanning points of the medical imaging video according to the acquisition order of the recognition information of the medical imaging video; wherein, one recognition information of the medical imaging video corresponds to one scanning point.

[0105] (3) Obtain the scanning path of the medical imaging video according to the scanning order of the scanning points of the medical imaging video.

[0106] In some embodiments of the present invention, when recognition information is recognized in each frame of the medical imaging, the acquisition order of the recognition information can be set as the scanning order of the scanning points to obtain the scanning path; in some embodiments of the present invention, when there is a medical imaging in the medical imaging video that does not have recognition information, interpolation processing can be performed on the medical imaging without recognition information and the medical imaging where the recognition information is located to determine the adjacency relationship between the image area in the medical imaging without recognition information and the image area in the medical imaging where the recognition information is located, and the entire scanning order of the medical imaging video is obtained according to this adjacency relationship, and the entire scanning order of the medical imaging video is set as the scanning path of the medical imaging video.

[0107] In some embodiments of the present invention, before the interested part of the scanning object, at least two preset scanning paths are preset, multiple target preset scanning points in each preset scanning path are marked, and the preset recognition information corresponding to each target preset scanning point is determined. Each preset scanning path corresponds to a preset scanning order of the preset scanning points, and the preset acquisition orders of the target preset scanning points in the preset scanning order of each preset scanning path are different; after the medical imaging video of the interested part of the object is acquired, the acquired medical imaging video is recognized by a preset recognition model to obtain the recognition information of the medical imaging video and the medical imaging where the recognition information is located, and according to the frame order of the medical imaging in the medical imaging video and the medical imaging corresponding to the recognition information, the acquisition order of the recognition information of the medical imaging video is obtained; according to the preset scanning path and the recognition information of the medical imaging video, a candidate scanning path where the preset recognition information corresponding to the target preset scanning point in the preset scanning path matches the recognition information of the medical imaging video is determined; the acquisition order of the recognition information of the medical imaging video is compared with the preset acquisition orders of the target preset scanning points in each candidate scanning path to determine the target preset acquisition order of the target preset scanning points that is the same or similar to the acquisition order of the recognition information of the medical imaging video, and the preset scanning path of the preset scanning points corresponding to the target acquisition order is set as the scanning order of the scanning points of the medical imaging video. According to the scanning order and the preset scanning order corresponding to each candidate scanning path, a candidate scanning path corresponding to the scanning order of the scanning points of the medical imaging video is obtained, and the candidate scanning path corresponding to the scanning order of the scanning points of the medical imaging video is set as the scanning path of the medical imaging video.

[0108] 303. Determine the medical images corresponding to each scan point on the scan path of the medical image video, and associate each scan point on the scan path of the medical image video with the medical image corresponding to the scan point to obtain a preset mapping relationship.

[0109] In some embodiments of the present invention, after obtaining the scan path of the medical image video, each scan point on the scan path of the medical image video is associated with the medical image corresponding to each scan point to obtain a preset mapping relationship.

[0110] In some embodiments of the present invention, when the recognition information is the name of a key area or a target area, the pixel value of each scan point on the scan path can be calculated for similarity with the pixel value of each frame of the medical image in the medical image video, and the medical images in the pixel value of each frame of the medical image in the medical image video whose similarity with the scan point is greater than or equal to a preset similarity threshold are extracted, and the medical images whose similarity with the scan point in the extracted pixel values is greater than or equal to the preset similarity threshold are set as the medical images corresponding to the scan point; each scan point on the scan path of the medical image video is associated with the medical image corresponding to each scan point to obtain a preset mapping relationship; and the key area or target area is associated with the medical image corresponding to the scan point in the key area or target area to establish a mapping relationship between the area and the medical image.

[0111] In some embodiments of the present invention, when the recognition information is the name of a key point, the pixel value of each scan point on the scan path can be calculated for similarity with the pixel value of each frame of the medical image in the medical image video, and the medical images in the pixel value of each frame of the medical image in the medical image video whose similarity with the scan point is greater than or equal to a preset similarity threshold are extracted, and the medical images whose similarity with the scan point in the extracted pixel values is greater than or equal to the preset similarity threshold are set as the medical images corresponding to the scan point; each scan point on the scan path of the medical image video is associated with the medical image corresponding to each scan point to obtain a preset mapping relationship; the name of the key point and the medical image where the key point is located are associated to establish a mapping relationship between the key point and the medical image.

[0112] In some embodiments of the present invention, when the scan information is a scan point, in order to enable the user to quickly view the video images of the area to be observed, reduce the fatigue of the user, and improve the diagnosis efficiency, after scanning the interested part of the object to obtain the medical image video, the present invention embodiment can also establish a mapping relationship according to the spatial change sequence of the scanning device of the interested part of the scanned object. Specifically, as Figure 4 shown Figure 4It is another flowchart of the embodiment for establishing a mapping relationship based on scan points in the medical image display method provided by the embodiments of the present invention. The method for establishing the mapping relationship includes 401 to 404:

[0113] 401. Obtain the spatial coordinate sequence of the scanning device during the acquisition of the medical image video.

[0114] The scanning device is used to acquire the medical image video.

[0115] In some embodiments of the present invention, the coordinate sequence of the scanning device during the acquisition of the medical image video can be collected by a position sensor set in the scanning device, and this coordinate sequence is set as the spatial coordinate sequence of the scanning device during the acquisition of the medical image video. Among them, the position sensor can be a gyroscope.

[0116] In some embodiments of the present invention, by collecting the scanning device image video of the scanning device during the acquisition of the medical image video, the difference is taken between every two adjacent frames of the scanning device images in the scanning device image video to obtain the relative coordinates between the scanning devices in two adjacent frames of the scanning device images. According to the relative coordinates between the scanning devices in every two adjacent frames of the scanning device images, the position change sequence of the scanning device in the scanning device image video is obtained, and this position change sequence is set as the spatial coordinate sequence.

[0117] 402. Obtain the device scanning path of the scanning device according to the spatial coordinate sequence.

[0118] Among them, at least one frame of medical image in the medical image video corresponds to each device scanning point on the device scanning path of the scanning device.

[0119] In some embodiments of the present invention, step 402 includes: taking the difference of each spatial coordinate in the spatial coordinate sequence to obtain the spatial coordinate change sequence corresponding to the spatial coordinate sequence, setting the elements in the spatial coordinate change sequence whose values are not the preset value as each device scanning point, and sorting the device scanning points according to the time sequence between the elements of the spatial coordinate change sequence to obtain the device scanning path of the scanning device. Among them, the preset value can be 0.

[0120] In some embodiments of the present invention, after obtaining each device scanning point, the time information of the device scanning point is determined according to the time information of each element in the spatial coordinate change sequence, and the medical image whose time difference between the time information in the medical image video and the time information of the device scanning point is less than or equal to the preset time difference is associated with the device scanning point.

[0121] 403. Compare each device scan point on the device scan path with each preset scan point on the preset scan path respectively to obtain the target preset scan points on the preset scan path that match each device scan point on the device scan path.

[0122] In some embodiments of the present invention, step 403 includes: obtaining the preset scan path of the region of interest of the object and each preset scan point on the preset scan path, comparing the position information of each device scan point on the device scan path with the position information of each preset scan point on the preset scan path respectively, selecting from the preset scan path of the region of interest of the object the preset scan points whose position difference between the position information of the preset scan points and the position information of each device scan point on the device scan path is less than or equal to the preset position difference threshold, and setting the selected preset scan points as the target preset scan points on the preset scan path that match each device scan point on the device scan path.

[0123] 404. For each device scan point on the device scan path, associate the medical image corresponding to the device scan point with the target preset scan point that matches the device scan point to obtain a preset mapping relationship.

[0124] In some embodiments of the present invention, step 404 includes: for each device scan point on the device scan path, associate the medical image corresponding to the device scan point with the target preset scan point that matches the device scan point to obtain a preset mapping relationship.

[0125] In some embodiments of the present invention, when the scan information is feature points, in order to enable the user to quickly view the video image of the area to be observed, reduce the user's fatigue, and improve the diagnosis efficiency, after scanning the region of interest of the object to obtain a medical image video, the present invention embodiment can also perform feature point detection on the medical image video to obtain the feature points in the medical image video and the medical image where each feature point is located, and associate each feature point in the medical image video with the medical image where the feature point is located to obtain a preset mapping relationship. Specifically, as Figure 5 shown, Figure 5 is a flowchart of an embodiment of establishing a mapping relationship based on feature points provided by the present invention embodiment. The method of establishing a mapping relationship based on feature points shown includes steps 501 to 503:

[0126] 501. Detect the feature points of the obtained medical image video through a preset feature detection model to obtain the feature points in the medical image video and the medical image where each feature point is located.

[0127] Among them, the preset feature detection model can be a machine learning-based recognition model, such as a recognition model based on logistic regression, a recognition model based on random forest, a recognition model based on dictionary, etc.; the preset recognition model can also be a neural network-based recognition model, such as a Convolutional Neural Networks (CNN), a De-Convolutional Networks (DN), a Deep Neural Networks (DNN), a Deep Convolutional Inverse Graphics Networks (DCIGN), a Region-based Convolutional Networks (RCNN), a Faster Region-based Convolutional Networks (Faster RCNN), and a Bidirectional Encoder Representations from Transformers (BERT) model.

[0128] In some embodiments of the present invention, similar to step 301, each frame of medical image in the obtained medical image video is input into the preset feature detection model for feature point detection to determine the feature points in the medical image video, and the medical images with feature points in the medical image video are marked; the marked medical images are set as the medical images where the feature points are located.

[0129] 502. Segment the medical images where each feature point is located to obtain a medical image sequence corresponding to each said feature point.

[0130] In some embodiments of the present invention, step 502 includes: for each feature point, determine whether there are multiple consecutive frames of medical images marked with the feature point in the medical image video according to the medical image where the feature point is located. If there are multiple consecutive frames of medical images marked with the feature point in the medical image video, then group the medical image video according to the positions of the multiple consecutive frames of medical images marked with the feature point in the medical image video, divide the multiple consecutive frames of medical images marked with the feature point into a group of image sequences, and set the group of image sequences as the medical image sequence corresponding to the feature point.

[0131] In some embodiments of the present invention, if there are no consecutive multiple frames of medical images marked with the feature points in the medical image video, it is determined whether the number of images of the medical image where the feature point is located is less than 2; if the number of images of the medical image where the feature point is located is less than 2, the medical image where the feature point is located in the medical image video and the adjacent frame medical images of the medical image where the feature point is located are divided into a group of image sequences, and this group of image sequences is set as the medical image sequence corresponding to the feature point; if the number of images of the medical image where the feature point is located is greater than or equal to 2, according to the position of the medical image where the feature point is located in the medical image video, the medical image with the smallest number of frames in the medical image where the feature point is located and the medical image with the largest number of frames in the medical image where the feature point is located are determined, and the medical images between the medical image with the smallest number of frames and the medical image with the largest number of frames in the medical image where the feature point is located in the medical image video are divided into a group of image sequences, and this group of image sequences is set as the medical image sequence corresponding to the feature point. Wherein, the adjacent frame medical images of the medical image where the feature point is located may be the previous frame medical image and the subsequent frame medical image of the medical image where the feature point is located.

[0132] In some embodiments of the present invention, if there are no consecutive multiple frames of medical images marked with the feature points in the medical image video, it is determined whether there are medical images marked with feature points among the adjacent multiple frames of medical images of the medical image where the feature point is located in the medical image video; if there are medical images marked with feature points among the adjacent multiple frames of medical images of the medical image where the feature point is located in the medical image video, the consecutive multiple frames of medical images between the medical images marked with feature points among the adjacent multiple frames of medical images of the medical image where the feature point is located and the medical image where the feature point is located are divided into a group of image sequences, and this group of image sequences is set as the medical image sequence corresponding to the feature point. Wherein, the adjacent multiple frames of medical images of the medical image where the feature point is located may be the previous multiple frames of medical images of the medical image where the feature point is located, or the subsequent multiple frames of medical images of the medical image where the feature point is located.

[0133] 503, Associate each feature point with the medical image sequence corresponding to the feature point to obtain a preset mapping relationship.

[0134] In some embodiments of the present invention, after determining the medical image sequence corresponding to each feature point of the medical image video, record the spatial coordinate information of each feature point, and associate each feature point of the medical image video with the medical image sequence corresponding to the feature point to obtain a preset mapping relationship.

[0135] In some embodiments of the present invention, after establishing the mapping relationship, the mapping relationship is stored, and according to the method for obtaining the target medical image video described above, the target position information input by the user based on the displayed 3D model view is obtained, and the target medical image video corresponding to the target position information is searched from the medical image video corresponding to the 3D model.

[0136] In some embodiments of the present invention, when the scanning information is a scanning point, after obtaining the target medical image video corresponding to the target position information, the target scanning point matching the target position information can be determined according to the error between the scanning point on each frame of the candidate target medical image in the target medical image video and the target position information. Specifically, the steps for determining the target scanning point matching the target position information include:

[0137] (1) Obtain the scanning points on each frame of the candidate target medical image in the target medical image video.

[0138] (2) According to the first error between the target coordinate information in the target position information and the coordinates of each scanning point, select the scanning points among each scanning point whose second error with the target position information is less than or equal to the preset second error threshold, and set the selected scanning points as the target scanning information matching the target position information.

[0139] In some embodiments of the present invention, the second error between the target position information and each scanning point can be obtained by calculating the coordinate distance between the spatial coordinates of the target position information and the spatial coordinates of each scanning point. Among them, the coordinate distance between the spatial coordinates of the target position information and the spatial coordinates of each scanning point can be obtained by Euclidean distance, Mahalanobis distance, L1 norm or L2 norm to obtain the second error between the target position information and each scanning point.

[0140] In some embodiments of the present invention, the first error between the target position information and each scanning point can be respectively compared with the preset second error threshold, and the scanning points among each scanning point whose first error with the target position information is less than or equal to the preset second error threshold are selected, and the selected scanning points are set as the target scanning information matching the target position information.

[0141] In some embodiments of the present invention, when the scanning information is a scanning point, after obtaining the target scanning information, that is, after obtaining the target scanning point, the preset mapping relationship is obtained, and according to this mapping relationship, the target medical image corresponding to the target scanning point is selected from the target medical image video, and the target medical image is displayed.

[0142] In some embodiments of the present invention, when the scanning information is a scanning point, after obtaining the target scanning information, that is, after obtaining the target scanning point, according to step 203, the scanning points corresponding to each frame of medical image in the target medical image video can be obtained according to a preset mapping relationship, and the scanning points corresponding to each frame of medical image in the target medical image video are compared with the target scanning point, and the medical images in the target medical image video whose scanning points are consistent with the target scanning point are selected, and the medical images whose selected scanning points are consistent with the target scanning point are set as the target medical images, and the target medical images are displayed.

[0143] In some embodiments of the present invention, when the scanning information is a feature point, after obtaining the target medical image video corresponding to the target position information, the error between the feature points in the target medical image video and the target position information can be determined, and the target feature point matching the target position information can be determined. Specifically, the steps of determining the target feature point matching the target position information include:

[0144] (1) Obtain the feature points in the target medical image video.

[0145] (2) According to the first error between the target coordinate information in the target position information and the coordinate information of each feature point, determine the target feature points among each feature point whose first error with the target position information is less than or equal to a preset first error threshold, and set the selected target feature points as the target scanning information matching the target position information.

[0146] In some embodiments of the present invention, the spatial coordinates of the feature points in the target medical image video are obtained, and the first error between the target position information and each feature point is obtained through the coordinate distance between the spatial coordinates of the target position information and each feature point by using the Euclidean distance, Mahalanobis distance, L1 norm or L2 norm.

[0147] In some embodiments of the present invention, the first error between the target position information and each feature point can be respectively compared with a preset second error threshold, and the scanning points whose second error between the target position information and each feature point is less than or equal to the preset first error threshold are selected, and the selected feature points are set as the target scanning information matching the target position information.

[0148] In some embodiments of the present invention, when the scanned information is feature points, after obtaining the target scanned information, step 203 can be followed to obtain the feature points of the medical image in the target medical image video according to a preset mapping relationship, compare the feature points of the medical image in the target medical image video with the target scanned information, select the medical image in the target medical image video whose feature points are consistent with the target scanned information, set the selected medical image whose feature points are consistent with the target scanned information as the target medical image, and display the target medical image. Among them, the medical image whose feature points are consistent with the target scanned information refers to the medical image in the target medical image video with the smallest distance between the spatial coordinates of the feature points of the medical image and the spatial coordinates of the target feature points in the target scanned information.

[0149] In some embodiments of the present invention, after displaying the target medical image, for the convenience of the user to view, the user can input query information based on the view of the displayed target medical image, select the medical image matching the query information from the target medical image video according to the query information, and display the medical image matching the query information. Specifically, the method for querying and displaying medical images includes:

[0150] (1) Respond to the target part information or target lesion information input by the user based on the view of the displayed target medical image, and select the target marked medical image corresponding to the target part information or target lesion information from the pre-stored marked medical image video. Among them, the marked medical image video is obtained by identifying the target medical image video through a preset part recognition model or a preset lesion recognition model.

[0151] (2) Display the target marked medical image.

[0152] In some embodiments of the present invention, after obtaining the target medical imaging video, the location of each frame of medical imaging in the target medical imaging video can be identified according to a preset location recognition model, the location information on each frame of medical imaging in the target medical imaging video can be determined, and each frame of medical imaging can be marked according to the location information on each frame of medical imaging in the target medical imaging video to obtain marked medical imaging. The marked medical imaging is sorted according to the frame order of the target medical imaging video to obtain a marked medical imaging video. In some embodiments of the present invention, marking each frame of medical imaging according to the location information on each frame of medical imaging in the target medical imaging video includes: querying the pre-stored location identification data according to the location information on each frame of medical imaging in the target medical imaging video to obtain the location identification corresponding to the location information on each frame of medical imaging in the target medical imaging video, and marking each frame of medical imaging according to the location identification. Among them, the location identification data stores multiple location information and the location identification corresponding to each location information. The location identification is unique and can be one or more of numbers, letters, characters, and words.

[0153] In some embodiments of the present invention, in the process of the location recognition model identifying the location of each frame of medical imaging in the target medical imaging video, in order to improve the recognition efficiency, the target medical imaging video can be divided into at least one group of frames to be recognized according to the number of frames of the target medical imaging video, and each group of frames to be recognized is identified by the preset location recognition model to determine the location information on each frame of medical imaging in the group of frames to be recognized. In some embodiments of the present invention, the division can be equal division. For example, when the number of frames of the target medical imaging video is 32, starting from the first frame of medical imaging of the target medical imaging video, every 4 frames of medical imaging are divided once to form a group of frames to be recognized, and 8 groups of frames to be recognized are obtained. Each group of frames to be recognized includes 4 consecutive frames of medical imaging.

[0154] In the same way as the method of identifying locations through the location recognition model, the target medical imaging video can also be identified by a preset lesion recognition model to obtain the lesion information in the target medical imaging video and the medical imaging where the lesion information is located. The pre-stored lesion identification data is queried according to the lesion information in the target medical imaging video to obtain the lesion identification corresponding to the lesion information in the target medical imaging video, and the medical imaging where the lesion information is located in the target medical imaging video is marked according to the lesion identification. Among them, the lesion information refers to the lesion area. The lesion identification is unique and can be one or more of numbers, letters, characters, and words.

[0155] Similar to the way of part recognition through the part recognition model, during the recognition of the target medical image video by the preset lesion recognition model, at least one group of frames to be recognized can also be divided according to the number of frames of the target medical image video. The lesion recognition model is used to perform lesion recognition on each group of frames to be recognized, so as to obtain the lesion information in the target medical image video and the medical image where the lesion information is located.

[0156] Among them, the part recognition model and the lesion recognition model can be recognition models based on machine learning, such as recognition models based on logistic regression, recognition models based on random forests, recognition models based on dictionaries, etc. The part recognition model and the lesion recognition model can also be recognition models based on neural networks, such as Convolutional Neural Networks (CNN), De-Convolutional Networks (DN), Deep Neural Networks (DNN), Deep Convolutional Inverse Graphics Networks (DCIGN), Region-based Convolutional Networks (RCNN), Faster Region-based Convolutional Networks (Faster RCNN), and Bidirectional Encoder Representations from Transformers (BERT) models.

[0157] In some embodiments of the present invention, after collecting the medical image video of the object, the part information or lesion information of the medical image video can be obtained according to the above-mentioned part recognition method or lesion recognition method, and the medical image video can be marked according to the part information or lesion information of the medical image video to obtain a marked medical image video, which is then stored.

[0158] In some embodiments of the present invention, after responding to the target part information or target lesion information input by the user based on the view of the displayed target medical image, the target part identifier or target lesion identifier is obtained. According to the target part identifier, the marked medical images with the same part identifier as the target part identifier are selected from the marked medical image video, or according to the target lesion identifier, the marked medical images with the same lesion identifier as the target lesion identifier are selected from the marked medical image video. The selected marked medical images are set as the target marked medical images corresponding to the target part information or target lesion information, and the currently displayed target medical image is switched to the target marked medical image.

[0159] The medical image display method provided by the embodiments of the present invention associates the 3D model of the target part with the video frames corresponding to the target part in multiple medical images through a mapping relationship. When the user views the 3D model through operations such as dragging and clicking, the video frames corresponding to the area where the user operates on the 3D model of the target part in multiple medical images are provided to the user at the same time, so that the video images of the area to be observed can be quickly retrieved, the display efficiency of multiple medical image videos is improved, and the time cost is reduced.

[0160] To better implement the medical image display method provided by the embodiments of the present invention, on the basis of the medical image display method, a medical image display device is provided, as Figure 6 shown Figure 6 is a schematic structural diagram of the medical image display device provided by the embodiments of the present invention. The shown medical image display device includes:

[0161] An interaction module 601, configured to obtain target position information input by the user based on the displayed 3D model view and the target medical image video corresponding to the target position information;

[0162] An acquisition module 602, configured to determine target scan information matching the target position information;

[0163] A display module 603, configured to obtain a target medical image corresponding to the target scan information in the target medical image video according to a preset mapping relationship, and display the target medical image; the mapping relationship is obtained by acquiring the scan information of the medical image video and associating and matching according to the scan information and the medical image video.

[0164] In some embodiments of the present invention, the medical image display device includes an association module 603; the association module 603 includes:

[0165] An identification unit, configured to identify the acquired medical image video through a preset identification model to obtain the identification information of the medical image video and the medical image where the identification information is located;

[0166] A scan path unit, configured to obtain the scan path of the medical image video according to the medical image where the identification information is located;

[0167] An association unit, configured to determine the medical image corresponding to each scan point on the scan path of the medical image video, and associate each scan point on the scan path of the medical image video with the medical image corresponding to the scan point to obtain a preset mapping relationship.

[0168] In some embodiments of the present invention, the scan path unit is configured to:

[0169] Obtain the acquisition order of the identification information of the medical image video according to the frame order of the medical images in the medical image video and the medical images corresponding to the identification information;

[0170] Obtain the scanning order of the scanning points of the medical image video according to the acquisition order of the identification information of the medical image video; wherein, one identification information of the medical image video corresponds to at least one scanning point;

[0171] Obtain the scanning path of the medical image video according to the scanning order of the scanning points of the medical image video.

[0172] In some embodiments of the present invention, the association module 603 includes:

[0173] A coordinate acquisition unit, configured to acquire a sequence of spatial coordinates of the scanning device during the acquisition of the medical image video; the scanning device is used to acquire the medical image video;

[0174] A scanning path unit, configured to obtain the device scanning path of the scanning device according to the sequence of spatial coordinates; each device scanning point on the device scanning path of the scanning device corresponds to at least one frame of medical image in the medical image video;

[0175] A scanning point unit, configured to compare each device scanning point on the device scanning path with each preset scanning point on the preset scanning path, and obtain a target preset scanning point on the preset scanning path that matches each device scanning point on the device scanning path;

[0176] An association unit, configured to associate the medical image corresponding to each device scanning point with the target preset scanning point that matches the device scanning point for each device scanning point on the device scanning path, and obtain a preset mapping relationship.

[0177] In some embodiments of the present invention, the association module 603 includes:

[0178] A feature point detection unit, configured to detect feature points of the acquired medical image video through a preset feature detection model, and obtain the feature points in the medical image video and the medical image where each feature point is located;

[0179] An image sequence determination unit, configured to segment the medical image where each feature point is located, and obtain a medical image sequence corresponding to each feature point;

[0180] An association unit, configured to associate each feature point with the medical image sequence corresponding to the feature point, and obtain a preset mapping relationship.

[0181] In some embodiments of the present invention, the acquisition module 602 is configured to:

[0182] Determine the feature points in the target medical imaging video;

[0183] According to the first error between the target coordinate information in the target position information and the coordinate information of each feature point, determine the target feature points whose first error is less than or equal to the preset first error threshold, and set the target feature points as the target scanning information matching the target position information.

[0184] In some embodiments of the present invention, the acquisition module 602 is used for:

[0185] Acquire the scanning points on each frame of candidate target medical imaging in the target medical imaging video;

[0186] According to the second error between the target coordinate information in the target position information and the coordinate information of each scanning point, select the scanning points among all scanning points whose second error with the target position information is less than or equal to the preset second error threshold, and set the selected scanning points as the target scanning information matching the target position information.

[0187] In some embodiments of the present invention, the interaction module 601 is used for:

[0188] Acquire the target position information input by the user based on the displayed 3D model view;

[0189] Select at least one frame of medical imaging that matches the target coordinate information in the target position information from the pre-stored medical imaging video, and set the selected at least one frame of medical imaging and a preset number of consecutive frames of medical imaging adjacent to the at least one frame of medical imaging as the first candidate medical imaging video corresponding to the target coordinate information;

[0190] According to the target view information in the target position information, obtain the target area information corresponding to the 3D model in the 3D model view, select at least one frame of medical imaging with the target area information from the pre-stored medical imaging video, and set the selected at least one frame of medical imaging and a preset number of consecutive frames of medical imaging adjacent to the at least one frame of medical imaging as the second candidate medical imaging video corresponding to the target view information;

[0191] Determine the target medical imaging video corresponding to the target position information according to the first candidate medical imaging video and the second candidate medical imaging video.

[0192] In some embodiments of the present invention, the interaction module 601 is used for:

[0193] Acquire the target view information and target coordinate information in the target position information input by the user based on the displayed 3D model view;

[0194] Obtain a second candidate medical image video corresponding to the perspective that matches the target perspective information from the pre-stored medical image video according to the target angle information;

[0195] Select at least one frame of medical image that matches the target coordinate information from the second candidate medical image video, and a preset number of consecutive frames of medical images adjacent to the at least one frame of medical image;

[0196] Set the at least one frame of medical image selected and the preset number of consecutive frames of medical images adjacent to the at least one frame of medical image as the target medical image video corresponding to the target position information.

[0197] In some embodiments of the present invention, the interaction module 601 is configured to, in response to the target part information or target lesion information input by the user based on the view of the displayed target medical image, select a target marked medical image corresponding to the target part information or target lesion information from the pre-stored marked medical image video; the marked medical image video is obtained by identifying the target medical image video through a preset part recognition model or a preset lesion recognition model;

[0198] The display module 603 is configured to display the target marked medical image.

[0199] The medical image display device provided by the embodiments of the present invention associates the scan information with the video frames corresponding to the scan information through a mapping relationship. When the user inputs target position information through operations such as dragging and clicking, at the same time, the video frames that match the target scan information corresponding to the target position information input by the user in multiple medical image videos are provided to the user, so that the video images of the area to be observed can be quickly retrieved, the display efficiency of multiple medical image videos is improved, and the time cost is reduced.

[0200] The embodiments of the present invention also provide an electronic device, as Figure 7 shown, which shows the structural schematic diagram of the electronic device involved in the embodiments of the present invention. Specifically:

[0201] The electronic device may include a processor 701 with one or more processing cores, a memory 702 with one or more computer-readable storage media, a power supply 703, an input unit 704 and other components. Those skilled in the art can understand that Figure 7 the structure of the electronic device shown in

[0202] The processor 701 is the control center of the electronic device, connecting various parts of the entire electronic device through various interfaces and circuits. By running or executing software programs and / or modules stored in the memory 702, and by calling the data stored in the memory 702, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 701 may include one or more processing cores; preferably, the processor 701 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 701 either.

[0203] The memory 702 can be used to store software programs and modules. The processor 701 executes various functional applications and data processing by running the software programs and modules stored in the memory 702. The memory 702 mainly includes a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as the sound playback function, image playback function, etc.); the data storage area can store the data created according to the use of the electronic device. In addition, the memory 702 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices. Correspondingly, the memory 702 may also include a memory controller to provide the processor 701 with access to the memory 702.

[0204] The electronic device further includes a power supply 703 for supplying power to each component. Preferably, the power supply 703 can be logically connected to the processor 701 through a power management system, thereby implementing functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 703 may also include any components such as one or more DC or AC power supplies, a recharge system, a power failure detection circuit, a power converter or inverter, and a power status indicator.

[0205] The electronic device may further include an input unit 704, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0206] Although not shown, the electronic device may further include a display unit, etc., which will not be elaborated here. Specifically, in this embodiment, the processor 701 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 702 according to the following instructions, and the processor 701 will run the application programs stored in the memory 702 to implement various functions as follows:

[0207] Obtain the target position information input by the user based on the displayed 3D model view and the target medical image video corresponding to the target position information;

[0208] Determine the target scan information that matches the target position information;

[0209] According to the preset mapping relationship, obtain the target medical image corresponding to the target scan information in the target medical image video and display the target medical image; the mapping relationship is obtained by acquiring the scan information of the medical image video and associating and matching according to the scan information and the medical image video.

[0210] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions. The instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0211] For this reason, an embodiment of the present invention provides a storage medium in which multiple instructions are stored. The instructions can be loaded by a processor to execute the steps in any one of the medical image display methods provided by the embodiments of the present invention. For example, the instructions can execute the following steps:

[0212] Obtain the target position information input by the user based on the displayed 3D model view and the target medical image video corresponding to the target position information;

[0213] Determine the target scan information that matches the target position information;

[0214] According to the preset mapping relationship, obtain the target medical image corresponding to the target scan information in the target medical image video and display the target medical image; the mapping relationship is obtained by acquiring the scan information of the medical image video and associating and matching according to the scan information and the medical image video.

[0215] For the specific implementation of each of the above operations, reference can be made to the previous embodiments and will not be elaborated here.

[0216] Among them, the storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc.

[0217] Since the instructions stored in the storage medium can execute the steps in any one of the medical image display methods provided by the embodiments of the present invention, the beneficial effects that can be achieved by any one of the medical image display methods provided by the embodiments of the present invention can be realized. For details, reference can be made to the previous embodiments and will not be elaborated here.

[0218] The above has introduced in detail a medical image display method, device, electronic device, and storage medium provided by the embodiments of the present invention. Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A medical image display method, characterized in that, The method includes: Obtaining target position information input by a user based on a displayed 3D model view and a target medical imaging video corresponding to the target position information; wherein, the target position information includes target coordinate information and viewing angle information; Determining target scan information matching the target position information; Obtaining, according to a preset mapping relationship, a target medical image corresponding to the target scan information in the target medical imaging video, and displaying the target medical image; the mapping relationship is obtained by acquiring scan information of a medical imaging video and associatively matching the scan information with the medical imaging video; Wherein, the determining target scan information matching the target position information includes: Determining feature points in the target medical imaging video; determining, according to a first error between the target coordinate information in the target position information and the coordinate information of each of the feature points, target feature points for which the first error is less than or equal to a preset first error threshold, and setting the target feature points as target scan information matching the target position information; and / or, Obtaining scan points on each frame of a candidate target medical image in the target medical imaging video; selecting, according to a second error between the target coordinate information in the target position information and the coordinate information of each of the scan points, scan points for which the second error with the target position information is less than or equal to a preset second error threshold from among all the scan points, and setting the selected scan points as target scan information matching the target position information.

2. The medical image display method according to claim 1, characterized in that, Before the step of obtaining, according to a preset mapping relationship, a target medical image corresponding to the target scan information in the target medical imaging video, the method includes: Identifying an acquired medical imaging video through a preset identification model to obtain identification information of the medical imaging video and the medical image where the identification information is located; Obtaining a scan path of the medical imaging video according to the medical image where the identification information is located; Determining medical images corresponding to each scan point on the scan path of the medical imaging video, and associating each scan point on the scan path of the medical imaging video with the medical image corresponding to the scan point to obtain a preset mapping relationship.

3. The medical image display method according to claim 2, characterized in that, The obtaining a scan path of the medical imaging video according to the medical image where the identification information is located includes: Obtaining an acquisition sequence of the identification information of the medical imaging video according to the frame sequence of medical images in the medical imaging video and the medical image corresponding to the identification information; Obtaining a scan sequence of scan points of the medical imaging video according to the acquisition sequence of the identification information of the medical imaging video; wherein, one piece of identification information of the medical imaging video corresponds to at least one scan point; Obtaining a scan path of the medical imaging video according to the scan sequence of scan points of the medical imaging video.

4. The medical image display method according to claim 1, wherein Before the step of obtaining, according to a preset mapping relationship, a target medical image corresponding to the target scan information in the target medical imaging video, the method includes: Obtaining a sequence of spatial coordinates during the process of a scanning device acquiring a medical imaging video; the scanning device is used for acquiring a medical imaging video; Based on the spatial coordinate sequence, obtain the device scanning path of the scanning device; each device scanning point on the device scanning path of the scanning device corresponds to at least one frame of medical image in the medical image video; Compare each device scanning point on the device scanning path with each preset scanning point on the preset scanning path to obtain the target preset scanning points on the preset scanning path that match the device scanning points on the device scanning path; For each device scanning point on the device scanning path, associate the medical image corresponding to this device scanning point with the target preset scanning point that matches this device scanning point to obtain a preset mapping relationship.

5. The medical image display method according to claim 1, wherein Before the step of obtaining the target medical image corresponding to the target scanning information in the target medical image video according to the preset mapping relationship, the method includes: Detect feature points of the obtained medical image video through a preset feature detection model to obtain the feature points in the medical image video and the medical image where each feature point is located; Segment the medical image where each feature point is located to obtain a medical image sequence corresponding to each feature point; Associate each feature point with the medical image sequence corresponding to this feature point to obtain a preset mapping relationship.

6. The medical image display method according to claim 1, characterized in that The obtaining of the target position information input by the user based on the displayed 3D model view and the target medical image video corresponding to the target position information includes: Obtain the target position information input by the user based on the displayed 3D model view; Select at least one frame of medical image from the pre-stored medical image videos that matches the target coordinate information in the target position information, and set the selected at least one frame of medical image and a preset number of consecutive frames of medical images adjacent to the at least one frame of medical image as the first candidate medical image video corresponding to the target coordinate information; Obtain the target area information corresponding to the 3D model in the 3D model view according to the target viewing angle information in the target position information, select at least one frame of medical image from the pre-stored medical image videos where the target area information exists, and set the selected at least one frame of medical image and a preset number of consecutive frames of medical images adjacent to the at least one frame of medical image as the second candidate medical image video corresponding to the target viewing angle information; Determine the target medical image video corresponding to the target position information according to the first candidate medical image video and the second candidate medical image video.

7. The medical image display method according to claim 1, wherein The obtaining of the target position information input by the user based on the displayed 3D model view and the target medical image video corresponding to the target position information includes: Obtain the target viewing angle information and target coordinate information in the target position information input by the user based on the displayed 3D model view; Obtain the second candidate medical image video that matches the target viewing angle information from the pre-stored medical image videos; Select at least one frame of medical image from the second candidate medical image video that matches the target coordinate information, and a preset number of consecutive frames of medical images adjacent to the at least one frame of medical image; Set the selected at least one medical image and a preset number of consecutive medical images adjacent to the at least one medical image as the target medical image video corresponding to the target position information.

8. The medical image display method according to any one of claims 1 to 7, characterized in that After obtaining, according to a preset mapping relationship, a target medical image corresponding to the target scan information in the target medical image video and displaying the target medical image, the method includes: In response to the target part information or target lesion information input by the user based on the view of the displayed target medical image, select a target labeled medical image corresponding to the target part information or the target lesion information from a pre-stored labeled medical image video; the labeled medical image video is obtained by identifying the target medical image video through a preset part recognition model or a preset lesion recognition model; Display the target labeled medical image.

9. A medical image display device, characterized in that, The device includes: An interaction module, configured to obtain target position information input by the user based on a displayed 3D model view and a target medical image video corresponding to the target position information; wherein the target position information includes target coordinate information and viewing angle information; An acquisition module, configured to determine target scan information matching the target position information; A display module, configured to obtain, according to a preset mapping relationship, a target medical image corresponding to the target scan information in the target medical image video and display the target medical image; the mapping relationship is obtained by acquiring scan information of a medical image video and associatively matching according to the scan information and the medical image video; Wherein, the acquisition module is further configured to: Determine feature points in the target medical image video; determine, according to a first error between the target coordinate information in the target position information and the coordinate information of each of the feature points, target feature points for which the first error is less than or equal to a preset first error threshold, and set the target feature points as the target scan information matching the target position information; and / or, Acquire scan points on each frame of candidate target medical image in the target medical image video; select, according to a second error between the target coordinate information in the target position information and the coordinate information of each of the scan points, scan points for which the second error with the target position information is less than or equal to a preset second error threshold from each of the scan points, and set the selected scan points as the target scan information matching the target position information.

10. An electronic device, characterized in that, It includes a memory and a processor; the memory stores an application program, and the processor is configured to run the application program in the memory to perform the operations in the medical image display method according to any one of claims 1 to 7.

11. A storage medium, characterized in that, The storage medium stores multiple instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the medical image display method according to any one of claims 1 to 7.

Images