Image annotation method, device, equipment, medium and computer program product

Abstract

The present invention relates to the technical field of image processing. The present invention provides an image annotation method, apparatus, device, medium and computer program product. The method includes: determining the spatial pose information of the virtual capsule endoscope based on the positional relationship between the virtual capsule endoscope and the gastric cavity model; the gastric cavity model is created based on historical gastric cavity data; establishing an association relationship between the virtual gastric cavity image and the spatial pose information; comparing the real gastric cavity image with the virtual gastric cavity image, and determining the multi-dimensional spatial data of the real gastric cavity image based on the comparison result and the association relationship; annotating the real gastric cavity image based on the multi-dimensional spatial data to obtain multi-dimensional image annotation information. By annotating multi-dimensional spatial data for gastric cavity images, the present invention reduces the workload of personnel, improves the annotation efficiency of gastric cavity image data, and annotates more comprehensive and accurate data for gastric cavity images, which is beneficial to the accurate construction of gastric cavity analysis-related algorithms.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to an image annotation method, apparatus, device, medium, and computer program product. Background Art

[0002] In the medical field, image analysis is widely used in the diagnosis of gastrointestinal diseases. A capsule endoscope is an available tool for obtaining images of the inside of the gastrointestinal tract. However, the movement of the capsule endoscope inside the gastrointestinal tract is uncontrollable, and the spatio-temporal correlation between the obtained images is not strong, which makes it particularly difficult to annotate data for gastric cavity images. The construction and improvement of various algorithms for gastric cavity analysis (such as the gastric cavity mucosal coverage rate evaluation algorithm) rely on gastric cavity images with accurate and comprehensive data annotation. Therefore, how to annotate more comprehensive and accurate data for gastric cavity images, and how to improve the data annotation efficiency of gastric cavity images have become technical problems to be solved urgently. Summary of the Invention

[0003] The present invention provides an image annotation method, apparatus, device, medium, and computer program product to solve the defects of inaccurate and incomplete annotation data and low annotation efficiency in the existing gastric cavity image data annotation technology, and to achieve comprehensive, accurate, and efficient gastric cavity image data annotation.

[0004] The present invention provides an image annotation method, including the following steps.

[0005] Based on the positional relationship between a virtual capsule endoscope and a gastric cavity model, determine the spatial pose information of the virtual capsule endoscope; the gastric cavity model is created based on historical gastric cavity data;

[0006] Establish an association relationship between a virtual gastric cavity image and the spatial pose information;

[0007] Compare a real gastric cavity image with the virtual gastric cavity image, and based on the comparison result and the association relationship, determine the multi-dimensional spatial data of the real gastric cavity image;

[0008] Annotate the real gastric cavity image based on the multi-dimensional spatial data to obtain multi-dimensional image annotation information.

[0009] According to the image annotation method provided by the present invention, the determining the spatial pose information of the virtual capsule endoscope based on the positional relationship between the virtual capsule endoscope and the gastric cavity model includes:

[0010] Based on the size information of a real capsule endoscope, construct a virtual capsule endoscope in the gastric cavity model; the virtual capsule endoscope is provided with shooting parameters;

[0011] Determine the global coordinate system of the gastric cavity and the local coordinate system of the capsule. During the rotation of the virtual capsule endoscope, determine the change information of the virtual capsule endoscope in the global coordinate system of the gastric cavity;

[0012] Based on the change information, convert the capsule local coordinates into the global coordinates in the global coordinate system of the gastric cavity;

[0013] Based on the global coordinates and the shooting parameters, determine the spatial pose information of the virtual capsule endoscope.

[0014] According to an image annotation method provided by the present invention, the establishment of the association relationship between the virtual gastric cavity image and the spatial pose information includes:

[0015] Perform regional analysis on the virtual gastric cavity image, and determine the proportion of each virtual gastric cavity part in the virtual gastric cavity image based on the analysis results;

[0016] Associate the proportion of each virtual gastric cavity part with the spatial pose information to obtain the association relationship.

[0017] According to an image annotation method provided by the present invention, the comparison of the real gastric cavity image with the virtual gastric cavity image, and the determination of the spatial multi-dimensional data of the real gastric cavity image based on the comparison result and the association relationship include:

[0018] Determine the proportion of each real gastric cavity part in the real gastric cavity image;

[0019] Compare the proportion of each real gastric cavity part in the real gastric cavity image with the proportion of each virtual gastric cavity part in the virtual gastric cavity image;

[0020] Based on the comparison result, determine the target gastric cavity image; the target gastric cavity image is a virtual gastric cavity image that matches the real gastric cavity image;

[0021] Based on the spatial pose information associated with the target gastric cavity image, determine the spatial multi-dimensional data of the real gastric cavity image.

[0022] According to an image annotation method provided by the present invention, the spatial pose information includes the capsule global coordinates and the capsule shooting information; the determination of the spatial multi-dimensional data of the real gastric cavity image based on the spatial pose information associated with the target gastric cavity image includes:

[0023] In the case where the real gastric cavity image is a plurality of consecutive real images in time, generate a capsule motion trajectory based on the capsule global coordinates corresponding to each real image;

[0024] Based on the capsule global coordinates, the capsule shooting information, and the capsule motion trajectory, determine the spatial multi-dimensional data of the real gastric cavity image;

[0025] When the real gastric cavity image is a single real image, based on the global coordinates of the capsule corresponding to the real image and the capsule shooting information, determine the spatial multi-dimensional data of the real gastric cavity image.

[0026] According to an image annotation method provided by the present invention, after annotating the real gastric cavity image based on the spatial multi-dimensional data to obtain multi-dimensional image annotation information, it includes:

[0027] Receive an output instruction containing coordinate information;

[0028] Based on the coordinate information, determine the spatial multi-dimensional data corresponding to the output instruction;

[0029] Visually display the spatial coordinates, shooting angles, and movement trajectories included in the spatial multi-dimensional data corresponding to the output instruction.

[0030] The present invention also provides an image annotation device, including the following modules:

[0031] A spatial pose information determination module, configured to determine the spatial pose information of the virtual capsule endoscope based on the positional relationship between the virtual capsule endoscope and the gastric cavity model; the gastric cavity model is created based on historical gastric cavity data;

[0032] An association relationship establishment module, configured to establish an association relationship between the virtual gastric cavity image and the spatial pose information;

[0033] A spatial multi-dimensional data determination module, configured to compare the real gastric cavity image with the virtual gastric cavity image, and based on the comparison result and the association relationship, determine the spatial multi-dimensional data of the real gastric cavity image;

[0034] An image annotation module, configured to annotate the real gastric cavity image based on the spatial multi-dimensional data to obtain multi-dimensional image annotation information.

[0035] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the image annotation method as described in any one of the above.

[0036] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the image annotation method as described in any one of the above.

[0037] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the image annotation method as described in any one of the above.

[0038] The image annotation method, device, equipment, medium and computer program product provided by the present invention determine the spatial pose information of the virtual capsule endoscope through the pre-constructed positional relationship between the virtual capsule endoscope and the gastric cavity model, then associate the virtual gastric cavity image corresponding to the virtual capsule endoscope with the spatial pose information, compare the real gastric cavity image with the virtual gastric cavity image to be marked, and determine the multi-dimensional spatial data for annotating the real gastric cavity image according to the association relationship and the comparison result. Finally, the real gastric cavity image is annotated with the multi-dimensional spatial data. By annotating the gastric cavity image with multi-dimensional spatial data, the present invention reduces the workload of relevant technicians, improves the annotation efficiency of gastric cavity image data, and annotates more comprehensive and accurate data for the gastric cavity image, which is beneficial to the accurate construction of relevant algorithms for gastric cavity analysis. Brief Description of the Drawings

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0040] Figure 1 is one of the schematic flowcharts of the image annotation method provided by the present invention.

[0041] Figure 2 is the second schematic flowchart of the image annotation method provided by the present invention.

[0042] Figure 3 is the schematic structural diagram of the image annotation device provided by the present invention.

[0043] Figure 4 is the schematic structural diagram of the electronic device provided by the present invention. Detailed Embodiments

[0044] To make the objectives, technical solutions and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention with reference to the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments in the present invention belong to the scope of protection of the present invention.

[0045] The following will describe Figures 1 - 4 the image annotation method, device, equipment, medium and computer program product of the present invention.

[0046] Figure 1This is one of the schematic flowcharts of the image annotation method provided by the present invention. As Figure 1 shown, the method includes the following:

[0047] Step 100: Determine the spatial pose information of the virtual capsule endoscope based on the positional relationship between the virtual capsule endoscope and the gastric cavity model; the gastric cavity model is created based on historical gastric cavity data;

[0048] Specifically, the implementation of the image annotation method provided by the present invention mainly includes the following steps.

[0049] Step 1: Construction of a three-dimensional gastric cavity model.

[0050] During the construction process of the three-dimensional gastric cavity model, it mainly includes processes such as data acquisition, data processing, model optimization, hierarchical modeling, and model verification.

[0051] During the data acquisition process, gastric cavity data of a person at different degrees of fullness (related to the amount of water drunk) is obtained through a scanner (computer tomography CT or magnetic resonance imaging MRI). These gastric cavity data serve as the basis for the construction process of the three-dimensional gastric cavity model to ensure a high degree of consistency between the three-dimensional gastric cavity model and the internal structure of the real gastric cavity.

[0052] During the data processing process, by loading and analyzing the gastric cavity data, the internal contour and structure of the gastric cavity are extracted to generate a preliminary gastric cavity model; during the model optimization process, the preliminary generated gastric cavity model is optimized (such as smoothing processing and simplification processing) to optimize the surface details of the gastric cavity. The higher the level of detail of the three-dimensional model, the higher its complexity. During the hierarchical modeling process, three-dimensional models with different levels of detail are created to meet different requirements.

[0053] Step 2: Model format and compatibility. This step mainly ensures the compatibility of the gastric cavity model in different development environments.

[0054] Step 3: Loading and display of the gastric cavity model. This step is used to load and render the gastric cavity model to ensure the rendering performance in complex scenarios and to improve the interaction process between the gastric cavity model and the user.

[0055] After the construction of the gastric cavity model is completed, on the basis of the gastric cavity model, a capsule endoscope model, that is, the virtual capsule endoscope in this embodiment, is created. And relevant parameters related to the real capsule endoscope are set for the virtual capsule endoscope, such as the field of view angle, pose, and movement mode. By defining the global coordinate system of the gastric cavity model and the local coordinate system of the virtual capsule endoscope, through matrix transformation and position transformation and other processes, the coordinates of the virtual capsule endoscope are transformed into the coordinates of the gastric cavity model, and then the spatial pose information of the virtual capsule endoscope, such as coordinates and spatial attitude, is represented by the global coordinate system. The spatial pose information also includes the above-mentioned relevant parameters.

[0056] Step 200: Establish an association relationship between the virtual gastric cavity image and the spatial pose information;

[0057] Specifically, the virtual gastric cavity image in this embodiment refers to the gastric cavity image corresponding to a certain position, pose, and viewing angle of the virtual capsule endoscope. Associating the virtual gastric cavity image with the spatial pose information enables each virtual gastric cavity image to have its corresponding spatial pose information. It can be known that the virtual gastric cavity image includes each gastric cavity division region and its corresponding proportion. For example, the gastric cavity can be divided into regions 1 to 6. A certain virtual gastric cavity image includes region 1 with a proportion of 80% and region 2 with a proportion of 20%.

[0058] Step 300: Compare the real gastric cavity image with the virtual gastric cavity image, and based on the comparison result and the association relationship, determine the spatial multi-dimensional data of the real gastric cavity image;

[0059] Specifically, through a joint model that combines a convolutional neural network (CNN) and an edge detection algorithm, the real gastric cavity image obtained by the real capsule endoscope can be automatically classified by part and its edges detected, and the gastric cavity regions included in the real gastric cavity image and their corresponding proportions can be obtained. Through the design and training of the joint loss function, the joint model can achieve good performance in both the gastric cavity region classification task and the gastric cavity region edge detection task. Compare the real gastric cavity image with each virtual gastric cavity image to determine a certain virtual gastric cavity image that is most similar to the real gastric cavity image among the virtual gastric cavity images. Then, according to the spatial pose information associated with this virtual gastric cavity image, determine the spatial multi-dimensional data of the real gastric cavity image, that is, use the spatial pose information associated with this virtual gastric cavity image as the spatial multi-dimensional data of the real gastric cavity image.

[0060] Step 400: Label the real gastric cavity image based on the spatial multi-dimensional data to obtain multi-dimensional image annotation information.

[0061] Specifically, use the above-obtained spatial multi-dimensional data to label the real gastric cavity image so that the labeled real gastric cavity image can be used for the construction and improvement of various algorithms for gastric cavity analysis, which helps to improve the accuracy of relevant gastric cavity analysis algorithms.

[0062] In this embodiment, the spatial pose information of the virtual capsule endoscope is determined through the pre-established positional relationship between the virtual capsule endoscope and the gastric cavity model. Then, the virtual gastric cavity image corresponding to the virtual capsule endoscope is associated with the spatial pose information, and the real gastric cavity image is compared with the virtual gastric cavity image to be marked. According to the association relationship and the comparison result, the multi-dimensional spatial data to be marked for the real gastric cavity image is determined. Finally, the real gastric cavity image is marked with the multi-dimensional spatial data. By marking multi-dimensional spatial data for the gastric cavity image, the workload of relevant technicians is reduced, the efficiency of gastric cavity image data marking is improved, and more comprehensive and accurate data is marked for the gastric cavity image, which is beneficial to the accurate construction of gastric cavity analysis-related algorithms.

[0063] Figure 2 is the second flow chart of the image marking method provided by the present invention. As Figure 2 shown, the method may further include:

[0064] Step 110: Based on the size information of the real capsule endoscope, construct a virtual capsule endoscope in the gastric cavity model; the virtual capsule endoscope is provided with shooting parameters;

[0065] Step 120: Determine the global coordinate system of the gastric cavity and the local coordinate system of the capsule. During the rotation of the virtual capsule endoscope, determine the change information of the virtual capsule endoscope in the global coordinate system of the gastric cavity;

[0066] Step 130: Based on the change information, convert the local coordinates of the capsule into the global coordinates in the global coordinate system of the gastric cavity;

[0067] Step 140: Based on the global coordinates and the shooting parameters, determine the spatial pose information of the virtual capsule endoscope.

[0068] Specifically, the implementation of the image marking method provided by the present invention further includes the following steps.

[0069] Step 4: Create a virtual capsule endoscope and establish a relative coordinate system.

[0070] During the creation of the virtual capsule endoscope, based on the size information of the actual capsule endoscope (i.e., the real capsule endoscope in this embodiment), a virtual capsule endoscope is constructed in the gastric cavity model. And the field of view angle (for example, 160 degrees) is set for the constructed virtual capsule endoscope, and it is verified whether the field of view angle coverage of the capsule endoscope is correct in the virtual gastric cavity model. Through this verification process, the virtual capsule endoscope or the gastric cavity model can be adjusted. At the same time, the parameters of the virtual capsule endoscope are also adjustable, and the adjustment process and results can be visualized.

[0071] Define the global coordinate system (Xg, Yg, Zg) of the gastric cavity and the local coordinate system (Xc, Yc, Zc) of the virtual capsule endoscope. Initialize the coincidence of the two coordinate systems, and during the movement and rotation of the virtual capsule endoscope, track the changes of the virtual capsule endoscope in the global coordinate system of the gastric cavity.

[0072] Use the quaternion Q = [qx, qy, qz, qw] to convert to the rotation matrix R. If the quaternion represents the rotation from the local coordinate system of the virtual capsule endoscope to the global coordinate system of the gastric cavity, the rotation matrix is shown in Equation 1.

[0073] ; (1)

[0074] ; (2)

[0075] During position conversion, convert the local coordinates of the virtual capsule endoscope to the coordinates in the global coordinate system through the rotation matrix R and the position vector T (T represents the translation in the global coordinate system), as shown in Equation 2.

[0076] Update the values of the rotation matrix and the position vector in real time to ensure the accuracy of the coordinate system conversion of the virtual capsule at any position and attitude. By adjusting the parameters, the position and direction of the virtual capsule endoscope can be accurately controlled, and the position and field of view coverage of the virtual capsule endoscope in the gastric cavity model can be viewed immediately.

[0077] In this embodiment, through the creation of the virtual capsule endoscope, the definition and conversion of the relative coordinate system, etc., on the basis of the gastric cavity model, the spatial pose information of the virtual capsule endoscope is accurately obtained.

[0078] In one embodiment, the image annotation method provided by the embodiments of the present application may further include:

[0079] Step 210, perform regional analysis on the virtual gastric cavity image, and determine the proportion of each virtual gastric cavity part in the virtual gastric cavity image based on the analysis result;

[0080] Step 220, associate the proportion of each virtual gastric cavity part with the spatial pose information to obtain an association relationship.

[0081] Specifically, by analyzing the virtual gastric cavity image, one or more gastric cavity regions included in the virtual gastric cavity image can be determined. In the case of multiple gastric cavity regions, regional analysis is performed on the virtual gastric cavity image to obtain each gastric cavity region included in the virtual gastric cavity image and its corresponding proportion. The respective gastric cavity regions included in each virtual gastric cavity image and their corresponding proportions are associated with the spatial pose information of each virtual gastric cavity image, obtaining the association relationship between the respective gastric cavity regions included in each virtual gastric cavity image and their corresponding proportions and the spatial pose information of each virtual gastric cavity image. The confirmation of the above association relationship helps with the subsequent spatial multi-dimensional data annotation of the real gastric cavity image.

[0082] In this embodiment, by performing regional analysis on the virtual gastric cavity image and establishing an association relationship based on the analysis results, it helps with the spatial multi-dimensional data annotation of the real gastric cavity image.

[0083] In one embodiment, the image annotation method provided by the embodiments of the present application may further include:

[0084] Step 310: Determine the proportion of each real gastric cavity part in the real gastric cavity image;

[0085] Step 320: Compare the proportion of each real gastric cavity part in the real gastric cavity image with the proportion of each virtual gastric cavity part in the virtual gastric cavity image;

[0086] Step 330: Determine the target gastric cavity image based on the comparison result; the target gastric cavity image is a virtual gastric cavity image that matches the real gastric cavity image;

[0087] Step 340: Determine the spatial multi-dimensional data of the real gastric cavity image based on the spatial pose information associated with the target gastric cavity image.

[0088] Specifically, by using a deep learning model to extract features from the real gastric cavity image and analyzing the extracted features, the proportion of each real gastric cavity part included in the real gastric cavity image can be determined. The proportion of each real gastric cavity part in the real gastric cavity image is compared with the proportion of each virtual gastric cavity part in each virtual gastric cavity image, and based on the comparison result, the virtual gastric cavity image most similar to the real gastric cavity image among each virtual gastric cavity image is determined. The judgment condition is: the comparison result between the proportion of each gastric cavity part in the real gastric cavity image and the proportion of each gastric cavity part in each virtual gastric cavity image. The virtual gastric cavity image most similar to the real gastric cavity image among each virtual gastric cavity image is the target gastric cavity image in this embodiment. The spatial pose information associated with the target gastric cavity image is used as the spatial multi-dimensional data of the real gastric cavity image to annotate the real gastric cavity image.

[0089] In this embodiment, by comparing the proportion of the actual gastric cavity area with the proportion of the virtual gastric cavity area, the spatial multi-dimensional data of the actual gastric cavity image is determined for data annotation.

[0090] In one embodiment, the image annotation method provided by the embodiments of the present application may further include:

[0091] Step 341: When the actual gastric cavity image is a plurality of consecutive actual images in time, generate a capsule motion trajectory based on the global coordinates of the capsule corresponding to each actual image;

[0092] Step 342: Determine the spatial multi-dimensional data of the actual gastric cavity image based on the global coordinates of the capsule, the capsule shooting information, and the capsule motion trajectory;

[0093] Step 343: When the actual gastric cavity image is a single actual image, determine the spatial multi-dimensional data of the actual gastric cavity image based on the global coordinates of the capsule and the capsule shooting information corresponding to the actual image.

[0094] Specifically, the spatial multi-dimensional data of each actual gastric cavity image includes the global coordinates of the capsule endoscope and the shooting information of the capsule endoscope. By obtaining a plurality of consecutive actual gastric cavity images in time, trajectory analysis is performed on the global coordinates of the capsule endoscope in the spatial multi-dimensional data of each actual gastric cavity image to generate a capsule motion trajectory. It can be known that the generated capsule motion trajectory also needs to be adjusted, including trajectory optimization. The generated capsule motion trajectory can also be subjected to quality evaluation, and the motion trajectory (path) is annotated in the form of a quality score, and the method of quality scoring is not limited. In summary, the spatial multi-dimensional data of a plurality of actual gastric cavity images includes the global coordinates of the capsule, the capsule shooting information, and the capsule motion trajectory; the content of the visual output includes the gastric cavity images captured on the moving path of the capsule played in the form of continuous images, and also includes the overall quality score of the moving path or the image quality score of each coordinate point on the moving path. For the coordinate point images with a low quality score (such as blurred images), they can be excluded to improve the overall data quality. The spatial multi-dimensional data of a single actual gastric cavity image includes the global coordinates of the capsule and the capsule shooting information.

[0095] In this embodiment, by performing trajectory analysis on the global coordinates of the capsule corresponding to each actual gastric cavity image, more comprehensive spatial multi-dimensional data is obtained.

[0096] In one embodiment, the image annotation method provided by the embodiments of the present application may further include:

[0097] Step 500: Receive an output instruction containing coordinate information;

[0098] Step 600: Determine the spatial multi-dimensional data corresponding to the output instruction based on the coordinate information;

[0099] Step 700: Visually display the spatial coordinates, shooting angles, and movement trajectories included in the spatial multi-dimensional data corresponding to the output instruction.

[0100] Specifically, in addition to implementing image data annotation, the image annotation method provided in this application also provides a more intuitive data display in terms of human-computer interaction. In the virtual gastric cavity model, when an output instruction containing coordinate information input by the user is received, the spatial multi-dimensional data associated with the coordinate information is obtained, such as spatial coordinates, shooting angles, and corresponding movement trajectories. The spatial multi-dimensional data associated with these coordinate information is visually displayed and output, enabling the user to view the spatial multi-dimensional data more intuitively.

[0101] This application also provides a richer interaction function. Through the interaction interface, the user can adjust the parameters of the gastric cavity model and the virtual capsule endoscope (model), and change shooting information such as the field of view angle of the virtual capsule endoscope. The gastric cavity model is related to the degree of gastric cavity fullness. Different degrees of gastric cavity fullness will cause changes in the shape of the gastric wall, and these changes are related to the gastric cavity model, that is, different degrees of gastric cavity fullness correspond to different gastric cavity models. The gastric cavity model can be adjusted by adjusting the degree of gastric cavity fullness.

[0102] This embodiment provides more intuitive and multi-dimensional data analysis for the user through the output of visual spatial multi-dimensional data.

[0103] Next, the image annotation device provided by the present invention will be described. The image annotation device described below can be correspondingly referred to the image annotation method described above.

[0104] Please refer to Figure 3 , the present invention also provides an image annotation device, including:

[0105] A spatial pose information determination module 301, configured to determine the spatial pose information of the virtual capsule endoscope based on the positional relationship between the virtual capsule endoscope and the gastric cavity model; the gastric cavity model is created based on historical gastric cavity data;

[0106] An association relationship establishment module 302, configured to establish an association relationship between the virtual gastric cavity image and the spatial pose information;

[0107] A spatial multi-dimensional data determination module 303, configured to compare the real gastric cavity image with the virtual gastric cavity image, and determine the spatial multi-dimensional data of the real gastric cavity image based on the comparison result and the association relationship;

[0108] An image annotation module 304, configured to annotate the real gastric cavity image based on the spatial multi-dimensional data to obtain multi-dimensional image annotation information.

[0109] Optionally, the spatial pose information determination module includes:

[0110] A virtual capsule endoscope construction unit, configured to construct a virtual capsule endoscope in the gastric cavity model based on the size information of the real capsule endoscope; the virtual capsule endoscope is provided with shooting parameters;

[0111] A change information determination unit, configured to determine the global coordinate system of the gastric cavity and the local coordinate system of the capsule, and determine the change information of the virtual capsule endoscope in the global coordinate system of the gastric cavity during the rotation of the virtual capsule endoscope;

[0112] A coordinate conversion unit, configured to convert the local coordinates of the capsule into global coordinates in the global coordinate system of the gastric cavity based on the change information;

[0113] A spatial pose information determination unit, configured to determine the spatial pose information of the virtual capsule endoscope based on the global coordinates and the shooting parameters.

[0114] Optionally, the association relationship establishment module includes:

[0115] A gastric cavity region analysis unit, configured to perform region analysis on the virtual gastric cavity image, and determine the proportion of each virtual gastric cavity part in the virtual gastric cavity image based on the analysis result;

[0116] An association relationship determination unit, configured to associate the proportion of each virtual gastric cavity part with the spatial pose information to obtain an association relationship.

[0117] Optionally, the spatial multi-dimensional data determination module includes:

[0118] A gastric cavity part proportion determination unit, configured to determine the proportion of each real gastric cavity part in the real gastric cavity image;

[0119] A gastric cavity part proportion comparison unit, configured to compare the proportion of each real gastric cavity part in the real gastric cavity image with the proportion of each virtual gastric cavity part in the virtual gastric cavity image;

[0120] A target gastric cavity image determination unit, configured to determine a target gastric cavity image based on the comparison result; the target gastric cavity image is a virtual gastric cavity image that matches the real gastric cavity image;

[0121] A spatial multi-dimensional data determination unit, configured to determine the spatial multi-dimensional data of the real gastric cavity image based on the spatial pose information associated with the target gastric cavity image.

[0122] Optionally, the spatial pose information includes the global coordinates of the capsule and the capsule shooting information; the spatial multi-dimensional data determination unit includes:

[0123] A capsule motion trajectory generation unit, configured to generate a capsule motion trajectory based on the global coordinates of the capsule corresponding to each real image when the real gastric cavity image is a plurality of consecutive real images in time;

[0124] A first determination unit, configured to determine the spatial multi-dimensional data of the real gastric cavity image based on the global coordinates of the capsule, the capsule shooting information, and the capsule motion trajectory;

[0125] A second determination unit, configured to determine the spatial multi-dimensional data of the real gastric cavity image based on the global coordinates of the capsule and the capsule shooting information corresponding to the real image when the real gastric cavity image is a single real image.

[0126] Optionally, the image annotation device further includes:

[0127] An output instruction receiving module, configured to receive an output instruction including coordinate information;

[0128] A first determination module, configured to determine the spatial multi-dimensional data corresponding to the output instruction based on the coordinate information;

[0129] A visualization output module, configured to visually display the spatial coordinates, shooting angles, and movement trajectories included in the spatial multi-dimensional data corresponding to the output instruction.

[0130] Figure 4 An example of the physical structure diagram of an electronic device is shown as Figure 4 shown. The electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440. Among them, the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call the logical instructions in the memory 430 to execute an image annotation method, which includes: determining the spatial pose information of the virtual capsule endoscope based on the positional relationship between the virtual capsule endoscope and the gastric cavity model; the gastric cavity model is created based on historical gastric cavity data; establishing an association relationship between the virtual gastric cavity image and the spatial pose information; comparing the real gastric cavity image with the virtual gastric cavity image, and determining the spatial multi-dimensional data of the real gastric cavity image based on the comparison result and the association relationship; annotating the real gastric cavity image based on the spatial multi-dimensional data to obtain multi-dimensional image annotation information.

[0131] In addition, when the logical instructions in the above-mentioned memory 430 can be implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0132] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image annotation method provided by the above-mentioned various methods. The method includes: determining the spatial pose information of the virtual capsule endoscope based on the positional relationship between the virtual capsule endoscope and the gastric cavity model; the gastric cavity model is created based on historical gastric cavity data; establishing an association relationship between the virtual gastric cavity image and the spatial pose information; comparing the real gastric cavity image with the virtual gastric cavity image, and based on the comparison result and the association relationship, determining the spatial multi-dimensional data of the real gastric cavity image; and annotating the real gastric cavity image based on the spatial multi-dimensional data to obtain multi-dimensional image annotation information.

[0133] In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the image annotation method provided by the above-mentioned various methods. The method includes: determining the spatial pose information of the virtual capsule endoscope based on the positional relationship between the virtual capsule endoscope and the gastric cavity model; the gastric cavity model is created based on historical gastric cavity data; establishing an association relationship between the virtual gastric cavity image and the spatial pose information; comparing the real gastric cavity image with the virtual gastric cavity image, and based on the comparison result and the association relationship, determining the spatial multi-dimensional data of the real gastric cavity image; and annotating the real gastric cavity image based on the spatial multi-dimensional data to obtain multi-dimensional image annotation information.

[0134] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative work.

[0135] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image annotation method, characterized in that: include: Determining spatial position information of the virtual capsule endoscope based on a positional relationship between the virtual capsule endoscope and the gastric cavity model; The gastric cavity model is created based on historical gastric cavity data; Establishing an association relationship between the virtual stomach cavity image and the spatial posture information; Comparing the real stomach cavity image with the virtual stomach cavity image, and determining spatial multi-dimensional data of the real stomach cavity image based on the comparison result and the association relationship; Annotating the actual stomach cavity image based on the spatial multi-dimensional data to obtain multi-dimensional image annotation information; The comparing the real stomach cavity image with the virtual stomach cavity image and determining the spatial multi-dimensional data of the real stomach cavity image based on the comparison result and the association relationship comprises: Determine the proportion of each actual stomach cavity part in the actual stomach cavity image; comparing the proportion of each real stomach cavity part in the real stomach cavity image with the proportion of each virtual stomach cavity part in the virtual stomach cavity image; Determine a target stomach cavity image based on the comparison result; the target stomach cavity image is a virtual stomach cavity image that matches the actual stomach cavity image; Determining spatial multi-dimensional data of the actual stomach cavity image based on the spatial posture information associated with the target stomach cavity image; The spatial posture information includes capsule global coordinates and capsule shooting information; the spatial multi-dimensional data of the actual gastric cavity image is determined based on the spatial posture information associated with the target gastric cavity image, including: In the case where the real stomach cavity image is a plurality of real images that are continuous in time, generating a capsule motion trajectory based on the capsule global coordinates corresponding to each real image; Determining spatial multi-dimensional data of the actual gastric cavity image based on the capsule global coordinates, the capsule shooting information, and the capsule motion trajectory; In the case that the actual gastric cavity image is a single actual image, the spatial multi-dimensional data of the actual gastric cavity image is determined based on the capsule global coordinates and capsule shooting information corresponding to the actual image.

2. The image annotation method according to claim 1, characterized in that: The determining of the spatial posture information of the virtual capsule endoscope based on the positional relationship between the virtual capsule endoscope and the gastric cavity model includes: Based on the size information of the real capsule endoscope, a virtual capsule endoscope is constructed in the gastric cavity model; the virtual capsule endoscope is provided with shooting parameters; Determine a gastric cavity global coordinate system and a capsule local coordinate system, and determine change information of the virtual capsule endoscope in the gastric cavity global coordinate system during the rotation of the virtual capsule endoscope; Based on the change information, the local coordinates of the capsule are converted into global coordinates in the global coordinate system of the stomach cavity; Based on the global coordinates and the shooting parameters, spatial position information of the virtual capsule endoscope is determined.

3. The image annotation method according to claim 1, characterized in that: The establishing of the association relationship between the virtual stomach cavity image and the spatial posture information comprises: Performing regional analysis on the virtual stomach cavity image, and determining the proportion of each virtual stomach cavity part in the virtual stomach cavity image based on the analysis result; The proportion of each virtual stomach cavity part is associated with the spatial posture information to obtain an associated relationship.

4. The image annotation method according to claim 1, characterized in that: The present stomach cavity image is annotated based on the spatial multi-dimensional data to obtain multi-dimensional image annotation information, and then includes: receiving an output instruction including coordinate information; Based on the coordinate information, determining the spatial multi-dimensional data corresponding to the output instruction; The spatial coordinates, shooting angles and movement trajectories included in the spatial multi-dimensional data corresponding to the output instruction are visualized.

5. An image annotation device, characterized in that: include: A spatial posture information determination module, used to determine the spatial posture information of the virtual capsule endoscope based on the positional relationship between the virtual capsule endoscope and the gastric cavity model; The gastric cavity model is created based on historical gastric cavity data; An association relationship establishing module, used for establishing an association relationship between the virtual stomach cavity image and the spatial posture information; A spatial multi-dimensional data determination module, used for comparing the real stomach cavity image with the virtual stomach cavity image, and determining the spatial multi-dimensional data of the real stomach cavity image based on the comparison result and the association relationship; An image annotation module, used to annotate the actual stomach cavity image based on the spatial multi-dimensional data to obtain multi-dimensional image annotation information; The spatial multi-dimensional data determination module includes: A stomach cavity part proportion determination unit, used to determine the proportion of each actual stomach cavity part in the actual stomach cavity image; a stomach cavity part proportion comparison unit, used for comparing the proportion of each real stomach cavity part in the real stomach cavity image with the proportion of each virtual stomach cavity part in the virtual stomach cavity image; a target stomach cavity image determining unit, configured to determine a target stomach cavity image based on the comparison result; the target stomach cavity image is a virtual stomach cavity image that matches the actual stomach cavity image; A spatial multi-dimensional data determining unit, configured to determine the spatial multi-dimensional data of the actual stomach cavity image based on the spatial posture information associated with the target stomach cavity image; The spatial posture information includes capsule global coordinates and capsule shooting information; the spatial multi-dimensional data determination unit includes: A capsule motion trajectory generating unit, configured to generate a capsule motion trajectory based on the capsule global coordinates corresponding to each real image when the real stomach cavity image is a plurality of real images that are continuous in time; A first determining unit, configured to determine spatial multi-dimensional data of the actual gastric cavity image based on the capsule global coordinates, the capsule shooting information, and the capsule motion trajectory; The second determination unit is used to determine the spatial multi-dimensional data of the real gastric cavity image based on the capsule global coordinates and capsule shooting information corresponding to the real image when the real gastric cavity image is a single real image.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the image annotation method according to any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image annotation method according to any one of claims 1 to 4 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image annotation method according to any one of claims 1 to 4 is implemented.

Images