Image annotation method, device, equipment, medium and computer program product

Abstract

The invention relates to the technical field of image processing, and provides an image labeling method, device and equipment, a medium and a computer program product.The method comprises the steps that based on the position relation between a virtual capsule endoscope and a gastral cavity model, space pose information of the virtual capsule endoscope is determined; the gastral cavity model is created based on historical gastral cavity data; establishing an association relationship between the virtual gastral cavity image and the spatial pose information; comparing a real gastral cavity image with the virtual gastral cavity image, and determining spatial multi-dimensional data of the real gastral cavity image based on a comparison result and the association relationship; and marking the real gastral cavity image based on the spatial multi-dimensional data to obtain multi-dimensional image marking information. According to the method, by labeling the multi-dimensional spatial data for the gastral cavity image, the workload of personnel is reduced, the data labeling efficiency of the gastral cavity image is improved, more comprehensive and accurate data are labeled for the gastral cavity image, and accurate construction of gastral cavity analysis related algorithms is facilitated.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image annotation method, device, equipment, medium and computer program product. Background Art

[0002] In the medical field, image analysis is widely used in the diagnosis of gastrointestinal diseases. Capsule endoscopes are available tools for obtaining images of the gastrointestinal tract, but the movement of capsule endoscopes inside the gastrointestinal tract is uncontrollable, and the temporal and spatial correlation between the acquired images is not strong, which makes data annotation of gastric cavity images particularly difficult. The construction and improvement of various algorithms for gastric cavity analysis (such as gastric mucosal coverage assessment algorithms) rely on gastric cavity images with accurate and comprehensive data annotation. Therefore, how to annotate gastric cavity images with more comprehensive and accurate data and how to improve the efficiency of data annotation of gastric cavity images have become technical problems that need to be solved urgently. Summary of the invention

[0003] The present invention provides an image annotation method, device, equipment, medium and computer program product, which are used to solve the defects of the existing gastric cavity image data annotation technology that the annotation data is not comprehensive and accurate and the annotation efficiency is low, and realize comprehensive, accurate and efficient annotation of gastric cavity image data.

[0004] The present invention provides an image annotation method, comprising the following steps.

[0005] Determining spatial position information of the virtual capsule endoscope based on a positional relationship between the virtual capsule endoscope and a gastric cavity model; the gastric cavity model is created based on historical gastric cavity data; Establishing an association relationship between the virtual stomach cavity image and the spatial posture information; Comparing the real stomach cavity image with the virtual stomach cavity image, and determining spatial multi-dimensional data of the real stomach cavity image based on the comparison result and the association relationship; The actual stomach cavity image is annotated based on the spatial multi-dimensional data to obtain multi-dimensional image annotation information.

[0006] According to an image annotation method provided by the present invention, determining the spatial posture information of the virtual capsule endoscope based on the positional relationship between the virtual capsule endoscope and the gastric cavity model includes: Based on the size information of the real capsule endoscope, a virtual capsule endoscope is constructed in the gastric cavity model; the virtual capsule endoscope is provided with shooting parameters; Determine a gastric cavity global coordinate system and a capsule local coordinate system, and determine change information of the virtual capsule endoscope in the gastric cavity global coordinate system during the rotation of the virtual capsule endoscope; Based on the change information, the local coordinates of the capsule are converted into global coordinates in the global coordinate system of the stomach cavity; Based on the global coordinates and the shooting parameters, spatial posture information of the virtual capsule endoscope is determined.

[0007] According to an image annotation method provided by the present invention, the establishing of an association relationship between a virtual stomach cavity image and the spatial posture information comprises: Performing regional analysis on the virtual stomach cavity image, and determining the proportion of each virtual stomach cavity part in the virtual stomach cavity image based on the analysis result; The proportion of each virtual stomach cavity part is associated with the spatial posture information to obtain an associated relationship.

[0008] According to an image annotation method provided by the present invention, comparing the real stomach cavity image with the virtual stomach cavity image, and determining the spatial multi-dimensional data of the real stomach cavity image based on the comparison result and the association relationship includes: Determine the proportion of each actual stomach cavity part in the actual stomach cavity image; comparing the proportion of each real stomach cavity part in the real stomach cavity image with the proportion of each virtual stomach cavity part in the virtual stomach cavity image; Determine a target stomach cavity image based on the comparison result; the target stomach cavity image is a virtual stomach cavity image that matches the actual stomach cavity image; Based on the spatial posture information associated with the target stomach cavity image, the spatial multi-dimensional data of the actual stomach cavity image is determined.

[0009] According to an image annotation method provided by the present invention, the spatial posture information includes capsule global coordinates and capsule shooting information; the spatial multi-dimensional data of the actual gastric cavity image is determined based on the spatial posture information associated with the target gastric cavity image, including: In the case where the real stomach cavity image is a plurality of real images that are continuous in time, generating a capsule motion trajectory based on the capsule global coordinates corresponding to each real image; Determining spatial multi-dimensional data of the actual gastric cavity image based on the capsule global coordinates, the capsule shooting information, and the capsule motion trajectory; In the case that the actual gastric cavity image is a single actual image, the spatial multi-dimensional data of the actual gastric cavity image is determined based on the capsule global coordinates and capsule shooting information corresponding to the actual image.

[0010] According to an image annotation method provided by the present invention, the real stomach cavity image is annotated based on the spatial multi-dimensional data to obtain multi-dimensional image annotation information, and then includes: receiving an output instruction including coordinate information; Based on the coordinate information, determining the spatial multi-dimensional data corresponding to the output instruction; The spatial coordinates, shooting angles and movement trajectories included in the spatial multi-dimensional data corresponding to the output instruction are visualized.

[0011] The present invention also provides an image annotation device, comprising the following modules: A spatial posture information determination module, used to determine the spatial posture information of the virtual capsule endoscope based on the positional relationship between the virtual capsule endoscope and the gastric cavity model; the gastric cavity model is created based on historical gastric cavity data; An association relationship establishing module, used for establishing an association relationship between the virtual stomach cavity image and the spatial posture information; A spatial multi-dimensional data determination module, used for comparing the real stomach cavity image with the virtual stomach cavity image, and determining the spatial multi-dimensional data of the real stomach cavity image based on the comparison result and the association relationship; The image annotation module is used to annotate the actual stomach cavity image based on the spatial multi-dimensional data to obtain multi-dimensional image annotation information.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any one of the above-mentioned image annotation methods is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image annotation method described in any one of the above is implemented.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the image annotation method described above is implemented.

[0015] The image annotation method, device, equipment, medium and computer program product provided by the present invention determine the spatial pose information of the virtual capsule endoscope through the positional relationship between the pre-constructed virtual capsule endoscope and the gastric cavity model, then associate the virtual gastric cavity image corresponding to the virtual capsule endoscope with the spatial pose information, compare the real gastric cavity image with the virtual gastric cavity image to be marked, and determine the spatial multi-dimensional data for annotating the real gastric cavity image according to the association relationship and the comparison result, and finally use the spatial multi-dimensional data to annotate the real gastric cavity image. The present invention reduces the workload of relevant technical personnel by annotating multi-dimensional spatial data for gastric cavity images, improves the efficiency of gastric cavity image data annotation, and annotates the gastric cavity image with more comprehensive and accurate data, which is conducive to the accurate construction of gastric cavity analysis related algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 This is one of the flow charts of the image annotation method provided by the present invention.

[0018] Figure 2 This is the second flowchart of the image annotation method provided by the present invention.

[0019] Figure 3 It is a structural schematic diagram of the image annotation device provided by the present invention.

[0020] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Combine the following Figure 1-Figure 4 The present invention describes an image annotation method, apparatus, device, medium and computer program product.

[0023] Figure 1 is one of the flow charts of the image annotation method provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 100: determining spatial position information of the virtual capsule endoscope based on the positional relationship between the virtual capsule endoscope and the gastric cavity model; the gastric cavity model is created based on historical gastric cavity data; Specifically, the implementation of the image annotation method provided by the present invention mainly includes the following steps.

[0024] Step 1: Construction of a three-dimensional gastric cavity model.

[0025] The construction process of the three-dimensional gastric cavity model mainly includes data acquisition, data processing, model optimization, hierarchical modeling and model verification.

[0026] During the data collection process, a scanner (computed tomography CT or magnetic resonance imaging MRI) is used to obtain the gastric cavity data of a person at different degrees of fullness (related to the amount of water consumed). These gastric cavity data serve as the basis for constructing a three-dimensional gastric cavity model to ensure that the three-dimensional gastric cavity model is highly consistent with the actual internal structure of the gastric cavity.

[0027] In the data processing process, the stomach cavity data is loaded and analyzed to extract the internal contour and structure of the stomach cavity and generate a preliminary stomach cavity model; in the model optimization process, the preliminary generated stomach cavity model is optimized (such as smoothing and simplifying) to optimize the surface details of the stomach cavity. The higher the level of refinement of the three-dimensional model, the higher its complexity. In the hierarchical modeling process, three-dimensional models of different refinement levels are created to meet different needs.

[0028] Step 2: Model format and compatibility: This step is mainly to ensure the compatibility of the gastric cavity model in different development environments.

[0029] Step 3: Loading and displaying the stomach cavity model. This step is used to load and render the stomach cavity model to ensure rendering performance in complex scenes and improve the interaction between the stomach cavity model and the user.

[0030] After the stomach cavity model is constructed, a capsule endoscope model is created on the basis of the stomach cavity model, that is, the virtual capsule endoscope in this embodiment. And the parameters related to the real capsule endoscope are set for the virtual capsule endoscope, such as the field of view angle, posture, and movement mode. By defining the global coordinate system of the stomach cavity model and the local coordinate system of the virtual capsule endoscope, the coordinates of the virtual capsule endoscope are converted to the coordinates of the stomach cavity model through matrix conversion and position conversion, and then the spatial posture information of the virtual capsule endoscope, such as coordinates and spatial posture, is represented by the global coordinate system. The spatial posture information also includes the above-mentioned related parameters.

[0031] Step 200, establishing an association relationship between the virtual stomach cavity image and the spatial posture information; Specifically, the virtual gastric cavity image in this embodiment refers to the corresponding gastric cavity image of the virtual capsule endoscope at a certain position and posture and at a certain viewing angle, and the virtual gastric cavity image is associated with the spatial posture information so that each virtual gastric cavity image has its corresponding spatial posture information. It can be seen that the virtual gastric cavity image includes each gastric cavity divided area and its corresponding proportion. For example, the gastric cavity can be divided into areas 1 to 6, and a certain virtual gastric cavity image includes area 1, accounting for 80%; area 2, accounting for 20%.

[0032] Step 300: Compare the real stomach cavity image with the virtual stomach cavity image, and determine the spatial multi-dimensional data of the real stomach cavity image based on the comparison result and the association relationship; Specifically, by combining the joint model of the convolutional neural network CNN and the edge detection algorithm, the real gastric cavity image obtained by the real capsule endoscope is automatically classified and edge detected, and the gastric cavity areas contained in the real gastric cavity image and their corresponding proportions can be obtained. Through the design and training of the joint loss function, the joint model can simultaneously obtain good performance in the gastric cavity area classification task and the gastric cavity area edge detection task. The real gastric cavity image is compared with each virtual gastric cavity image to determine a virtual gastric cavity image that is most similar to the real gastric cavity image among the virtual gastric cavity images. Then, based on the spatial posture information associated with the virtual gastric cavity image, the spatial multi-dimensional data of the real gastric cavity image is determined, that is, the spatial posture information associated with the virtual gastric cavity image is used as the spatial multi-dimensional data of the real gastric cavity image.

[0033] Step 400: annotate the actual stomach cavity image based on the spatial multi-dimensional data to obtain multi-dimensional image annotation information.

[0034] Specifically, the spatial multi-dimensional data obtained above are used to annotate the actual gastric cavity image, so that the annotated actual gastric cavity image can be used to construct and improve various algorithms for gastric cavity analysis, which helps to improve the accuracy of related gastric cavity analysis algorithms.

[0035] This embodiment determines the spatial posture information of the virtual capsule endoscope through the positional relationship between the pre-constructed virtual capsule endoscope and the gastric cavity model, then associates the virtual gastric cavity image corresponding to the virtual capsule endoscope with the spatial posture information, compares the real gastric cavity image with the virtual gastric cavity image to be marked, and determines the spatial multi-dimensional data for marking the real gastric cavity image based on the association relationship and the comparison result, and finally uses the spatial multi-dimensional data to mark the real gastric cavity image. The present invention reduces the workload of relevant technical personnel by marking multi-dimensional spatial data for gastric cavity images, improves the efficiency of gastric cavity image data marking, and marks the gastric cavity images with more comprehensive and accurate data, which is conducive to the accurate construction of gastric cavity analysis related algorithms.

[0036] Figure 2 FIG. 2 is a flow chart of the image annotation method provided by the present invention. Figure 2 As shown, the method may also include: Step 110: construct a virtual capsule endoscope in the gastric cavity model based on the size information of the real capsule endoscope; the virtual capsule endoscope is provided with shooting parameters; Step 120, determining a gastric cavity global coordinate system and a capsule local coordinate system, and determining change information of the virtual capsule endoscope in the gastric cavity global coordinate system during the rotation of the virtual capsule endoscope; Step 130: based on the change information, convert the capsule local coordinates into global coordinates in the gastric cavity global coordinate system; Step 140: Determine the spatial position information of the virtual capsule endoscope based on the global coordinates and the shooting parameters.

[0037] Specifically, the implementation of the image annotation method provided by the present invention also includes the following steps.

[0038] Step 4: Create a virtual capsule endoscope and establish a relative coordinate system.

[0039] In the process of creating a virtual capsule endoscope, a virtual capsule endoscope is constructed in the gastric cavity model based on the size information of the actual capsule endoscope (i.e., the real capsule endoscope in this embodiment). The field of view angle (e.g., 160 degrees) is set for the constructed virtual capsule endoscope, and the field of view angle coverage of the capsule endoscope is verified in the virtual gastric cavity model to see whether it is correct. Through this verification process, the virtual capsule endoscope or the gastric cavity model can be adjusted. At the same time, the parameters of the virtual capsule endoscope are also adjustable, and the adjustment process and results can be visualized.

[0040] Define the gastric cavity global coordinate system (Xg, Yg, Zg) and the local coordinate system (Xc, Yc, Zc) of the virtual capsule endoscope. Initialize the two coordinate systems to coincide with each other, and track the changes of the virtual capsule endoscope in the gastric cavity global coordinate system during the movement and rotation of the virtual capsule endoscope.

[0041] The quaternion Q = [qx, qy, qz, qw] is converted into a rotation matrix R. If the quaternion represents the rotation from the local coordinate system of the virtual capsule endoscope to the global coordinate system of the gastric cavity, the rotation matrix is ​​shown in Formula 1.

[0042] ; (1) ; (2) During position transformation, the local coordinates of the virtual capsule endoscope are The coordinates are transformed to the global coordinate system through the rotation matrix R and the position vector T (T represents the translation in the global coordinate system) , as shown in Formula 2.

[0043] The values ​​of the rotation matrix and position vector are updated in real time to ensure the accuracy of the coordinate system transformation of the virtual capsule in any position and posture. By adjusting the parameters, the position and direction of the virtual capsule endoscope can be accurately controlled, and the position and field of view coverage of the virtual capsule endoscope in the gastric cavity model can be instantly viewed.

[0044] In this embodiment, the spatial position information of the virtual capsule endoscope is accurately obtained on the basis of the gastric cavity model through the creation of the virtual capsule endoscope, the definition and conversion of the relative coordinate system and other processes.

[0045] In one embodiment, the image annotation method provided in the embodiment of the present application may further include: Step 210: performing regional analysis on the virtual stomach cavity image, and determining the proportion of each virtual stomach cavity part in the virtual stomach cavity image based on the analysis result; Step 220: Associating the proportion of each virtual stomach cavity part with the spatial posture information to obtain an association relationship.

[0046] Specifically, by analyzing the virtual gastric cavity image, one or more gastric cavity areas contained in the virtual gastric cavity image can be determined. When multiple gastric cavity areas are contained, regional analysis is performed on the virtual gastric cavity image to obtain the gastric cavity areas contained in the virtual gastric cavity image and their corresponding proportions. The gastric cavity areas contained in each virtual gastric cavity image and their corresponding proportions are associated with the spatial posture information of each virtual gastric cavity image to obtain the association relationship between the gastric cavity areas contained in each virtual gastric cavity image and their corresponding proportions and the spatial posture information of each virtual gastric cavity image. The confirmation of the above association relationship is helpful for the subsequent spatial multi-dimensional data annotation of real gastric cavity images.

[0047] This embodiment performs regional analysis on the virtual stomach cavity image and establishes a correlation based on the analysis result, which helps to label the spatial multi-dimensional data of the real stomach cavity image.

[0048] In one embodiment, the image annotation method provided in the embodiment of the present application may further include: Step 310, determining the proportion of each actual stomach cavity part in the actual stomach cavity image; Step 320: Compare the proportion of each real stomach cavity part in the real stomach cavity image with the proportion of each virtual stomach cavity part in the virtual stomach cavity image; Step 330: determining a target stomach cavity image based on the comparison result; the target stomach cavity image is a virtual stomach cavity image that matches the actual stomach cavity image; Step 340: Determine the spatial multi-dimensional data of the actual stomach cavity image based on the spatial posture information associated with the target stomach cavity image.

[0049] Specifically, by extracting features from the real gastric cavity image through a deep learning model and analyzing the extracted features, the proportion of each real gastric cavity part contained in the real gastric cavity image can be determined. The proportion of each real gastric cavity part in the real gastric cavity image is compared with the proportion of each virtual gastric cavity part in each virtual gastric cavity image, and the virtual gastric cavity image that is most similar to the real gastric cavity image in each virtual gastric cavity image is determined based on the comparison result. The judgment condition is: the comparison result between the proportion of each gastric cavity part in the real gastric cavity image and the proportion of each gastric cavity part in each virtual gastric cavity image. The virtual gastric cavity image that is most similar to the real gastric cavity image in each virtual gastric cavity image is the target gastric cavity image in this embodiment. The spatial posture information associated with the target gastric cavity image is used as the spatial multi-dimensional data of the real gastric cavity image to annotate the real gastric cavity image.

[0050] In this embodiment, the spatial multi-dimensional data of the real stomach cavity image is determined for data annotation by comparing the real stomach cavity portion ratio with the virtual stomach cavity portion ratio.

[0051] In one embodiment, the image annotation method provided in the embodiment of the present application may further include: Step 341: when the real stomach cavity image is a plurality of real images that are continuous in time, generate a capsule motion trajectory based on the capsule global coordinates corresponding to each real image; Step 342: determining spatial multi-dimensional data of the actual gastric cavity image based on the capsule global coordinates, the capsule shooting information, and the capsule motion trajectory; Step 343: When the actual gastric cavity image is a single actual image, determine the spatial multi-dimensional data of the actual gastric cavity image based on the capsule global coordinates and capsule shooting information corresponding to the actual image.

[0052] Specifically, the spatial multi-dimensional data of each real gastric cavity image includes the global coordinates of the capsule endoscope and the shooting information of the capsule endoscope. By acquiring multiple real gastric cavity images that are continuous in time, the global coordinates of the capsule endoscope in the spatial multi-dimensional data of each real gastric cavity image are analyzed by trajectory analysis to generate a capsule motion trajectory. It can be seen that the generated capsule motion trajectory also needs to be adjusted, including trajectory optimization. The generated capsule motion trajectory can also be evaluated for quality, and the motion trajectory (path) is marked in the form of a quality score, and the quality score method is not limited. In summary, the spatial multi-dimensional data of multiple real gastric cavity images include capsule global coordinates, capsule shooting information, and capsule motion trajectory; the content of the visual output includes the gastric cavity images captured on the capsule moving path played in the form of continuous images, and also includes the overall quality score of the moving path or the image quality score of each coordinate point on the moving path. Coordinate point images with low quality scores (such as blurred images) can be eliminated to improve the overall data quality. The spatial multi-dimensional data of a single real gastric cavity image includes capsule global coordinates and capsule shooting information.

[0053] This embodiment obtains more comprehensive spatial multi-dimensional data by performing trajectory analysis on the global coordinates of the capsule corresponding to each real stomach cavity image.

[0054] In one embodiment, the image annotation method provided in the embodiment of the present application may further include: Step 500, receiving an output instruction including coordinate information; Step 600: Determine the spatial multi-dimensional data corresponding to the output instruction based on the coordinate information; Step 700: Visually display the spatial coordinates, shooting angles, and movement trajectories included in the spatial multi-dimensional data corresponding to the output instruction.

[0055] Specifically, in addition to realizing image data annotation, the image annotation method provided by the present application also provides a more intuitive data display in terms of human-computer interaction. In the virtual gastric cavity model, after receiving an output instruction containing coordinate information input by the user, the spatial multi-dimensional data associated with the coordinate information is obtained, such as spatial coordinates, shooting angles, and corresponding movement trajectories. The spatial multi-dimensional data associated with these coordinate information is visualized and output, so that users can view the spatial multi-dimensional data more intuitively.

[0056] This application also provides richer interactive functions. Through the interactive interface, users can adjust the parameters of the gastric cavity model and the virtual capsule endoscope (model), and change the field of view angle and other shooting information of the virtual capsule endoscope. The gastric cavity model is related to the gastric cavity filling degree. Different gastric cavity filling degrees will cause changes in the morphology of the stomach wall. These changes are related to the gastric cavity model, that is, different gastric cavity filling degrees correspond to different gastric cavity models. The gastric cavity model can be adjusted by adjusting the gastric cavity filling degree.

[0057] This embodiment provides users with more intuitive and multi-dimensional data analysis through visualized spatial multi-dimensional data output.

[0058] The image annotation device provided by the present invention is described below. The image annotation device described below and the image annotation method described above can be referred to each other.

[0059] Please refer to Figure 3 The present invention also provides an image annotation device, comprising: A spatial posture information determination module 301 is used to determine the spatial posture information of the virtual capsule endoscope based on the positional relationship between the virtual capsule endoscope and the gastric cavity model; the gastric cavity model is created based on historical gastric cavity data; An association relationship establishing module 302 is used to establish an association relationship between the virtual stomach cavity image and the spatial posture information; A spatial multi-dimensional data determination module 303 is used to compare the real stomach cavity image with the virtual stomach cavity image, and determine the spatial multi-dimensional data of the real stomach cavity image based on the comparison result and the association relationship; The image annotation module 304 is used to annotate the actual stomach cavity image based on the spatial multi-dimensional data to obtain multi-dimensional image annotation information.

[0060] Optionally, the spatial posture information determination module includes: A virtual capsule endoscope construction unit is used to construct a virtual capsule endoscope in the gastric cavity model based on the size information of the real capsule endoscope; the virtual capsule endoscope is provided with shooting parameters; a change information determining unit, used for determining a gastric cavity global coordinate system and a capsule local coordinate system, and determining change information of the virtual capsule endoscope in the gastric cavity global coordinate system during the rotation of the virtual capsule endoscope; A coordinate conversion unit, configured to convert the capsule local coordinates into global coordinates in the gastric cavity global coordinate system based on the change information; A spatial posture information determination unit is used to determine the spatial posture information of the virtual capsule endoscope based on the global coordinates and the shooting parameters.

[0061] Optionally, the association relationship establishing module includes: A stomach cavity region analysis unit, used for performing a region analysis on the virtual stomach cavity image, and determining a proportion of each virtual stomach cavity part in the virtual stomach cavity image based on the analysis result; The association relationship determination unit is used to associate the proportion of each virtual stomach cavity part with the spatial posture information to obtain an association relationship.

[0062] Optionally, the spatial multi-dimensional data determination module includes: A stomach cavity part proportion determination unit, used to determine the proportion of each actual stomach cavity part in the actual stomach cavity image; a stomach cavity part proportion comparison unit, used for comparing the proportion of each real stomach cavity part in the real stomach cavity image with the proportion of each virtual stomach cavity part in the virtual stomach cavity image; a target stomach cavity image determining unit, configured to determine a target stomach cavity image based on the comparison result; the target stomach cavity image is a virtual stomach cavity image that matches the actual stomach cavity image; The spatial multi-dimensional data determining unit is used to determine the spatial multi-dimensional data of the actual stomach cavity image based on the spatial posture information associated with the target stomach cavity image.

[0063] Optionally, the spatial posture information includes capsule global coordinates and capsule shooting information; and the spatial multi-dimensional data determination unit includes: A capsule motion trajectory generating unit, configured to generate a capsule motion trajectory based on the capsule global coordinates corresponding to each real image when the real stomach cavity image is a plurality of real images that are continuous in time; A first determining unit, configured to determine spatial multi-dimensional data of the actual gastric cavity image based on the capsule global coordinates, the capsule shooting information, and the capsule motion trajectory; The second determination unit is used to determine the spatial multi-dimensional data of the real gastric cavity image based on the capsule global coordinates and capsule shooting information corresponding to the real image when the real gastric cavity image is a single real image.

[0064] Optionally, the image annotation device further includes: An output instruction receiving module, used for receiving an output instruction containing coordinate information; A first determination module, used to determine the spatial multi-dimensional data corresponding to the output instruction based on the coordinate information; The visualization output module is used to visualize the spatial coordinates, shooting angles and movement trajectories contained in the spatial multi-dimensional data corresponding to the output instruction.

[0065] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the image annotation method, which includes: determining the spatial posture information of the virtual capsule endoscope based on the positional relationship between the virtual capsule endoscope and the gastric cavity model; the gastric cavity model is created based on historical gastric cavity data; establishing an association relationship between the virtual gastric cavity image and the spatial posture information; comparing the real gastric cavity image with the virtual gastric cavity image, and determining the spatial multi-dimensional data of the real gastric cavity image based on the comparison result and the association relationship; annotating the real gastric cavity image based on the spatial multi-dimensional data to obtain multi-dimensional image annotation information.

[0066] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0067] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image annotation method provided by the above-mentioned methods, which includes: determining the spatial posture information of the virtual capsule endoscope based on the positional relationship between the virtual capsule endoscope and the gastric cavity model; the gastric cavity model is created based on historical gastric cavity data; establishing an association relationship between the virtual gastric cavity image and the spatial posture information; comparing the real gastric cavity image with the virtual gastric cavity image, and determining the spatial multi-dimensional data of the real gastric cavity image based on the comparison result and the association relationship; annotating the real gastric cavity image based on the spatial multi-dimensional data to obtain multi-dimensional image annotation information.

[0068] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the image annotation method provided by the above-mentioned methods, the method comprising: determining the spatial posture information of the virtual capsule endoscope based on the positional relationship between the virtual capsule endoscope and the gastric cavity model; the gastric cavity model is created based on historical gastric cavity data; establishing an association relationship between the virtual gastric cavity image and the spatial posture information; comparing the real gastric cavity image with the virtual gastric cavity image, and determining the spatial multi-dimensional data of the real gastric cavity image based on the comparison result and the association relationship; annotating the real gastric cavity image based on the spatial multi-dimensional data to obtain multi-dimensional image annotation information.

[0069] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0070] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image annotation method, characterized in that: include: Determining spatial position information of the virtual capsule endoscope based on a positional relationship between the virtual capsule endoscope and the gastric cavity model; The gastric cavity model is created based on historical gastric cavity data; Establishing an association relationship between the virtual stomach cavity image and the spatial posture information; Comparing the real stomach cavity image with the virtual stomach cavity image, and determining spatial multi-dimensional data of the real stomach cavity image based on the comparison result and the association relationship; The actual stomach cavity image is annotated based on the spatial multi-dimensional data to obtain multi-dimensional image annotation information.

2. The image annotation method according to claim 1, characterized in that: The determining of the spatial posture information of the virtual capsule endoscope based on the positional relationship between the virtual capsule endoscope and the gastric cavity model includes: Based on the size information of the real capsule endoscope, a virtual capsule endoscope is constructed in the gastric cavity model; the virtual capsule endoscope is provided with shooting parameters; Determine a gastric cavity global coordinate system and a capsule local coordinate system, and determine change information of the virtual capsule endoscope in the gastric cavity global coordinate system during the rotation of the virtual capsule endoscope; Based on the change information, the local coordinates of the capsule are converted into global coordinates in the global coordinate system of the stomach cavity; Based on the global coordinates and the shooting parameters, spatial position information of the virtual capsule endoscope is determined.

3. The image annotation method according to claim 1, characterized in that: The establishing of the association relationship between the virtual stomach cavity image and the spatial posture information comprises: Performing regional analysis on the virtual stomach cavity image, and determining the proportion of each virtual stomach cavity part in the virtual stomach cavity image based on the analysis result; The proportion of each virtual stomach cavity part is associated with the spatial posture information to obtain an associated relationship.

4. The image annotation method according to claim 1, characterized in that: The comparing the real stomach cavity image with the virtual stomach cavity image and determining the spatial multi-dimensional data of the real stomach cavity image based on the comparison result and the association relationship comprises: Determine the proportion of each actual stomach cavity part in the actual stomach cavity image; comparing the proportion of each real stomach cavity part in the real stomach cavity image with the proportion of each virtual stomach cavity part in the virtual stomach cavity image; Determine a target stomach cavity image based on the comparison result; the target stomach cavity image is a virtual stomach cavity image that matches the actual stomach cavity image; Based on the spatial posture information associated with the target stomach cavity image, the spatial multi-dimensional data of the actual stomach cavity image is determined.

5. The image annotation method according to claim 4, characterized in that: The spatial pose information includes the capsule global coordinates and capsule shooting information; The determining of the spatial multi-dimensional data of the actual stomach cavity image based on the spatial posture information associated with the target stomach cavity image comprises: In the case where the real stomach cavity image is a plurality of real images that are continuous in time, generating a capsule motion trajectory based on the capsule global coordinates corresponding to each real image; Determining spatial multi-dimensional data of the actual gastric cavity image based on the capsule global coordinates, the capsule shooting information, and the capsule motion trajectory; In the case that the actual gastric cavity image is a single actual image, the spatial multi-dimensional data of the actual gastric cavity image is determined based on the capsule global coordinates and capsule shooting information corresponding to the actual image.

6. The image annotation method according to claim 5, characterized in that: The present stomach cavity image is annotated based on the spatial multi-dimensional data to obtain multi-dimensional image annotation information, and then includes: receiving an output instruction including coordinate information; Based on the coordinate information, determining the spatial multi-dimensional data corresponding to the output instruction; The spatial coordinates, shooting angles and movement trajectories included in the spatial multi-dimensional data corresponding to the output instruction are visualized.

7. An image annotation device, characterized in that: include: A spatial posture information determination module, used to determine the spatial posture information of the virtual capsule endoscope based on the positional relationship between the virtual capsule endoscope and the gastric cavity model; The gastric cavity model is created based on historical gastric cavity data; An association relationship establishing module, used for establishing an association relationship between the virtual stomach cavity image and the spatial posture information; A spatial multi-dimensional data determination module, used for comparing the real stomach cavity image with the virtual stomach cavity image, and determining the spatial multi-dimensional data of the real stomach cavity image based on the comparison result and the association relationship; An image annotation module is used to annotate the actual stomach cavity image based on the spatial multi-dimensional data to obtain multi-dimensional image annotation information.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the image annotation method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image annotation method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image annotation method according to any one of claims 1 to 6 is implemented.

Images