Operation teaching annotation method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention provides an operation teaching annotation method and device, a storage medium and electronic equipment. The method comprises the steps of obtaining a first operation related image in an operation teaching process; the first operation related image is a live broadcast image or a recorded video image; the first operation related image comprises a first annotation, and the first annotation is used for indicating an annotation target in the first operation related image; and based on an association relationship between a first annotation in the first operation-related image and the first target area, identifying the position of an annotation target in the second operation-related image, and establishing display association between the first annotation and the position of the annotation target in the second operation-related image. According to the method and the device, continuous anchoring of the annotation target in the live broadcast or recorded broadcast process is realized, a surgeon does not need to manually erase the annotation track and re-annotate, and the efficiency of operation teaching annotation is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a surgical teaching annotation method, device, storage medium, and electronic device. Background Art

[0002] During the surgical teaching process, users can use the annotation function to annotate the surgical screen, but this function can only be used for static surgical screens. Once the screen moves or the target object moves, the annotation position cannot correctly specify the target object. Therefore, the current annotation function cannot meet the annotation needs of surgical videos, such as surgical recordings and live broadcasts. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present application provides a surgical teaching annotation method, device, storage medium and electronic device, which can improve the efficiency of surgical teaching annotation.

[0004] According to a first aspect of an embodiment of the present application, a surgical teaching annotation method is provided, comprising the following steps:

[0005] Acquire a first surgery-related image during the surgery teaching process; the first surgery-related image is a live image or a recorded video image; the first surgery-related image includes a first batch of annotations, and the first batch of annotations is used to indicate an annotation target in the first surgery-related image;

[0006] Based on the association between the first batch of annotations and the first target area in the first surgery-related image, the location of the annotation target in the second surgery-related image is identified, and the first batch of annotations is associated with the location of the annotation target in the second surgery-related image.

[0007] According to a second aspect of an embodiment of the present application, a surgical teaching annotation device is provided, comprising:

[0008] A first surgery-related image acquisition module is used to acquire a first surgery-related image during the surgical teaching process; the first surgery-related image is a live image or a recorded video image; the first surgery-related image includes a first batch of annotations, and the first batch of annotations is used to indicate an annotation target in the first surgery-related image;

[0009] The display association establishment module is used to identify the location of the annotation target in the second surgery-related image based on the association relationship between the first batch of annotations and the first target area in the first surgery-related image, and establish a display association between the first batch of annotations and the location of the annotation target in the second surgery-related image.

[0010] According to a third aspect of an embodiment of the present application, there is provided an electronic device, comprising a display, a processor and a memory; the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the surgical teaching annotation method as described above.

[0011] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the surgical teaching annotation method as described above is implemented.

[0012] Based on the association between the first batch of annotations and the first target area in the first surgery-related image, the present application performs regional identification of the annotation target in the second surgery-related image, and displays it in association with the first batch of annotations, thereby achieving continuous anchoring of the annotation target during live or recorded broadcasting. The surgeon no longer needs to manually erase the annotation track and re-annotate, thereby improving the efficiency of surgical teaching annotations.

[0013] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

[0014] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flowchart of a surgical teaching annotation method shown in one embodiment of the present application;

[0016] Figure 2 A schematic diagram showing a first batch of injection tracks in a first surgery-related image according to an embodiment of the present application;

[0017] Figure 3 This is a flow chart of step S30 in the surgical teaching annotation method according to one embodiment of the present application;

[0018] Figure 4 This is a flow chart of step S30 in the surgical teaching annotation method according to another embodiment of the present application;

[0019] Figure 5 This is a flow chart of step S20 in the surgical teaching annotation method according to one embodiment of the present application;

[0020] Figure 6 This is a flow chart of step S22 in the surgical teaching annotation method according to one embodiment of the present application;

[0021] Figure 7 This is a flow chart of step S223 in the surgical teaching annotation method according to one embodiment of the present application;

[0022] Figure 8 This is a flowchart of step S2231 in the surgical teaching annotation method according to one embodiment of the present application;

[0023] Fig. 9This is a flowchart of step S2231 in the surgical teaching annotation method according to another embodiment of the present application;

[0024] Fig.10 A schematic diagram showing a first batch of injection tracks in a second surgery-related image according to an embodiment of the present application;

[0025] Fig.11 This is a schematic structural block diagram of a surgical teaching annotation device according to an embodiment of the present application;

[0026] Fig.12 The present invention is a block diagram showing the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0028] It should be clear that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.

[0029] When the following description relates to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. The singular forms of "a", "said" and "the" used in the present application and the appended claims are also intended to include the majority form, unless the context clearly indicates other meanings. The words "if" / "if" used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0030] In addition, in the description of this application, unless otherwise specified, "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0031] The surgical teaching annotation method provided in the embodiment of the present application can be applied to the surgical teaching application device, and can be specifically applied to the live surgery and recorded surgery scenarios. For the recorded surgery environment, the surgical teaching application device includes one or more terminals. For the live surgery environment, the application device includes at least one camera used in the operation in addition to the terminal, and the camera is connected to the terminal by wire or wirelessly. During the surgical teaching process, the position of the camera is kept unchanged to capture the surgical screen. Specifically, the camera can be set directly above the surgical operation area or as a lens on facilities such as a gastroscope and colonoscope. The terminal can be optionally a device with a display screen, such as a conference tablet, a teaching tablet, a medical display, a mobile phone, a pad, and other smart terminals. The display screen is used to display the surgical screen. The user can use input devices such as a mouse and a touch screen to annotate the surgical screen, and it will be displayed on the display screen.

[0032] The surgical teaching annotation method of the embodiment of the present application can be applied to live surgical teaching or recorded teaching scenarios. The inventors found in the process of realizing the present invention that during the live surgical teaching process, the surgeon can broadcast the surgical screen live while performing the operation, and students can learn by watching the surgical screen. During the teaching process, the surgeon can use the annotation function to annotate the surgical screen for the area currently being operated or the position that needs to be emphasized. For example, the organ being operated can be outlined to generate an annotation track, which makes it easier for students to grasp the key points through the annotation track.

[0033] However, during live surgical demonstrations, organs may often be displaced due to camera movement or their own reasons, such as the regular beating of the heart, respiratory movement of the lungs, and peristalsis of the intestines. As a result, the annotation track is no longer accurate, requiring the surgeon to erase the annotation track and re-annotate the displaced organs, which complicates the surgeon's surgical teaching annotation operation and reduces the efficiency of surgical teaching.

[0034] To this end, the present application performs regional identification of the annotation target in the second surgery-related image based on the association between the first batch of annotations and the first target area in the first surgery-related image, and displays it in association with the first batch of annotations, thereby achieving continuous anchoring of the annotation target during live or recorded broadcasting. The surgeon no longer needs to manually erase the annotation track and re-annotate, thereby improving the efficiency of surgical teaching annotations.

[0035] Based on this, the present application proposes a surgical teaching annotation method, device, storage medium and electronic device.

[0036] See also Figure 1 The surgical teaching annotation method provided in the embodiment of the present application includes the following steps:

[0037] S10: Acquire a first surgery-related image in the surgical teaching process; the first surgery-related image is a live image or a recorded video image; the first surgery-related image includes a first batch of annotations, and the first batch of annotations are used to indicate annotation targets in the first surgery-related image;

[0038] S20: Based on the association between the first batch of annotations and the first target area in the first surgery-related image, identify the location of the annotation target in the second surgery-related image, and establish a display association between the first batch of annotations and the location of the annotation target in the second surgery-related image.

[0039] The first surgery-related image and the second surgery-related image may be selected as video frames or video clips composed of multiple video frames. In the case of live surgery, the first surgery-related image and the second surgery-related image are frames or live clips of continuous live broadcast; in the case of recorded surgery, the second surgery-related image is a frame or video clip adjacent to the first surgery-related image (the second surgery-related image may be the previous frame or previous video of the first surgery-related image, or may be the next frame or next video of the first surgery-related image).

[0040] The annotation target may be an organ, tissue or other target that can be selected during surgery. The annotation operation includes but is not limited to the annotation operation performed by the surgeon on the live video screen of the surgical teaching application device using a mouse, hand or touch device. The first batch of annotations includes the first batch of annotation tracks and annotation information. The first batch of annotation tracks includes but is not limited to graphics and text. The annotation information can be used for but is not limited to the explanation and description of the annotation target.

[0041] The first target area is used to indicate the location area of ​​the annotation target in the first surgery-related image.

[0042] In the embodiment of the present application, the first surgery-related image is the current frame of the live surgery video screen, and the second surgery-related image is the live surgery video screen of each frame after the current frame. During the surgical teaching process, the surgeon can annotate the annotation target in the current frame of the live surgery video screen according to teaching needs to form the first batch of annotations, which are used to indicate the position of the annotation target in the current frame of the live surgery video screen.

[0043] See also Figure 2The display screen of the surgical teaching application device displays the current frame of the live surgical video. The surgeon turns on the annotation mode on the display screen, and the display screen enters the annotation state. In the annotation state, the surgeon can annotate the annotation target 11 (organs and tissues in the example surgical teaching process) in the current frame of the live surgical video (the first surgical related image) 1, for example, to draw a graphic outline of a certain organ. The surgical teaching application device detects the annotation operation and generates the first batch of annotation tracks 12 corresponding to the annotation operation. For example, the first batch of annotation tracks can be irregular closed graphics.

[0044] In an embodiment of the present application, based on the association between the first batch of annotations and the first target area in the current frame of live surgical video, regional identification is performed on the annotation targets in each frame of live surgical video after the current frame, and the annotation targets are displayed in association with the first batch of annotations, thereby achieving continuous anchoring of the annotation targets during the live surgical video.

[0045] It should be noted that the method of the embodiment of the present application can also be applied to the playback process of recorded surgical videos.

[0046] In an optional embodiment, before the step of determining the association relationship between the first batch of annotations in the first surgery-related image and the first target area, steps S1 to S3 are included, which are specifically as follows:

[0047] S1: In response to a positioning operation on an annotated target in a first surgery-related image, a first target region of the first surgery-related image is determined.

[0048] Among them, at least part of the information of the first batch of annotations is used to indicate the position of the annotation target in the current frame of the live surgery video, and the area where the annotation target is located is used as the first target area, and the first target area at least covers part of the annotation target.

[0049] S3: Associating the first batch of annotations with the first target area.

[0050] The first batch of annotations may be associated with the first target area in such a way that the first batch of annotations are at least partially located in or around the first target area. The embodiment of the present application may also use an indicator line or other indicator method to realize the display association between the first batch of annotations and the first target area, so that the participants can understand the association between the first batch of annotations and the first target area.

[0051] The embodiment of the present application can achieve association with the first target area through the first batch of injections.

[0052] In an optional embodiment, the first batch of annotations at least includes a first batch of annotation tracks, and the first batch of annotation tracks are used to indicate the annotation target; step S3 includes step S30, which is specifically as follows:

[0053] S30: Determine a first target area in the first surgery-related image according to the position of the first injection trajectory; the first injection trajectory is a closed or semi-closed figure surrounding the first target area.

[0054] The first batch of annotation tracks includes a plurality of track points, and the plurality of track points are connected in sequence according to the order of drawing time to form the first batch of annotation tracks. The position of the first batch of annotation tracks includes the position coordinates of the plurality of track points.

[0055] The first target area may be an area including the first batch of injection tracks, and the first batch of injection tracks are located in the first target area. The first target area may also be an area surrounded by the first batch of injection tracks. The first target area includes but is not limited to a rectangular area, a triangular area, a circular area, and a diamond area.

[0056] In the embodiment of the present application, the surgical teaching application device can determine the first target area in the current frame of the live surgical video screen according to the position coordinates of the multiple track points that constitute the first batch of injection tracks. Specifically, a regular closed figure can be fitted according to the position coordinates of the multiple track points that constitute the first batch of injection tracks, and the regular closed figure can be enlarged according to a certain ratio, and the area covered by the enlarged closed figure is used as the first target area.

[0057] In an alternative embodiment, see Figure 3 , step S30, including steps S301 to S303, are as follows:

[0058] S301: Obtaining the position coordinates of each track point constituting the first batch of injection tracks;

[0059] S302: Determine the centroid of the first batch of injection trajectories according to the position coordinates of each trajectory point.

[0060] In the embodiment of the present application, the position coordinates of the centroid of the first batch of injection trajectories may be the average of the position coordinates of each trajectory point. Specifically, the abscissa of the centroid is the average of the abscissas of all trajectory points, and the ordinate of the centroid is the average of the ordinates of all trajectory points.

[0061] S303: Generate a first target area in the first surgery-related image with the centroid as the center and according to a preset size.

[0062] In the embodiment of the present application, the surgical teaching application device can generate a circular area with the centroid as the center of the circle and a preset radius, or can generate a rectangular area with the centroid as the geometric center and a preset width and a preset height.

[0063] By determining the centroid of the first batch of injection trajectories, the first target area in the current frame of live surgery video can be automatically and quickly generated.

[0064] In an alternative embodiment, see Figure 4 , step S30, including steps S304 to S305, are as follows:

[0065] S304: Obtain the position coordinates of each edge trajectory point of the first batch of annotation trajectories; wherein the value of the horizontal coordinate in the position coordinates of each edge trajectory point is the maximum or minimum coordinate value in the same row of the position coordinates of all the trajectory points constituting the annotation trajectory, and the value of the vertical coordinate in the position coordinates of each edge trajectory point is the maximum or minimum coordinate value in the same column of the position coordinates of all the trajectory points constituting the annotation trajectory.

[0066] In an embodiment of the present application, the surgical teaching application device can obtain the position coordinates of each edge trajectory point of the first batch of injection trajectories through an edge point detection algorithm.

[0067] S305: Performing region fitting on the position coordinates of each edge trajectory point to obtain a first target region in the first surgery-related image.

[0068] In an embodiment of the present application, the surgical teaching application device can use a polygon fitting algorithm to perform regional fitting on the position coordinates of each edge trajectory point to obtain a regular polygonal area, and enlarge the regular polygonal area according to a certain proportion to obtain the first target area in the current frame of the live surgical video.

[0069] By determining the edge track points of the first batch of injection tracks, the first target area in the current frame of live surgery video can be automatically and quickly generated.

[0070] In an alternative embodiment, see Figure 5 , step S20, including steps S21 to S23, are as follows:

[0071] S21: Identify characteristic image information of a first target area in a first surgery-related image.

[0072] In the embodiment of the present application, the characteristic image information may be the first pixel value of each first pixel point in the first target area, and the first pixel value includes but is not limited to a gray value, a chromaticity value, and a saturation value.

[0073] S22: Identify the location of the characteristic image information in the second surgery-related image and determine the second target area.

[0074] In the embodiment of the present application, the pixel values ​​of each pixel point in the second surgery-related image can be traversed to find the pixel points consistent with the characteristic image information, and the area formed by the pixel points can be used as the second target area.

[0075] S23: Establishing a display association between the first batch of annotations and a second target area in the second surgery-related image.

[0076] The first batch of annotations may be associated with the first target area in such a way that the first batch of annotations are at least partially located in or around the first target area. The embodiment of the present application may also use an indicator line or other indicator method to realize the display association of the first batch of annotations with the first target area, so that the participants can understand the association between the first batch of annotations and the first target area.

[0077] In the embodiment of the present application, after the second target area is determined, the first batch of annotations can be displayed around the second target area according to the relative positional relationship between the first batch of annotations and the first target area.

[0078] The second target area can be determined through the characteristic image information of the first target area, so as to establish a display association between the first batch of annotations and the second target area.

[0079] In an alternative embodiment, see Figure 6 Step S22 includes steps S221 to S223, which are as follows:

[0080] S221: Compare the first image information of the first target area in the first surgery-related image with the second image information of the preset area in the second surgery-related image; wherein the position information of the preset area is the same as that of the first target area; and the first image information is feature image information.

[0081] The preset area includes a plurality of second pixel points, and the second image information may be a second pixel value of the second pixel point, and the second pixel value includes but is not limited to a gray value, a chromaticity value, and a saturation value.

[0082] The position information of the first target area refers to the position coordinates of the first target area relative to the first reference point. The first reference point may be the coordinate origin of a plane rectangular coordinate system established in the current frame of the live surgery video.

[0083] The position information of the preset area refers to the position coordinates of the preset area relative to the second reference point. The second reference point can be the coordinate origin of the plane rectangular coordinate system established in each frame of the live surgery video after the current frame.

[0084] The same position information as the first target area means that the position coordinates of the first target area relative to the first reference point are the same as the position coordinates of the preset area relative to the second reference point, and the first reference point and the second reference point are the same reference point. The preset area and the first target area have the same area size, the number of second pixels in the preset area is the same as the number of first pixels in the first target area, and for any first pixel in the first target area, there is a unique second pixel corresponding to it in the preset area.

[0085] In the embodiment of the present application, the first pixel value of each first pixel point in the first target area can be compared with the second pixel value of the second pixel point corresponding to the preset area to determine the difference between the pixel values. The average value of the first pixel values ​​of all the first pixel points in the first target area can also be compared with the average value of the second pixel values ​​of all the second pixel points in the preset area to determine the difference between the average values ​​of the pixel values.

[0086] S222: If the first image information matches the second image information, determining the preset area as the second target area.

[0087] Taking the case of displacement as an example, since the annotation target will cover a certain area in the live surgery video, which includes a certain number of pixels, there will be a deviation between the first image information of the first target area in the live surgery video of the current frame before the annotation target is displaced or resized and the second image information of the preset area in the live surgery video of the frame after the annotation target is displaced or resized.

[0088] In the embodiment of the present application, the first image information matches the second image information, which means that the difference between the pixel values ​​or the average values ​​of the pixel values ​​included in the first image information and the second image information is within a preset threshold range. If the first image information matches the second image information, it can be determined that the annotation target of the frame of live surgery video screen has not shifted in position and / or changed in size relative to the current frame of live surgery video screen, and there is no need to update the position of the first batch of annotation tracks. The preset area of ​​the frame of live surgery video screen is the second target area.

[0089] S223: If the first image information does not match the second image information, and it is determined that the annotation target has a position shift and / or size change, the position and / or size of the preset area is adjusted until the second image information of the adjusted preset area matches the first image information of the first target area, and the adjusted preset area is determined to be the second target area.

[0090] In an embodiment of the present application, the mismatch between the first image information and the second image information means that the difference between the pixel values ​​included in the first image information and the second image information or the difference between the average values ​​of the pixel values ​​is not within a preset threshold range. If the first image information and the second image information do not match, it is determined that the annotation target of this frame of live surgical video screen is offset in position and / or changes in size relative to the current frame of live surgical video screen, and the position of the first batch of annotation tracks needs to be updated. Specifically, the position of the preset area is moved, and the second image information of the preset area will continue to change until the difference between the pixel values ​​included in the second image information of the preset area and the first image information of the first target area or the difference between the average values ​​of the pixel values ​​is within a preset threshold range.

[0091] By comparing the first image information with the second image information, it can be determined whether the annotation target has positional displacement and / or size change. When the annotation target has positional displacement and / or size change, the second target area can be determined.

[0092] In an optional embodiment, the first image information includes a first pixel value, and the second image information includes a second pixel value, see Figure 7 , step S223, including steps S2231 to S2232, are as follows:

[0093] S2231: If the deviation between the second pixel value and the first pixel value is not within a preset threshold range, determine a second surgery-related image when the annotation target undergoes positional displacement and / or size change.

[0094] If the deviation between the second pixel value and the first pixel value is not within the preset threshold range, it is determined that the annotation target of the second surgery-related image is offset in position and / or changed in size relative to the first surgery-related image, and the position of the first annotation track needs to be updated.

[0095] In the embodiment of the present application, the deviation between the second pixel value and the first pixel value includes the difference between the second pixel value and the first pixel value or the difference between the average value of the second pixel value and the average value of the first pixel value.

[0096] S2232: Adjust the position and / or size of the preset area in the second surgery-related image when the annotation target undergoes position shift and / or size change, until the deviation between the second pixel value of the preset area in the second surgery-related image when the annotation target undergoes position shift and / or size change and the first pixel value of the first target area is within a preset threshold range, and determine that the adjusted preset area is the second target area.

[0097] In an embodiment of the present application, a preset area can be moved in the live surgical video frame of the frame where the annotation target is positionally offset, and the deviation between the second pixel value of the preset area after the move and the first pixel value of the first target area is calculated. If the deviation is not within a preset threshold range, the preset area is continued to be moved until the deviation between the second pixel value of the preset area in the live surgical video frame where the annotation target is positionally offset and the first pixel value of the first target area is within the preset threshold range, and the preset area after the move is determined to be the second target area.

[0098] The size of the preset area can also be adjusted, and the deviation between the second pixel value of the preset area after the size adjustment and the first pixel value of the first target area is calculated. If the deviation is not within the preset threshold range, the size of the preset area is continued to be adjusted until the deviation between the second pixel value of the preset area in the live surgical video screen of the frame where the annotation target is positionally offset and the first pixel value of the first target area is within the preset threshold range, and the preset area after the size adjustment is determined to be the second target area.

[0099] By moving the position of the preset area and / or adjusting the size of the preset area in the live surgery video frame where the annotation target is offset, the second target area can be automatically and quickly obtained.

[0100] In an alternative embodiment, see Figure 8 , step S2231, including steps S22311 to S22312, are as follows:

[0101] S22311: Calculate a first average value of all first pixel points in the first target area and a second average value of all second pixel points in a preset area of ​​the second surgery-related image;

[0102] S22312: Determine the second surgery-related image corresponding to a deviation between the second average value and the first average value that is greater than a first preset threshold as the second surgery-related image when the annotation target undergoes positional displacement and / or size change.

[0103] The first target area includes a plurality of first pixels, and the preset area includes a plurality of second pixels. The first average value is an average value obtained by averaging the first pixel values ​​of all first pixels in the first target area, and the second average value is an average value obtained by averaging the second pixel values ​​of all second pixels in the preset area.

[0104] In the embodiment of the present application, if the annotation target does not shift position and / or change size, the first average value and the second average value have no deviation. If the annotation target shifts position and / or changes size, the first average value and the second average value have deviation.

[0105] By calculating the deviation between the second average value and the first average value and comparing the deviation with the first preset threshold, it is possible to automatically and quickly determine whether the annotation target has positional displacement and / or size change, and obtain the live surgical video frame in which the annotation target has positional displacement and / or size change.

[0106] In an alternative embodiment, see Fig. 9 , step S2231, including steps S22313 to S22314, are as follows:

[0107] S22313: Calculate the difference between the second pixel value of each second pixel point in the preset area of ​​the second surgery-related image and the first pixel value of the corresponding first pixel point in the first target area;

[0108] S22314: Determine the second surgery-related image corresponding to a difference greater than a second preset threshold and a number of second pixels greater than a preset value as the second surgery-related image when the annotation target undergoes positional displacement and / or size change.

[0109] In the embodiment of the present application, if the annotation target does not shift in position and / or change in size, the second pixel value of the second pixel point and the first pixel value of the corresponding first pixel point are not biased. If the annotation target shifts in position and / or changes in size, the second pixel value of the second pixel point and the first pixel value of the corresponding first pixel point are biased.

[0110] Since there may be noise in obtaining the pixel value of a single pixel, when the annotation target does not shift in position and / or change in size, there may be a deviation between the second pixel value of one or several second pixels and the first pixel value of the corresponding first pixel, and the second pixel value of most second pixels and the first pixel value of the corresponding first pixel do not deviate. Therefore, on the basis that the deviation between the second pixel value of the second pixel and the first pixel value of the corresponding first pixel is greater than the second preset threshold, the number of second pixels with a difference greater than the second preset threshold is counted, and it is required that the number of second pixels with a difference greater than the second preset threshold is greater than the preset value before it is determined that the annotation target has shifted in position and / or changed in size.

[0111] By calculating the deviation between the second pixel value and the first pixel value, comparing the deviation with the second preset threshold, and counting the number of second pixel points whose difference is greater than the second preset threshold, and comparing the number of second pixel points whose difference is greater than the second preset threshold with the preset value, it is possible to automatically and quickly determine whether the annotation target has positional displacement and / or size change, and obtain the live surgical video screen of the frame where the annotation target has positional displacement and / or size change.

[0112] In an optional embodiment, step S2232 includes step S22321, which is specifically as follows:

[0113] S22321: Control the preset area in the second surgery-related image to move a preset distance and / or change size along a preset direction when the annotation target is positionally offset and / or changes size, so that the deviations between the second pixel values ​​of each second pixel point in the preset area after movement and / or size change and the first pixel value of the corresponding first pixel point in the first target area are within a preset threshold range.

[0114] The preset direction includes but is not limited to the row direction and column direction of the pixels in the live surgery video screen, and the preset distance includes but is not limited to the distance between the preset number of pixels.

[0115] In an embodiment of the present application, a preset area in the live surgical video frame in which the annotation target is located can be controlled to be translated one pixel to the right along the row direction when the position shift and / or size change occurs, and the deviation between the second pixel value of each second pixel point in the preset area after the movement and the first pixel value of the corresponding first pixel point in the first target area is calculated. If the deviations are all within the preset threshold range, the movement of the preset area is stopped.

[0116] If there is a deviation between the second pixel value of one or more second pixel points and the first pixel value of the corresponding first pixel point that is not within the preset threshold range, the preset area in the live surgery video screen of the frame where the annotation target is controlled to shift in position and / or change in size is moved to the initial position, and translated one pixel to the left in the row direction, or translated one pixel upward in the column direction, or translated one pixel downward in the column direction, and the above steps are repeated until the deviations between the second pixel values ​​of each second pixel point in the preset area after the move and the first pixel value of the corresponding first pixel point in the first target area are all within the preset threshold range.

[0117] By controlling a preset area in a frame of live surgery video where the annotation target is located to move a preset distance along a preset direction, the position offset of the preset area and the second target area can be automatically and quickly obtained.

[0118] In an optional embodiment, the step of identifying the location of the annotation target in the second surgery-related image in step S20 and establishing a display association between the first annotation and the location of the annotation target in the second surgery-related image includes steps S24 to S25, which are specifically as follows:

[0119] S24: in the second surgery-related image when the annotation target is shifted in position and / or changed in size, obtaining the annotation target position according to the displacement offset and / or size change amount of the preset area and the position of the first annotation track.

[0120] In an embodiment of the present application, taking the offset situation as an example, the position of the first batch of injection trajectories includes the position coordinates of each trajectory point constituting the first batch of injection trajectories, the displacement offset of the preset area is used as the displacement offset of each trajectory point constituting the first batch of injection trajectories, and each trajectory point constituting the first batch of injection trajectories is translated by the displacement offset to obtain the target position of each trajectory point constituting the first batch of injection trajectories.

[0121] S25: moving and / or adjusting the size of the first batch of annotation tracks to the annotation target position for display.

[0122] In an embodiment of the present application, a first batch of annotation tracks are displayed in the live surgical video frame where the annotation target is offset in position and / or changes in size. After the target position is calculated, the first batch of annotation tracks are moved to the target position for display.

[0123] Please refer to Fig.10 The surgical teaching application device can use the displacement offset of the preset area in the live surgical video screen (second surgical related image) 2 of the frame where the annotation target 11 is offset as the displacement offset of the first batch of annotation tracks 12. Knowing the initial position of the first batch of annotation tracks 12, the target position of the first batch of annotation tracks 12 can be determined. The first batch of annotation tracks 12 can be moved to the target position, or the first batch of annotation tracks 12 can be cleared and redrawn at the target position for display.

[0124] By moving the first batch of annotation tracks to the target position, the annotation tracks follow the annotation target, and the surgeon no longer needs to manually erase the annotation tracks and re-annotate, thereby improving the efficiency of surgical teaching annotation.

[0125] It should be noted that the shapes of the first batch of injection tracks after the movement and the first batch of injection tracks before the movement can be selected to be the same or different.

[0126] See also Fig.11 , a surgical teaching annotation device 4 disclosed in an embodiment of the present application includes:

[0127] A first surgery-related image acquisition module 41 is used to acquire a first surgery-related image during the surgical teaching process; the first surgery-related image is a live image or a recorded video image; the first surgery-related image includes a first batch of annotations, and the first batch of annotations is used to indicate the annotation targets in the first surgery-related image;

[0128] The display association establishment module 42 is used to identify the location of the annotation target in the second surgery-related image based on the association relationship between the first batch of annotations and the first target area in the first surgery-related image, and establish a display association between the first batch of annotations and the location of the annotation target in the second surgery-related image.

[0129] The surgical teaching annotation device provided in the above embodiment and the surgical teaching annotation method provided in the embodiment of the present application belong to the same concept. For details on the implementation process, please refer to steps S10 to S20 of the method embodiment, which will not be repeated here.

[0130] The present application also provides a device embodiment that can be used to execute the contents of the surgical teaching annotation method in the embodiment of the present application. For details not disclosed in the device embodiment of the present application, please refer to the contents of the surgical teaching annotation method in the embodiment of the present application.

[0131] See also Fig.12 The present application also provides an electronic device 300, which can be a computer, a surgical teaching application device, etc. In an exemplary embodiment of the present application, the electronic device 300 is a surgical teaching application device, which includes: at least one processor 301, at least one memory 302, at least one display, at least one network interface 303, a user interface 304, and at least one communication bus 305.

[0132] The user interface 304 is mainly used to provide an input interface for the user and obtain data input by the user. Optionally, the user interface may also include a standard wired interface or a wireless interface.

[0133] The network interface 303 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0134] The communication bus 305 is used to realize the connection and communication between these components.

[0135] Among them, the processor 301 may include one or more processing cores. The processor uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor can integrate one or more combinations of central processing unit (Central Processing Unit, CPU), image processor (Graphics Processing Unit, GPU) and modem. Among them, the CPU mainly processes the operating system, user interface and application program, etc.; the GPU is responsible for rendering and drawing the content to be displayed by the display layer; the modem is used to process wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented by a single chip.

[0136] Among them, the memory 302 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area may store data involved in the above-mentioned method embodiments, etc. The memory may also be optionally at least one storage device located away from the aforementioned processor. As Fig.12 As shown, the memory as a computer storage medium may include an operating system, a network communication module, a user interface module, and an operating application program.

[0137] The processor can be used to call the application of the surgical teaching annotation method of the surgical teaching application device stored in the memory, and specifically execute the method steps of the above-mentioned embodiment. The specific execution process can refer to the specific description shown in the method embodiment, which will not be repeated here.

[0138] The present application also provides a computer-readable storage medium on which a computer program is stored, and the instructions are suitable for being loaded by a processor and executing the method steps of the above-mentioned embodiment. The specific execution process can refer to the specific description shown in the embodiment, and will not be repeated here. The device where the storage medium is located can be a personal computer, a laptop computer, a smart phone, a tablet computer, a picture director device and other electronic devices.

[0139] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0140] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0141] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the function selected in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 function selected in a box or multiple boxes.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 steps for the function selected in a box or multiple boxes.

[0143] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0144] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0145] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0146] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0147] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A surgical teaching annotation method, characterized in that: The steps include: Acquire a first surgery-related image during the surgical teaching process; the first surgery-related image is a live image or a recorded video image; the first surgery-related image includes a first batch of annotations, and the first batch of annotations is used to indicate annotation targets in the first surgery-related image; Based on the association between the first batch of annotations and the first target area in the first surgery-related image, the location of the annotation target in the second surgery-related image is identified, and the first batch of annotations is associated with the location of the annotation target in the second surgery-related image.

2. The surgical teaching annotation method according to claim 1, characterized in that: Before the step of associating the first batch of annotations with the first target area in the first surgery-related image, the method includes: In response to a positioning operation on the annotated target in the first surgery-related image, determining a first target area of ​​the first surgery-related image; The first batch of annotations is associated with the first target area.

3. The surgical teaching annotation method according to claim 2, characterized in that: The first batch of annotations at least includes a first batch of annotation tracks, and the first batch of annotation tracks are used to indicate the annotation target; The step of determining a first target area of ​​the first surgery-related image in response to a positioning operation of the annotated target in the first surgery-related image comprises: determining a first target area in the first surgery-related image according to the position of the first batch of injection tracks; The first batch of injection tracks is a closed or semi-closed figure surrounding the first target area.

4. The surgical teaching annotation method according to any one of claims 1 to 3, characterized in that: The step of identifying the location of the annotation target in the second surgery-related image based on the association relationship between the first batch of annotations and the first target area in the first surgery-related image, and establishing a display association between the first batch of annotations and the location of the annotation target in the second surgery-related image includes: Identifying characteristic image information of the first target area in the first surgery-related image; Identify the location of the characteristic image information in the second surgery-related image, and determine a second target area; The first batch of annotations is associated with the second target area in the second surgery-related image for display.

5. The surgical teaching annotation method according to claim 4, characterized in that: The step of identifying the location of the characteristic image information in the second surgery-related image and determining the second target area includes: Compare the first image information of the first target area in the first surgery-related image with the second image information of the preset area in the second surgery-related image; wherein the preset area has the same position information as the first target area; and the first image information is the characteristic image information; If the first image information matches the second image information, determining the preset area as the second target area; If the first image information does not match the second image information, and it is determined that the annotation target has undergone position shift and / or size change, the position and / or size of the preset area is adjusted until the adjusted second image information of the preset area matches the first image information of the first target area, and the adjusted preset area is determined to be the second target area.

6. The surgical teaching annotation method according to claim 5, characterized in that: The first image information includes a first pixel value, and the second image information includes a second pixel value; If the first image information does not match the second image information, it is determined that the annotation target has a position shift and / or a size change, then the position and / or size of the preset area is adjusted until the adjusted second image information of the preset area matches the first image information of the first target area, and the step of determining that the adjusted preset area is the second target area includes: If the deviation between the second pixel value and the first pixel value is not within a preset threshold range, determining a second surgery-related image when the annotation target undergoes positional displacement and / or size change; Adjust the position and / or size of the preset area in the second surgery-related image when the annotation target undergoes position shift and / or size change, until a deviation between a second pixel value of the preset area in the second surgery-related image when the annotation target undergoes position shift and / or size change and a first pixel value of the first target area is within a preset threshold range, and determine that the adjusted preset area is the second target area.

7. The surgical teaching annotation method according to claim 6, characterized in that: The step of determining the second surgery-related image when the annotation target has positional displacement and / or size change if the deviation between the second pixel value and the first pixel value is not within a preset threshold range comprises: Calculating a first average value of all first pixels in the first target area and a second average value of all second pixels in a preset area of ​​the second surgery-related image; The second surgery-related image corresponding to the second average value having a deviation from the first average value greater than a first preset threshold is determined as the second surgery-related image when the annotation target undergoes positional displacement and / or size change.

8. The surgical teaching annotation method according to claim 6, characterized in that: The step of determining the second surgery-related image when the annotation target undergoes positional displacement and / or size change if the deviation between the second pixel value and the first pixel value is within a preset threshold range comprises: Calculate the difference between the second pixel value of each second pixel point in the preset area of ​​the second surgery-related image and the first pixel value of the corresponding first pixel point in the first target area; The second surgery-related image corresponding to the difference being greater than the second preset threshold and the number of second pixels being greater than the preset value is determined as the second surgery-related image when the annotation target undergoes positional displacement and / or size change.

9. The surgical teaching annotation method according to any one of claims 7 or 8, characterized in that: The step of adjusting the position and / or size of the preset area in the second surgery-related image when the annotation target is positionally shifted and / or changed in size, until a deviation between a second pixel value of the preset area in the second surgery-related image when the annotation target is positionally shifted and / or changed in size and a first pixel value of the first target area is within a preset threshold range, and determining that the adjusted preset area is the second target area, comprises: When the annotation target is positionally shifted and / or changes in size, the preset area in the second surgery-related image is controlled to move a preset distance along a preset direction and / or change in size, so that the deviations between the second pixel values ​​of each of the second pixel points in the preset area after the movement and / or size change and the first pixel values ​​of the corresponding first pixel points in the first target area are both within the preset threshold range.

10. The surgical teaching annotation method according to any one of claims 5 to 8, characterized in that: The identifying the location of the annotation target in the second surgery-related image and establishing a display association between the first annotation and the location of the annotation target in the second surgery-related image includes: In the second surgery-related image when the annotation target undergoes positional displacement and / or size change, obtaining the annotation target position according to the displacement offset and / or size change amount of the preset area and the position of the first batch of annotation tracks; The first batch of annotation tracks are moved and / or resized to the annotation target position for display.

11. The surgical teaching annotation method according to claim 2, characterized in that: Determining a first target area in the first surgery-related image according to the position of the first batch of injection tracks includes: Obtaining the position coordinates of each trajectory point constituting the first batch of injection trajectories; Determining the centroid of the first batch of injection trajectories according to the position coordinates of each trajectory point; A first target area in the first surgery-related image is generated based on a preset size and with the centroid as the center.

12. The surgical teaching annotation method according to claim 2, characterized in that: Determining a first target area in the first surgery-related image according to the position of the first batch of injection tracks includes: Obtaining the position coordinates of each edge track point of the first batch of annotation tracks; wherein the value of the horizontal coordinate in the position coordinates of each edge track point is the maximum or minimum coordinate value in the same row of the position coordinates of all track points constituting the annotation track, and the value of the vertical coordinate in the position coordinates of each edge track point is the maximum or minimum coordinate value in the same column of the position coordinates of all track points constituting the annotation track; Performing regional fitting on the position coordinates of each edge trajectory point to obtain a first target region in the first surgery-related image.

13. A surgical teaching annotation device, characterized in that: include: A first surgery-related image acquisition module, used to acquire a first surgery-related image during the surgery teaching process; The first surgery-related image is a live image or a recorded video image; The first surgery-related image includes a first batch of annotations, and the first batch of annotations are used to indicate annotation targets in the first surgery-related image; A display association establishing module is used to identify the location of the annotation target in the second surgery-related image based on the association relationship between the first batch of annotations and the first target area in the first surgery-related image, and establish a display association between the first batch of annotations and the location of the annotation target in the second surgery-related image.

14. An electronic device comprising a display, a processor and a memory; characterized in that: The memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the surgical teaching annotation method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the surgical teaching annotation method according to any one of claims 1 to 12 is implemented.

Images