Robotic laparoscope based anomaly recognition method and system

Abstract

The application provides a robot laparoscope-based anomaly recognition method and system, relates to the technical field of laparoscope data processing, and aims to recognize short-time anomalies. The method steps include: collecting video data of a working area by a laparoscope; obtaining extracted features of each video frame, recognizing changed extracted features in adjacent video frames, recording the changed extracted features and corresponding feature values and change amounts; determining whether the extracted features are abnormal by comparing feature thresholds and feature values of the extracted features; obtaining change amounts of the extracted features in a preset time period, and determining change rates of the extracted features in the preset time period; determining whether the change of the extracted features is abnormal by comparing change rate thresholds and change rates of the extracted features in the preset time period; and generating a solution strategy based on the abnormal extracted features, feature values and change rates. By focusing on the change of features in adjacent video frames, subtle changes in a short time are found in time, and short-time anomaly recognition is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of laparoscope data processing, in particular to an abnormality recognition method and system based on a robotic laparoscope. BACKGROUND

[0002] During laparoscopic operation, special instruments are usually used in cooperation. The laparoscope collects dynamic video images in the abdominal cavity in real time, identifies key structures and lesion areas based on image analysis technology, and formulates operation strategies accordingly; the operator completes the corresponding operation in cooperation with the instruments according to the real-time feedback of the strategy information. When analyzing and identifying the laparoscopic operation process, in order to reduce the risk of postoperative complications, it is necessary to realize the timely detection and processing of abnormal conditions.

[0003] At present, the recognition of laparoscopic operation abnormalities mainly relies on the analysis of static images, which is difficult to capture the dynamic correlation between image sequences, resulting in insufficient perception of dynamic changes and inability to effectively detect short-term abnormalities (such as temporary tissue ischemia, local slight bleeding or instantaneous instrument misoperation). When short-term abnormalities occur, if measures are not taken in time, the abnormal state may continue to accumulate or worsen, thereby causing postoperative complications such as infection, adhesion and bleeding. SUMMARY

[0004] The purpose of the present application is to provide an abnormality recognition method and system based on a robotic laparoscope, and the technical problem to be solved is how to recognize short-term abnormalities.

[0005] The present application is realized by the following technical solutions:

[0006] The first aspect provides an abnormality recognition method based on a robotic laparoscope, comprising the following steps:

[0007] A working area is preset, and video data of the working area is collected by a laparoscope;

[0008] Based on the above-mentioned video data, the extracted features of each video frame are obtained, the changed extracted features in adjacent video frames are identified, and the changed extracted features and the corresponding feature values and change amounts are recorded;

[0009] By comparing the feature threshold value and the feature value of the above-mentioned extracted features, it is determined whether the extracted features are abnormal; if the above-mentioned extracted features are abnormal, the extracted features and the corresponding feature values are stored in an abnormality database;

[0010] The change amount of the above-mentioned extracted features in a preset period is obtained, and the change rate of the extracted features in the preset period is determined;

[0011] By comparing the change rate threshold value and the change rate of the above-mentioned extracted features in the preset period, it is determined whether the change of the extracted features is abnormal; if the change of the above-mentioned extracted features is abnormal, the extracted features and the corresponding change rate are stored in an abnormality database;

[0012] A solution strategy is generated based on the extracted features, feature values and change rates in the above abnormality database.

[0013] Since the collection range of the laparoscope is limited, the complete video collection of the working area requires time, and during the collection, the laparoscope gradually moves, and while new extraction features are extracted, the extraction features also gradually disappear in the subsequent video frames. By focusing on the changes of the features in adjacent video frames, subtle changes in a short period of time are discovered in time; by setting a feature threshold and comparing it with the actual feature value, it is quantitatively judged whether the extraction feature is in an abnormal state, realizing static abnormality recognition; in addition, the change rate of the extraction feature in a preset period of time is also considered, reflecting the dynamic change of the extraction feature in a period of time; for some dynamic changes, there are requirements for the change of the extraction feature in a short period of time, even if the analysis of each frame meets the static requirements, there will also be cases that do not meet the dynamic change requirements, therefore, by comparing the preset change rate threshold with the actual change rate, the change of the extraction feature is further judged from the perspective of dynamic change whether it is abnormal, realizing short-time abnormality recognition.

[0014] Further, the above-mentioned extraction features that change in adjacent video frames are identified, and the changed extraction features and the corresponding feature values and change amounts are recorded, and the specific steps include:

[0015] The video frames at adjacent time points are acquired, the video frame at the previous time point in the adjacent time video frames is taken as a background image, and the video frame at the next time point is taken as a processing image;

[0016] Based on the above-mentioned background image, the background area of the processing image is removed to obtain a change area;

[0017] The extraction features and feature values are obtained from the above-mentioned change area;

[0018] After associating the above-mentioned extraction features, corresponding feature values and video frame time stamps, the association data is obtained; the above-mentioned association data is stored in a record database;

[0019] The feature value of the extraction feature in the background image is called from the above-mentioned record database, the difference between the feature values of the extraction feature in the processing image and the background image is determined, and the change amount of the extraction feature is obtained.

[0020] The video frame rate can be adjusted according to the dynamic change of adjacent video frames; the previous frame is taken as a background, and the next frame is taken as a processing image, the background region is removed, and the part different from the background in the processing image is highlighted, which reduces the interference of irrelevant information, focuses the analysis on the area actually changed, and improves the efficiency of short-time anomaly detection; the features and feature values in the changed area are extracted to obtain information related to short-time anomaly. Extracting features from the changed area can avoid the influence of irrelevant area features, improve the pertinence of feature extraction, and help identify short-time anomalies.

[0021] Further, if the extracted feature is retrieved from the background image, the extracted feature is marked as a changed feature; and the difference between the feature values of the changed feature in the processing image and the background image is taken as the change amount of the extracted feature.

[0022] If the extracted feature is not retrieved from the background image, the extracted feature is marked as a new feature; and the feature value of the new feature is taken as the change amount of the extracted feature.

[0023] The extracted feature that can be retrieved from the background image is marked as a changed feature, which indicates that the feature has changed based on the original feature and focuses on the feature that has changed in a short time. For example, the color and texture of the tissue may change slightly during transient ischemia. By marking the feature as a changed feature, the feature that may have an abnormality is located, and the pertinence of short-time anomaly detection is improved. The difference between the feature values of the changed feature in the processing image and the background image is taken as the change amount, which quantifies the degree of change of the feature. For short-time anomalies, the severity and development trend of the anomaly are quantitatively judged. For example, when there is a slight bleeding in a local area, the color feature value of the bleeding area will change. By calculating the change amount, it can be known whether the bleeding is getting worse, which provides a basis for taking timely measures.

[0024] The extracted feature that cannot be retrieved from the background image is marked as a new feature, which can timely discover new features appearing in a short time (i.e. features appearing when the laparoscope moves to different areas). In laparoscopic operations, new features such as new indentations may be generated at the contact site of the instrument. By marking the feature as a new feature, unexpected situations can be quickly detected, and more serious consequences can be avoided due to the lack of timely discovery. For the new feature, it is a change information itself, and the new feature may indicate the occurrence of an anomaly in short-time anomaly recognition. Taking the feature value of the new feature as the change amount helps to timely discover potential short-time anomalies.

[0025] Further, the change amount of the extracted feature in the preset time period is obtained, and the change rate of the extracted feature in the preset time period is determined, including the following steps:

[0026] The change amount of the extracted feature of each timestamp in the preset time period is obtained.

[0027] determine the change rate of the extracted feature at each timestamp by the change amount of the extracted feature and the feature value of the extracted feature in the background image;

[0028] accumulate the change rate of the extracted feature at each timestamp in the preset period to obtain the change rate sum of the extracted feature in the preset period;

[0029] obtain the total number of timestamps in the preset period, and determine the change rate of the extracted feature in the preset period by the ratio of the total number of timestamps and the change rate sum.

[0030] The preset period can be set according to actual needs, for example, set to several seconds or tens of seconds. The change amount of the extracted feature at each timestamp is obtained, and the change of the feature at different times in a short time is recorded. For short-time abnormalities, this record helps to capture the dynamic process of abnormal occurrence. For example, when the tissue is temporarily ischemic, the change amount of the feature at different timestamps can reflect the dynamic change of the ischemia degree.

[0031] The change rate can reflect the speed of the change of the feature. The change rate is calculated by the change amount and the feature value of the extracted feature in the background image, and the change intensity of the feature at each time point is measured. For local slight bleeding, the feature value will change with the change of the bleeding condition, and the calculated change rate can intuitively reflect the speed of the bleeding, which helps to judge the severity and development trend of the abnormality. The change rate sum comprehensively considers the change of the feature at each time point in the preset period, calculates the mean change rate in the preset period, and reflects the overall change degree of the feature in the entire preset period. For example: for instrument instantaneous mis-touch, although the mis-touch time is short, by accumulating the change rate at each time point, the influence of the entire mis-touch process on the feature can be quantified, which helps to evaluate the influence of the abnormality on the entire operation process.

[0032] Further, the step of determining the change rate of the extracted feature at each timestamp comprises:

[0033] obtain the mark of the extracted feature at the timestamp;

[0034] when the mark of the extracted feature at the timestamp is a change feature, obtain the change amount of the extracted feature at the timestamp and the feature value of the corresponding background image to determine the change rate of the extracted feature at the timestamp;

[0035] when the mark of the extracted feature at the timestamp is a new feature, the change rate of the extracted feature at the timestamp is 1.

[0036] By acquiring the label, the type of the extracted feature at the corresponding time point is determined, i.e. whether it is a changed feature or a new feature. For the changed feature, the change rate is calculated by acquiring the change amount at the current timestamp and the feature value of the corresponding background image, and the change intensity of the feature at the time point is quantified; the dynamic change process of the feature in the time dimension is reflected in real time, and through the calculation of the change rate at different time points, it is understood whether the feature change is gradually intensified or tends to be stable. For example, when the tissue is temporarily ischemic, the ratio of the change amount of the tissue color feature to the feature value in the background image can reflect the change of the ischemia degree at the time point, which helps to judge the severity of the abnormality.

[0037] In laparoscopic operations, new features are a significant dynamic change, and setting 1 can reflect the intensity of the change to some extent, which helps to discover potential abnormalities in a timely manner; for example, new indentations or reflections produced by instantaneous accidental touch of instruments, new ranges in the working area where the laparoscope moves, and new features collected.

[0038] Further, based on the above background image, the background area of the processed image is removed to obtain a change area, and the specific steps include:

[0039] The above processed image and the background image are compared in pixels to determine the difference value of each pixel in the processed image and the background image;

[0040] A difference threshold is preset, the pixels with a difference value greater than the difference threshold are marked as foreground pixels, and the pixels with a difference value less than or equal to the difference threshold are marked as background pixels;

[0041] According to the above foreground pixels and background pixels, a binary foreground mask image is generated;

[0042] The above foreground mask image and the processed image are subjected to bitwise AND operation to obtain a change area.

[0043] The above processed image and the background image are compared in pixels to calculate the difference value, which helps to distinguish static background and dynamic foreground, and helps to locate the change area and speed up image recognition.

[0044] A difference threshold is set to determine whether a pixel belongs to the foreground or the background; based on the division of the foreground pixels and the background pixels, a binary (black and white) foreground mask image is generated, in which the foreground pixels are usually represented as white (value 1) and the background pixels are represented as black (value 0); the foreground mask image clearly distinguishes the foreground region (i.e. the change region) that needs to be concerned and the background region that does not need to be considered, thereby simplifying the complexity of subsequent processing. The foreground mask image is used as a "filter" and is subjected to a bitwise AND operation with the processing image, so that only the part considered as the foreground is retained, thereby obtaining the final change region. This process effectively removes the background interference, so that only the foreground of interest is contained in the change region, thereby improving the efficiency and accuracy of the subsequent feature extraction and recognition steps.

[0045] The second aspect provides an abnormality recognition system based on a robot laparoscope, which adopts the abnormality recognition method described above.

[0046] The abnormality recognition system comprises:

[0047] a laparoscope, which collects video data of a working area;

[0048] a control center, which is connected to the laparoscope; the control center is used to preset the working area and perform the following steps:

[0049] extract features of each video frame based on the video data, identify the changed features between adjacent video frames, and record the changed features, corresponding feature values and change amounts;

[0050] determine whether the features are abnormal by comparing the feature threshold and the feature value of the features;

[0051] obtain the change amount of the features in a preset period, determine the change rate of the features in the preset period, and determine whether the change of the features is abnormal by comparing the change rate threshold and the change rate of the features in the preset period;

[0052] generate a solution strategy based on the abnormal features, feature values and change rates;

[0053] an abnormality database, which is connected to the control center; the abnormality database is used to store the abnormal and changed features, feature values and change rates.

[0054] Further, the abnormality recognition system further comprises a record database, which stores the associated data after associating the features, corresponding feature values and video frame timestamps.

[0055] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0056] Since the collection range of the laparoscope is limited, the complete video collection of the working area needs time, and the laparoscope is gradually moved during the collection, and the extracted features are gradually added and extracted features are gradually disappeared in the subsequent video frames. By focusing on the changes of the features in adjacent video frames, the subtle changes in a short time are found in time; by setting the feature threshold and comparing it with the actual feature value, it is quantitatively judged whether the extracted feature is in an abnormal state, and the static abnormality recognition is realized; in addition, the change rate of the extracted feature in the preset period is also considered, which reflects the dynamic change of the extracted feature in a period of time; for some dynamic changes, the change of the extracted feature in a short time is required, even if the analysis of each frame meets the static requirements, the dynamic change requirements will also appear, therefore, by comparing the preset change rate threshold with the actual change rate, the change of the extracted feature is further judged from the perspective of dynamic change, and the short-time abnormality recognition is realized. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical scheme of the exemplary embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments, and it should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor. In the drawings:

[0058] Figure 1 The abnormality recognition method flow chart. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical scheme and advantages of the present application more clear and obvious, the following will further explain the present application in combination with the embodiments and drawings, the exemplary embodiments of the present application and the explanation thereof are only used to explain the present application, and should not be regarded as a limitation on the present application.

[0060] First embodiment:

[0061] In combination Figure 1 , the abnormality recognition method based on the robot laparoscope comprises the following steps:

[0062] Preset working area, collect video data of the working area by the laparoscope;

[0063] Based on the above video data, the extracted features of each video frame are obtained, the changed extracted features in adjacent video frames are identified, and the changed extracted features and the corresponding feature values and change amounts are recorded;

[0064] By comparing the feature threshold and the feature value of the above extracted features, it is determined whether the extracted feature is abnormal; if the above extracted feature is abnormal, the extracted feature and the corresponding feature value are stored in the abnormality database;

[0065] acquiring a variation of the extracted feature in a preset period, and determining a variation rate of the extracted feature in the preset period;

[0066] comparing the variation rate threshold of the extracted feature in the preset period with the variation rate, and determining whether the variation of the extracted feature is abnormal; if the variation of the extracted feature is abnormal, storing the extracted feature and the corresponding variation rate into an abnormal database;

[0067] generating a solution strategy based on the extracted feature, the feature value and the variation rate in the abnormal database.

[0068] Since the collection range of the laparoscope is limited, the complete video collection of the working area needs time, and during the collection, the laparoscope gradually moves, and the extracted feature is gradually added and disappears in the subsequent video frames. By focusing on the change of the feature in the adjacent video frames, the subtle change in a short time is found in time; by setting the feature threshold and comparing it with the actual feature value, it is quantitatively judged whether the extracted feature is in an abnormal state, and static abnormality recognition is realized; in addition, the variation rate of the extracted feature in a preset period is also considered, which reflects the dynamic change of the extracted feature in a period of time; for some dynamic changes, there are requirements for the change of the extracted feature in a short time, even if the analysis of each frame meets the static requirements, there will also be a situation that does not meet the dynamic change requirements, therefore, by comparing the preset variation rate threshold with the actual variation rate, it is further judged from the perspective of dynamic change whether the variation of the extracted feature is abnormal, and short-time abnormality recognition is realized.

[0069] According to the abnormal data identified in the foregoing and stored in the abnormal database, a corresponding solution strategy is generated, measures are taken in time for the short-time abnormality, and the abnormal state is avoided from being continuously accumulated or deteriorated, thereby reducing the risk of postoperative complications and solving the problem that the prior art cannot effectively handle short-time abnormality.

[0070] A referenceable use scenario is that, in a urology surgery test, carbon dioxide is injected into the peritoneal cavity of a target (which can be a human model) through a gas needle, the abdomen of the target is inflated, a laparoscope and instruments are inserted into the peritoneal cavity, and video data in the peritoneal cavity of the target is collected by the laparoscope; features and feature values in adjacent video frames are extracted, whether the features and the feature changes are abnormal is judged based on the extracted features and feature values, and short-time abnormality recognition is realized.

[0071] Second embodiment:

[0072] On the basis of the first embodiment, the extracted features changed in the adjacent video frames are identified, the changed extracted features and the corresponding feature values and variation amounts are recorded, and the specific steps include:

[0073] Acquire the video frame of the adjacent time, take the video frame of the last time in the adjacent time video frame as the background image, and take the video frame of the next time as the processing image;

[0074] Based on the above background image, the background area of the processing image is removed to obtain the change area;

[0075] The extraction feature and the feature value are obtained from the above change area;

[0076] After associating the above extraction feature, the corresponding feature value and the video frame timestamp, the association data is obtained; the above association data is stored in the record database;

[0077] The feature value of the extraction feature in the background image is called from the above record database, the difference between the feature value of the extraction feature in the processing image and the background image is determined to obtain the change amount of the extraction feature.

[0078] According to the video frame rate, the dynamic change of the adjacent video frame is analyzed, the video frame rate can be adjusted by itself; the previous frame is taken as the background, and the next frame is taken as the processing image, the background area is removed to highlight the part different from the background in the processing image, which reduces the interference of irrelevant information, makes the analysis focus on the area actually changed, improves the efficiency of short-time anomaly detection; the feature and the feature value are extracted in the change area to obtain the information related to the short-time anomaly. The extraction of the feature from the change area can avoid the influence of the feature of the irrelevant area, improve the pertinence of the feature extraction, and help to identify the short-time anomaly.

[0079] In a specific embodiment, if the extraction feature is called from the above background image, the extraction feature is marked as a change feature; the difference between the feature value of the change feature in the above processing image and the background image is taken as the change amount of the extraction feature, and the formula is: wherein, represents the change amount of the extraction feature , the feature value of the extraction feature in the processing image, the feature value of the extraction feature in the background image;

[0080] If the extraction feature is not called from the above background image, the extraction feature is marked as a new feature; the feature value of the new feature is taken as the change amount of the extraction feature, .

[0081] ​The extracted feature that can be called from the background image is marked as a change feature, and it is clear that the feature has changed on the original basis, and the feature that changes in a short time is focused on. For example, the color, texture and other features of the tissue may change slightly during transient ischemia. By marking the change feature, the feature that may be abnormal is located, and the pertinence of short-term anomaly detection is improved. The difference between the feature values of the change feature in the processing image and the background image is taken as the change amount, and the change degree of the feature is quantified. For short-term anomalies, the severity and development trend of the anomaly are quantitatively judged. For example, when there is slight bleeding in a local area, the color feature value of the bleeding area will change. By calculating the change amount, it can be known whether the degree of bleeding is increasing, thereby providing a basis for taking timely measures.

[0082] The extracted feature that cannot be called from the background image is marked as a new feature, which can timely discover new features appearing in a short time (i.e., features appearing when the laparoscope moves to different regions). In laparoscopic operations, new features such as new indentations may be generated at the contact site of the instrument. By marking the new feature, unexpected situations can be quickly perceived, and more serious consequences caused by not timely discovery can be avoided. For the new feature, it is a change information itself, and in short-term anomaly recognition, the new feature may indicate the occurrence of an anomaly. The feature value of the new feature is taken as the change amount, which is helpful to timely discover potential short-term anomalies.

[0083] Third embodiment:

[0084] On the basis of the second embodiment, the change amount of the extracted feature in the preset period is obtained, and the change rate of the extracted feature in the preset period is determined. The specific steps include:

[0085] The change amount of the extracted feature of each timestamp in the preset period is obtained.

[0086] The change rate of the extracted feature at each timestamp is determined by the change amount of the extracted feature and the feature value of the extracted feature in the background image.

[0087] The change rates of the extracted feature at the timestamps in the preset period are accumulated to obtain the total change rate of the extracted feature in the preset period.

[0088] The total number of timestamps in the preset period is obtained, and the change rate of the extracted feature in the preset period is determined by the ratio of the total number of timestamps to the total change rate, and the formula is as follows: , wherein, represents the extracted feature in the preset period, represents the total number of timestamps, represents the extracted feature in the timestamp .

[0089] The preset time period can be set according to actual needs, for example, set to several seconds or tens of seconds. The change amount of the extracted feature at each timestamp is obtained, and the change of the feature at different times in a short time is recorded. For short-time abnormalities, this record helps to capture the dynamic process of abnormal occurrence. For example, when the tissue is temporarily ischemic, the change amount of the feature at different timestamps can reflect the dynamic change of the ischemia degree.

[0090] The change rate can reflect the speed of the change of the feature. The change rate is calculated by the change amount and the feature value of the extracted feature in the background image, which measures the change intensity of the feature at each time point. For local slight bleeding, the feature value will change with the change of the bleeding condition, and the calculated change rate can intuitively reflect the speed of the bleeding, which helps to judge the severity and development trend of the abnormality. The change rate sum comprehensively considers the change of the feature at each time point in the preset time period, calculates the average change rate in the preset time period, and reflects the overall change degree of the feature in the whole preset time period. For example: for instrument instantaneous mis-touch, although the mis-touch time is short, by accumulating the change rate at each time point, the influence of the whole mis-touch process on the feature can be quantified, which helps to evaluate the influence of the abnormality on the whole operation process.

[0091] In a specific embodiment, the step of determining the change rate of the extracted feature at each timestamp includes:

[0092] Obtaining the mark of the extracted feature at the timestamp;

[0093] When the mark of the extracted feature at the timestamp is a change feature, the change amount of the extracted feature at the timestamp and the feature value of the corresponding background image are obtained, and the change rate of the extracted feature at the timestamp is determined, and the formula is as follows:

[0094] wherein, represents the extracted feature at the timestamp , the change amount of the extracted feature at the timestamp , and the feature value of the extracted feature in the background image

[0095] When the mark of the extracted feature at the timestamp is a new feature, the change rate of the extracted feature at the timestamp is 1.

[0096] By acquiring the label, the type of the extracted feature at the corresponding time point is determined, i.e., whether it is a changed feature or a new feature. For the changed feature, the change rate is calculated by acquiring the change amount at the current timestamp and the feature value of the corresponding background image, and the change intensity of the feature at the time point is quantified; the dynamic change process of the feature in the time dimension is reflected in real time, and through the calculation of the change rate at different time points, it is understood whether the feature change is gradually intensified or tends to be stable. For example, when the tissue is temporarily ischemic, the ratio of the change amount of the tissue color feature to the feature value in the background image can reflect the change of the ischemia degree at the time point, which helps to judge the severity of the abnormality.

[0097] In laparoscopic operations, new features are a significant dynamic change, and setting 1 can reflect the intensity of the change to some extent, which helps to discover potential abnormalities in a timely manner; for example, new indentations or reflections produced by instantaneous accidental touch of instruments, new ranges in the working area where the laparoscope moves, and new features collected.

[0098] Fourth embodiment:

[0099] On the basis of the second embodiment, based on the above background image, the background area of the processed image is removed to obtain a change area, and the specific steps include:

[0100] The above processed image and the background image are compared in pixels to determine the difference value of each pixel in the processed image and the background image;

[0101] A difference threshold is preset, the pixels with a difference value greater than the difference threshold are marked as foreground pixels, and the pixels with a difference value less than or equal to the difference threshold are marked as background pixels;

[0102] According to the above foreground pixels and background pixels, a binary foreground mask image is generated;

[0103] The above foreground mask image and the processed image are subjected to bitwise AND operation to obtain a change area.

[0104] The above processed image and the background image are compared in pixels to calculate the difference value, which helps to distinguish static background and dynamic foreground, and helps to locate the change area and speed up image recognition.

[0105] A difference threshold is set to determine whether a pixel belongs to the foreground or the background; based on the division of the foreground pixels and the background pixels, a binary (black and white) foreground mask image is generated, in which the foreground pixels are usually represented as white (value 1) and the background pixels are represented as black (value 0); the foreground mask image clearly distinguishes the foreground region (i.e. the change region) that needs to be concerned and the background region that does not need to be considered, thereby simplifying the complexity of subsequent processing. The foreground mask image is used as a "filter" and is subjected to a bitwise AND operation with the processing image, so as to only retain the part considered as the foreground, thereby obtaining the final change region. This process effectively removes the background interference, so that only the foreground of interest is contained in the change region, thereby improving the efficiency and accuracy of the subsequent feature extraction and recognition steps.

[0106] Fifth embodiment:

[0107] An abnormality recognition system based on a robot laparoscope, the abnormality recognition system adopts the above abnormality recognition method;

[0108] The abnormality recognition system comprises:

[0109] a laparoscope, which collects video data of a working area;

[0110] a control center, which is connected to the laparoscope; the control center is used to preset a working area and perform the following steps:

[0111] extracting features of each video frame based on the video data, identifying changed features in adjacent video frames, recording the changed features and corresponding feature values and change amounts;

[0112] determining whether the features are abnormal by comparing the feature threshold and the feature value of the features;

[0113] obtaining the change amount of the features in a preset period, determining the change rate of the features in the preset period, and determining whether the change of the features is abnormal by comparing the change rate threshold and the change rate of the features in the preset period;

[0114] generating a solution strategy based on the abnormal features, feature values and change rates;

[0115] an abnormality database, which is connected to the control center; the abnormality database is used to store abnormal and changed features, feature values and change rates.

[0116] In a specific embodiment, the abnormality recognition system further comprises a record database, which stores associated data after associating the features, corresponding feature values and video frame timestamps.

[0117] The above detailed description of the specific embodiments of the present application has been given to understand the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An anomaly identification method based on robotic laparoscopy, characterized in that, Includes the following steps: A pre-defined working area is selected, and video data of the working area is acquired using a laparoscope. Based on the video data, extract features of each video frame are obtained, and the changing extract features in adjacent video frames are identified. The changing extract features, their corresponding feature values, and the amount of change are recorded. By comparing the feature threshold and feature value of the extracted feature, it is determined whether the extracted feature is abnormal; if the extracted feature is abnormal, the extracted feature and its corresponding feature value are stored in the abnormal database. Obtain the amount of change of the extracted features during a preset time period, and determine the rate of change of the extracted features during that preset time period; By comparing the rate of change threshold and the rate of change of the extracted feature in the preset time period, it is determined whether the change of the extracted feature is abnormal; if the change of the extracted feature is abnormal, the extracted feature and the corresponding rate of change are stored in the abnormal database. A solution strategy is generated based on the extracted features, feature values, and rates of change from the aforementioned anomaly database; The steps for identifying and extracting features that change in adjacent video frames, and recording the extracted features, their corresponding feature values, and the magnitude of change, include: Acquire video frames at adjacent time points, use the video frame of the previous time point in the adjacent time point video frames as the background image, and the video frame of the next time point as the processed image; Based on the background image, the background area of ​​the processed image is removed to obtain the changed area; Extract features and feature values ​​from the changed region; After associating the extracted features, corresponding feature values, and video frame timestamps, associated data is obtained; the associated data is then stored in a record database. The feature value of the extracted feature is retrieved from the background image from the recorded database, and the difference between the feature values ​​of the extracted feature in the processed image and the background image is determined to obtain the change in the extracted feature.

2. The anomaly identification method according to claim 1, characterized in that, If the extracted feature is retrieved from the background image, the extracted feature is marked as a changed feature; the difference between the feature values ​​of the changed feature in the processed image and the background image is taken as the change amount of the extracted feature; If the extracted feature is not retrieved from the background image, the extracted feature is marked as a new feature; the feature value of the new feature is used as the change in the extracted feature.

3. The anomaly identification method according to claim 1, characterized in that, The steps for obtaining the change amount of the extracted features during a preset time period and determining the change rate of the extracted features during that preset time period include: Obtain the change in the extracted features of each timestamp within the preset time period; The rate of change of the extracted feature at each time point is determined by the amount of change of the extracted feature and the feature value of the extracted feature in the background image. The cumulative rate of change of the extracted features at each timestamp within the preset time period is used to obtain the sum of the rate of change of the extracted features within the preset time period. Obtain the total number of timestamps for the preset time period, and determine the rate of change of the extracted features during the preset time period by the ratio of the total number of timestamps to the sum of the change rates.

4. The anomaly identification method according to claim 1, characterized in that, The steps to determine the rate of change of the extracted features at each time stamp include: Obtain the timestamp marker of the extracted features; When the extracted feature is marked as a change feature at the timestamp, the change amount of the extracted feature at that timestamp and the feature value of the corresponding background image are obtained to determine the change rate of the extracted feature at that timestamp. When the extracted feature is marked as a newly added feature at the timestamp, the change rate of the extracted feature at that timestamp is 1.

5. The anomaly identification method according to claim 1, characterized in that, Based on the background image, the background region of the processed image is removed to obtain the changed region. The specific steps include: The processed image is compared pixel by pixel with the background image to determine the difference value of each pixel between the processed image and the background image; A preset difference threshold is set, and pixels with a difference value greater than the difference threshold are marked as foreground pixels, while pixels with a difference value less than or equal to the difference threshold are marked as background pixels. A binarized foreground mask image is generated based on the foreground pixels and background pixels; The foreground mask image and the processed image are subjected to a bitwise AND operation to obtain the changed region.

6. An anomaly recognition system based on robotic laparoscopy, characterized in that, The anomaly identification system employs the anomaly identification method described in any one of claims 1 to 5; The anomaly detection system includes: Laparoscope, used to acquire video data of the working area; The control center, connected to the laparoscope, is used to preset the working area and execute the following steps: Based on video data, extract features from each video frame, identify the changing extract features in adjacent video frames, and record the changing extract features, as well as the corresponding feature values ​​and changes. By comparing the feature threshold and feature value of the extracted feature, it is determined whether the extracted feature is abnormal; The change amount of the extracted features within a preset time period is obtained, and the change rate of the extracted features within that preset time period is determined. By comparing the change rate threshold and the change rate of the extracted features within that preset time period, it is determined whether the change of the extracted features is abnormal. Based on the extracted features, feature values, and rate of change of the aforementioned anomalies, a solution strategy is generated. An anomaly database is connected to the control center; the anomaly database is used to store the extracted features, feature values, and change rates of anomalies and anomaly changes.

7. The anomaly identification system according to claim 6, characterized in that, The anomaly detection system also includes a record database, which stores associated extracted features, corresponding feature values, and associated data after video frame timestamps.

Images