An automatic control method and device for an endoscope

Abstract

The application discloses an automatic control method and device of an endoscope, and the method comprises the following steps: acquiring an environment image and constructing a three-dimensional voxel map based on the environment image and a SLAM technology; when it is detected that the environment image clarity of the environment where the endoscope is currently located does not reach a set clarity requirement, acquiring environment data corresponding to each sensor through a plurality of sensors on the endoscope, and fusing the environment data into a group of current scene data; comparing the current scene data with a plurality of groups of preset candidate scene data, and then selecting a group of target scene data with the highest similarity range; and operating the endoscope according to a control strategy corresponding to the three-dimensional voxel map and the target scene data. The application can control the endoscope according to a target control strategy corresponding to a target scene described by each environment data, can provide sufficient reaction time for an operator when the environment image is not clear, and ensures the safety of endoscope operation control when the environment image is not clear. The application can be applied to the field of endoscope control.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic control, in particular to an automatic control method and device of an endoscope. BACKGROUND

[0002] At present, an endoscope is an essential device in many diagnosis and treatment processes. The endoscope comprises an insertion part, an operation part and a general cable extending from the operation part. The general cable is connected with an external device through a connector, and a doctor can control the operation of corresponding components through the external device. The endoscope needs to be inserted into a dark space for diagnosis and treatment. When the endoscope enters different cavities, the imaging will be affected by the size of the space. For example, when the endoscope enters the esophagus, the movable space range of the endoscope is relatively small due to the narrow esophagus. During the movement, the endoscope tip is too close to the cavity wall due to the small space, and the image is overexposed. For another example, when the endoscope enters the stomach, the image in the dark area is too dark due to the large space of the stomach and the irregular plane of the stomach. The reasons above will cause differences in imaging quality. When the endoscope is affected by the current environment, the sensor obtains an unclear environment image. The operator of the endoscope may not obtain target information from the environment image according to the naked eye, which leads to inaccurate control of the endoscope and easily causes damage to the internal tissues of the human body. Therefore, an endoscope control scheme is needed to accurately control the endoscope in the case that the sensor obtains an unclear environment image, so as to avoid damage to the human body. SUMMARY

[0003] Therefore, the embodiments of the present application provide an automatic control method and device which can accurately control the endoscope in the case that the sensor obtains an unclear environment image.

[0004] An aspect of the embodiments of the present application provides an automatic control method of an endoscope, comprising:

[0005] An environment image of an internal cavity environment is obtained through an image sensor arranged on the endoscope, and a three-dimensional voxel map is constructed based on the environment image and a SLAM technology;

[0006] If the definition of the environment image does not reach the set definition requirement, environment data corresponding to each sensor of the endoscope in the operation process is obtained through other different sensors arranged on the endoscope, and the environment data obtained by the multiple sensors is fused into a group of current scene data;

[0007] From a plurality of groups of candidate scene data, a group of candidate scene data with a similarity reaching a set similarity range to the current scene data is selected as target scene data;

[0008] determine a navigation route based on the three-dimensional voxel map, and control the endoscope to operate according to a target control strategy corresponding to the target scene data.

[0009] Optionally, the environment data corresponding to each of the sensors during the operation of the endoscope is acquired by other different sensors arranged on the endoscope, including:

[0010] The obstacle ultrasonic image detected by the endoscope during the operation of the endoscope is acquired by an ultrasonic sensor arranged on the endoscope.

[0011] The infrared environment image of the environment in which the endoscope is located is acquired by an infrared sensor arranged on the endoscope.

[0012] Optionally, the types of the obstacles detected by the endoscope during the operation of the endoscope include protrusions, cavity walls, bubbles, mucus, and tumors.

[0013] The control of the endoscope to operate according to the target control strategy corresponding to the target scene data includes:

[0014] When the obstacle is a protrusion or a cavity wall, a first position of the protrusion or the cavity wall and a first distance between the protrusion or the cavity wall and the endoscope are determined according to the SLAM technology, and the endoscope is controlled to move around the protrusion or the cavity wall according to the first position and the second distance.

[0015] When the obstacle is a bubble, the endoscope is controlled to move through the bubble.

[0016] When the obstacle is mucus, the mucus is adsorbed by a suction seat arranged on the endoscope, so that the mucus is separated from the inner cavity environment.

[0017] When the obstacle is a tumor, a second distance between the tumor and the endoscope is determined, and the endoscope is controlled to perform a setting operation according to the second distance.

[0018] Optionally, the control of the endoscope to perform the setting operation according to the second distance includes:

[0019] When the second distance is less than or equal to a preset first threshold value, the endoscope is controlled to perform a preset warning operation.

[0020] When the second distance is less than or equal to a preset second threshold value, the endoscope is controlled to decelerate or brake.

[0021] The second threshold value is less than the first threshold value.

[0022] Optionally, the construction of the three-dimensional voxel map based on the environment image and the SLAM technology includes:

[0023] The ORB-SLAM2 algorithm is used to extract feature points from multiple time-stamped environment images, and a first voxel map is constructed according to the feature points, wherein the feature points are points with a set feature in the environment images;

[0024] The feature points in the first voxel map are two-dimensionally triangulated, the triangles obtained by triangulation are projected into a three-dimensional space, and triangles with a side length exceeding a set threshold are filtered out, and a point cloud map is obtained by supplementing points in the region of the filtered-out triangles;

[0025] New point clouds are supplemented in the idle region of the point cloud map by a ray projection method, and the point clouds in the point cloud map are inserted into a preset second voxel map based on an OctoMAP map in time-stamped order, to obtain a three-dimensional voxel map.

[0026] Optionally, the fusion of the environment data obtained by the plurality of sensors into a set of current scene data comprises:

[0027] The data collected by each sensor at the latest time and the corresponding collection time are determined and saved as a time stamp;

[0028] The time stamp with the latest collection time is taken as a matching center, the time difference between each time stamp and the matching center is calculated, time stamps outside the allowed error range are discarded, and all time stamps of the sensors are compared until the time difference, and the obtained time stamps are fused into a set of current scene data.

[0029] Optionally, it further comprises:

[0030] If the definition of the environment image meets the set definition requirement, the endoscope is controlled to operate according to the environment image.

[0031] Another aspect of the embodiment of the application further provides an automatic control device of an endoscope, comprising:

[0032] An environment image acquisition unit is configured to acquire an environment image of an internal cavity environment through an image sensor arranged on an endoscope, and construct a three-dimensional voxel map based on the environment image and a SLAM technology;

[0033] An environment data acquisition unit is configured to acquire environment data corresponding to each sensor of the endoscope during operation through other different sensors arranged on the endoscope if the definition of the environment image does not meet the set definition requirement, and fuse the environment data obtained by the plurality of sensors into a set of current scene data;

[0034] The scene data selecting unit is configured to select, from a plurality of preset candidate scene data groups, a candidate scene data group with a similarity to the current scene data reaching a set similarity range as target scene data.

[0035] The operation control unit is configured to control the endoscope to operate according to a target control strategy corresponding to the target scene data based on a navigation route determined by the three-dimensional voxel map.

[0036] Optionally, the environment data acquisition unit comprises:

[0037] The first environment data acquisition subunit is configured to acquire an environment image of the endoscope in the operation process through an image sensor arranged on the endoscope.

[0038] The second environment data acquisition subunit is configured to acquire an obstacle ultrasonic image detected by the endoscope in the operation process through an ultrasonic sensor arranged on the endoscope.

[0039] The third environment data acquisition subunit is configured to acquire an infrared environment image of an environment in which the endoscope is located through an infrared sensor arranged on the endoscope.

[0040] Optionally, the types of the obstacles detected by the endoscope in the operation process include a protrusion, a cavity wall, a bubble, mucus and a tumor.

[0041] The operation control unit comprises:

[0042] The first operation control subunit is configured to determine a first position of the protrusion or the cavity wall and a first distance between the protrusion or the cavity wall and the endoscope according to a SLAM technology when the obstacle is the protrusion or the cavity wall, and control the endoscope to move around the protrusion or the cavity wall according to the first position and the first distance.

[0043] The second operation control subunit is configured to control the endoscope to move through the bubble when the obstacle is the bubble.

[0044] The third operation control subunit is configured to control an attraction seat arranged on the endoscope to adsorb the mucus so as to separate the mucus from the cavity environment when the obstacle is the mucus.

[0045] The fourth operation control subunit is configured to determine a second distance between the tumor and the endoscope when the obstacle is the tumor, and control the endoscope to perform a set operation according to the second distance.

[0046] Another aspect of the embodiment of the present application further provides an electronic device comprising a processor and a memory.

[0047] The memory is configured to store a program.

[0048] The processor executes the program to implement the method.

[0049] Another aspect of the embodiment of the present application also provides a computer readable storage medium, which stores a program, and the program is executed by a processor to implement the method.

[0050] The embodiment of the present application also discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the method.

[0051] The present application acquires the environment image of the inner cavity environment through the image sensor arranged on the endoscope, and constructs a three-dimensional voxel map based on the environment image and the SLAM technology. When it is detected that the environment image of the current environment where the endoscope is located is not clear, that is, the definition of the environment image does not reach the set definition requirement, the environment data corresponding to each sensor is acquired through the plurality of sensors on the endoscope, and is fused into a group of scene data as the current scene data. Then, the current scene data can be compared with the preset plurality of candidate scene data to determine the similarity range of each candidate scene data, and the candidate scene data with the highest similarity range is selected as the target scene data. Then, the endoscope can be operated according to the target control strategy corresponding to the target scene data based on the navigation route determined by the voxel map. The present application can acquire a plurality of environment data in the operation process of the endoscope by using a plurality of sensors. For example, the infrared sensor can accurately acquire environment data without being affected by the external environment, but the light intensity received by the receiving tube is affected by the distance of the reflecting object, that is, the closer the distance, the stronger the reflected light, and the farther the distance, the weaker the reflected light. The ultrasonic sensor can accurately measure the distance of the obstacle within the distance range of 30-300 cm to compensate for the defects of the infrared sensor distance measurement. After the environment data collected by the above-mentioned plurality of sensors is fused, the target control strategy corresponding to the target scene described by each environment data is controlled, the reaction time of the operator when the environment image obtained by the sensor is not clear is reduced, the accuracy of the operation is improved, the safety of the operation control of the endoscope when the environment image is not clear is ensured, and the occurrence of damage is avoided. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort based on these drawings.

[0053] Figure 1 A flowchart of an automatic control method of an endoscope provided by an embodiment of the present application is shown in the figure;

[0054] Figure 2 A flowchart of another automatic control method of an endoscope provided by an embodiment of the present application is shown in the figure;

[0055] Figure 3 A specific example diagram of an automatic control strategy of an endoscope provided by an embodiment of the present application is shown in the figure;

[0056] Figure 4 A structural block diagram of an automatic control device of an endoscope provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present application clearer and more apparent, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0058] With reference to Figure 1 The embodiment of the present application provides an automatic control method of an endoscope, which specifically comprises the following steps:

[0059] Step S100: acquiring an environment image of an internal cavity environment through an image sensor arranged on the endoscope, and constructing a three-dimensional voxel map based on the environment image and a SLAM technology.

[0060] Specifically, the internal cavity environment where the endoscope is located can refer to the environment where the part entering the human tissue is located, and the acquired environment image can refer to the image of the environment where the part entering the human tissue is located. Then, whether the set definition requirement is met can be determined by detecting the definition of the acquired environment image. The set definition requirement can refer to that the light intensity of the current environment where the endoscope is located is too low or too high, the distance to the internal cavity wall is too close or too far, so that the operator of the endoscope cannot see the environment around the operating part of the endoscope through the display device of the endoscope, and thus the sensor on the endoscope needs to be turned on.

[0061] It should be noted that the set definition requirement introduced above can be freely set by the operator of the endoscope. Different operators or different diagnosis and treatment processes can have different definition requirements for the observation field of view of the endoscope. Therefore, the present application does not strictly limit the set definition requirement.

[0062] The three-dimensional voxel map of the human body currently examined by the endoscope can be determined according to the environment image of the internal cavity environment and the SLAM technology. The three-dimensional voxel map can provide a navigation route for the operation of the endoscope.

[0063] Step S110: If the definition of the environment image does not reach the set definition requirement, environment data corresponding to each sensor of the endoscope during operation is acquired through other different sensors arranged on the endoscope, and the environment data acquired by the plurality of sensors is fused into a set of current scene data.

[0064] Specifically, when the definition of the environment image does not reach the definition requirement, it can be considered that the light intensity of the environment where the endoscope is currently located is low or high, the distance to the inner cavity wall is too close or too far, and the endoscope operator cannot be provided with sufficient observation field of view. At this time, the environment data corresponding to each sensor of the endoscope during operation can be acquired through a plurality of sensors on the endoscope, and the environment data obtained by each sensor can be fused into a set of scene data as the current scene data.

[0065] Step S120: From a plurality of sets of candidate scene data, a set of candidate scene data with a similarity to the current scene data reaching a set similarity range is selected as target scene data.

[0066] Specifically, each set of candidate scene data can include a plurality of types of environment data, and the types of environment data included in each set of candidate scene data can include the types of data included in the current scene data. The embodiment of the application takes the same types of data of the candidate scene data and the current scene as an example.

[0067] In an optional implementation, the scene data of the application can include one or more of the following environment data: whether there is an obstacle, the position of the obstacle, the type of the obstacle, the distance between the endoscope and the obstacle, the position of the cavity where the endoscope is currently located, and body mucus.

[0068] The specific values of each scene data in each set of candidate scene data can be freely set in advance, or the values of each scene data can be set according to industry experience, or the values of each scene data can be set according to a certain specific condition.

[0069] The current scene data is compared with each set of candidate scene data to obtain the similarity of each set of candidate scene data, and the sets of candidate scene data reaching the similarity range are determined, and then the set of candidate scene data with the highest similarity is selected from the sets of candidate scene data reaching the similarity range as the target scene data.

[0070] Step S130: Based on the navigation route determined based on the three-dimensional voxel map, the endoscope is controlled to operate according to the target control strategy corresponding to the target scene data.

[0071] Specifically, each set of candidate scene data can correspond to a control strategy, the control strategy can be prepared in advance, each control strategy can include specific operation steps of the endoscope in the corresponding candidate scene, therefore, the target control strategy corresponding to the target scene data can be determined first, and then the endoscope can be controlled to perform corresponding operations according to the target control strategy and under the navigation route provided by the three-dimensional voxel map.

[0072] In some embodiments of the present application, the above step S110 introduces the processing process when the environmental image definition does not meet the set definition requirement, on this basis, the embodiment of the present application can also add a processing process when the environmental image definition meets the set definition requirement, for details, please refer to Figure 2 .

[0073] Step S140: If the definition of the environmental image meets the set definition requirement, the endoscope is controlled to operate according to the navigation route provided by the environmental image and the three-dimensional voxel map.

[0074] Specifically, if the definition of the environmental image meets the set definition requirement and can provide a good observation field of view for the endoscope operator, the actual environment in which the endoscope is currently located can be determined according to the environmental image, and the endoscope is controlled to operate in combination with the navigation route provided by the three-dimensional voxel map.

[0075] In some embodiments of the present application, the above step S110 introduces the process of acquiring environmental data corresponding to each sensor of the endoscope in the operation process through a plurality of different sensors arranged on the endoscope, and the process of acquiring environmental data corresponding to each sensor will be further described.

[0076] Specifically, different scene data can be acquired through each sensor on the endoscope, and the embodiment of the present application can acquire the following scene data:

[0077] The environmental image of the endoscope in the operation process is acquired through the image sensor arranged on the endoscope.

[0078] Specifically, the image acquired by the image sensor can be detected, and the type of obstacle in the image can be judged according to the image.

[0079] The distance between the tip of the endoscope and the obstacle is acquired through the obstacle image acquired by the ultrasonic sensor arranged on the endoscope.

[0080] The infrared environmental image of the environment in which the endoscope is located is acquired through the infrared sensor arranged on the endoscope.

[0081] Further, when the environmental data in the target scene data includes environmental data of an obstacle, it can be indicated that there is an obstacle in front of or near the current environment of the endoscope, and various obstacles can be encountered in the detection process. The types of obstacles can include protrusions, cavity walls, bubbles, mucus, and tumors, as well as other types of obstacles.

[0082] With reference to Figure 3 , the process of controlling the endoscope to operate according to the target control strategy corresponding to the target scene data in step S130 can include the following cases:

[0083] Case one, when the obstacle is a protrusion or a cavity wall, a first position of the protrusion or the cavity wall and a first distance between the protrusion or the cavity wall and the endoscope are determined according to the SLAM technology, and the endoscope is controlled to move around the protrusion or the cavity wall according to the first position and the first distance;

[0084] Case two, when the obstacle is a bubble, the endoscope is controlled to move through the bubble;

[0085] Case three, when the obstacle is mucus, an attraction seat provided on the endoscope is controlled to adsorb the mucus to separate the mucus from the cavity environment;

[0086] Case four, when the obstacle is a tumor, a second distance between the tumor and the endoscope is determined, and the endoscope is controlled to perform a setting operation according to the second distance.

[0087] Next, the process of controlling the endoscope to perform a setting operation according to the second distance in the above case four is described in detail.

[0088] S1, when the second distance is less than or equal to a preset first threshold value, the endoscope is controlled to perform a setting warning operation.

[0089] Specifically, when the distance between the endoscope and the tumor is less than or equal to the first threshold value, the endoscope can be controlled to perform a setting warning operation.

[0090] Specifically, the warning operation can include reminding the endoscope operator that an obstacle will be encountered through an external display device of the endoscope, and careful operation is required. The warning operation can include sending a warning reminder identifier, a sound reminder, etc. to the external display device of the endoscope.

[0091] The first threshold value can be freely set by the endoscope operator, for example, 30mm, 40mm, etc. The specific value of the first threshold value is not strictly limited by the present application.

[0092] S2, when the second distance is less than or equal to a preset second threshold value, the endoscope is controlled to slow down or brake.

[0093] Specifically, when the endoscope is closer and closer to the tumor, until less than the second threshold, at this time the endoscope can be controlled to decelerate or brake to provide sufficient response time for the endoscope operator.

[0094] The second threshold can be freely set by the endoscope operator, for example, 10mm, 15mm, etc., and the second threshold is less than the first threshold. The specific value of the second threshold is not strictly limited in the present application.

[0095] In some embodiments of the present application, the process of fusing the environment data obtained by the plurality of sensors into a set of current scene data is introduced in the above step S110, and the process will be further described.

[0096] S1, determine the latest time data collected by each sensor and the corresponding collection time, and save it as a timestamp.

[0097] S2, take the timestamp with the latest collection time as the matching center, calculate the time difference between each timestamp and the matching center, discard the timestamp whose time difference is outside the allowed error range, until all sensor timestamps have been compared for time difference, and fuse the obtained timestamps into a set of current scene data.

[0098] Specifically, each timestamp is compared with the matching center. If the comparison result is within the allowed error range, it can be considered that the environment data of the timestamp and the environment data of the matching center are data at the same time; if the comparison result is outside the allowed error range, it can be considered that the environment data of the timestamp and the environment data of the matching center are not data at the same time. In order to ensure the time consistency of the data, the timestamp whose time difference is outside the allowed error range can be discarded, and the remaining timestamps can be fused into a set of scene data as the current scene data.

[0099] Next, the process of fusing the data is described with specific examples.

[0100] Collecting environment data obtained by various sensors, the environment data can include environment images collected by image sensors, obstacle ultrasonic images collected by ultrasonic sensors, and infrared environment images collected by infrared sensors, and the fusion process is as follows:

[0101] 1.1 Collect environment data of different sensors and record collection time of each environment data, save as timestamp;

[0102] 1.2 calibrate the environment data collected by different sensors with timestamp through multi-sensor information matching, to obtain different sensor environment data after matching;

[0103] 1.2.1 Different sensor environment data is established into different arrays qi (i = 1, 2,..., n), and each sensor environment data is stored;

[0104] 1.2.2 A set S is established, and when different arrays qi all contain a piece of sensor environment data, the different arrays qi are saved in the set S;

[0105] 1.2.3 In the set S, the sensor environment data at the latest time is taken as a matching center point, and the time difference from the matching center point is calculated from the sensor environment data farthest from the matching center point; if the time difference is less than or equal to the matching allowed error, the data is retained, otherwise the data is discarded;

[0106] 1.2.4 Repeat steps 1.2.1 to 1.2.3 until all sensor environment data is processed.

[0107] Next, the process of constructing a three-dimensional voxel map based on the environment image and SLAM technology in step S100 is described.

[0108] Specifically, the process of constructing a three-dimensional voxel map can include:

[0109] S1, using an ORB-SLAM2 algorithm to extract feature points from a plurality of time-stamped continuous environment images, and constructing a first voxel map according to the feature points, the feature points being points in the environment image having a set feature.

[0110] Specifically, the internal cavity environment is photographed by an image sensor, and a plurality of environment images can be obtained. After arranging the plurality of environment images in time stamp order, the SLAM system is transmitted, and the ORB-SLAM2 algorithm is used to extract features from each environment image, realizing endoscope pose estimation and constructing a first voxel map.

[0111] Since the first voxel map cannot completely and clearly show the internal structure of the human body, and the adjacent feature points of the first voxel map cannot be determined whether they are connected, it cannot be used for endoscope movement navigation, so the first voxel map can be densified subsequently.

[0112] S2, two-dimensional triangulation is performed on the feature points in the first voxel map, the triangles obtained by triangulation are projected into a three-dimensional space, and triangles with a side length exceeding a set threshold are filtered out, and a point cloud map is obtained by supplementing points in the area of the filtered triangles.

[0113] Specifically, monocular SLAM algorithm inevitably has feature point drift phenomenon in the mapping process, and setting the side length threshold of the triangle can filter out abnormal feature points generated due to feature point drift in the mapping process.

[0114] In addition, the point cloud refers to a set of three-dimensional space (xyz coordinate) points, and the cloud map after the point supplementing does not store the connection information of the surface, the point cloud cannot represent the connection relationship of the cavity surface, and therefore the endoscope cannot be navigated, and therefore the point cloud map can be voxelized.

[0115] S3, supplement new point cloud to the idle area in the point cloud map by ray projection, and insert the point cloud in the point cloud map into the preset second voxel map based on the OctoMAP graph in the time stamp order, to obtain a three-dimensional voxel map.

[0116] Specifically, the idle area in the point cloud map is updated by ray projection, and in the observation process of the environment, whether each feature point is occupied changes with time due to noise and movement of the sensor. Based on the OctoMAP graph, the point cloud in the point cloud map is inserted into the preset second voxel map in the time stamp order of the plurality of environment images, each environment image and the generated feature point are taken as input, the preset second voxel map is updated in the time stamp order of the environment images, and finally the construction of the three-dimensional voxel map is completed.

[0117] The present application can obtain the environment data in the endoscope operation process through the image sensor, the ultrasonic sensor and the infrared sensor, and the like, so as to perform early warning control according to the control strategy corresponding to the environment scene described by the environment data, thereby reducing the reaction time of the endoscope operator when the environment image obtained by the sensor is not clear, and since the infrared sensor is less affected by light, the environment data can be accurately obtained in an environment with insufficient or excessive light, thereby improving the operation accuracy, ensuring the safety of the operation while obtaining a clear environment image, and reducing the possibility of injury.

[0118] Next, the process of the present application for automatic control of the endoscope will be described with specific examples, and the following examples take the application of the endoscope automatic control method of the present application to the central processor of the endoscope external device as an example.

[0119] The central processor of the endoscope external device can obtain whether the environment image currently collected by the endoscope is clear through the cable connected with the endoscope, and the clarity of the environment image is the condition for starting the infrared sensor of the endoscope. The conditions affecting the clarity of the environment image can include: light intensity, distance between the tip of the endoscope and the detected environment, etc.

[0120] In the case that the environment image collected by the endoscope currently located environment is not clear, the central processing unit obtains the environment data in the operation process of the endoscope through the image sensor, ultrasonic sensor and infrared sensor installed on the endoscope; and then can compare the environment data with the pre-stored multiple candidate scene data, select a group of candidate scene data with the highest similarity reaching the set similarity range as the target scene data; determine the target scene described by the target scene data as the environment scene where the endoscope currently locates, and the target scene data can include: the existence of obstacles, obstacle position, distance between the endoscope and the obstacle, type of the obstacle, position of the cavity where the current endoscope locates, body mucus and other environment data.

[0121] The central processing unit obtains the target control strategy corresponding to the target scene data, controls the endoscope to perform a pre-warning operation when the distance between the endoscope and the obstacle is less than 30 mm, and controls the endoscope to perform braking when the distance between the endoscope and the obstacle is less than 10 mm, so as to provide sufficient time for the endoscope operator to cope with the obstacle.

[0122] Reference Figure 4 The embodiment of the present application provides an automatic control device of an endoscope, which comprises:

[0123] An environment image acquisition unit is configured to acquire an environment image of an inner cavity environment through an image sensor arranged on the endoscope, and construct a three-dimensional voxel map based on the environment image and a SLAM technology;

[0124] An environment data acquisition unit is configured to acquire environment data corresponding to each sensor in an operation process of the endoscope through a plurality of different sensors arranged on the endoscope if the definition of the environment image does not reach a set definition requirement, and fuse the environment data acquired by the plurality of sensors into a group of current scene data;

[0125] A scene data selection unit is configured to select a group of candidate scene data with a similarity reaching a set similarity range from a plurality of preset candidate scene data as target scene data.

[0126] An operation control unit is configured to control the endoscope to operate according to a target control strategy corresponding to the target scene data based on a navigation route determined by the three-dimensional voxel map.

[0127] The embodiment of the present application also discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method shown in the embodiment of the present application. Figure 1 The embodiment of the present application also discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method shown in the embodiment of the present application.

[0128] In some alternative embodiments, the function / operations mentioned in the block diagrams can not occur in the order mentioned in the operational illustrations. For example, two blocks shown in succession can in fact be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality / operations involved. Also, embodiments presented and described in the flow diagrams are only examples of the method that can be employed in the present application. The disclosed method is not limited to the order of operations presented in the operational illustrations. Alternative embodiments are possible where the order of various operations is changed and where sub operations described as part of a larger operation are executed in a different order or are executed concurrently.

[0129] Further, while the present application has been described in the context of functional modules, it is to be understood that one or more of the functions and / or features described can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It will also be appreciated that detailed discussion of the actual implementation of each module is not necessary to an understanding of the application. Rather, the properties, functions and internal relationships of the various functional modules disclosed in the devices herein are deemed to be of a nature that would be understood by an engineer once presented with the property, function and internal relationships of the modules. Therefore, the engineer, using ordinary skill in the art, will be able to implement the application as claimed in the claims without undue experimentation. It will also be appreciated that the particular conceptualization disclosed is merely exemplary but the scope of the application is determined by the full breadth of the claims and equivalents thereof.

[0130] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0131] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0132] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0133] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0134] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0135] While the embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely exemplary and are not to be construed as limiting the scope of the application. The scope of the application is defined by the appended claims and their equivalents.

[0136] The above is a specific description of the preferred embodiment of the present application, but the present application is not limited to the described embodiment, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. An automatic control device for an endoscope, characterized in that, include: An environmental image acquisition unit is used to acquire environmental images of the internal cavity environment through an image sensor installed on the endoscope, and to construct a three-dimensional voxel map based on the environmental images and SLAM technology. An environmental data acquisition unit is used to acquire environmental data corresponding to each sensor during the operation of the endoscope through other different sensors set on the endoscope if the clarity of the environmental image does not meet the set clarity requirements, and to fuse the environmental data acquired by multiple sensors into a set of current scene data. The scene data selection unit is used to select a set of candidate scene data from a preset set of candidate scene data whose similarity to the current scene data reaches a set similarity range, and use it as the target scene data; An operation control unit is used to control the endoscope to operate according to the target control strategy corresponding to the target scene data based on the navigation route determined by the three-dimensional voxel map. The step of fusing environmental data acquired by multiple sensors into a set of current scene data includes: Determine the latest data collected by each sensor and its corresponding collection time, and save it as a timestamp; Using the latest timestamp of the acquisition time as the matching center, calculate the time difference between each timestamp and the matching center, discard timestamps whose time difference is outside the allowable error range, until all the timestamps of all sensors have been compared by time difference, and merge the obtained timestamps into a set of current scene data; The environmental data acquisition unit includes: The first environmental data acquisition subunit is used to acquire environmental images of the endoscope during operation through an image sensor installed on the endoscope. The second environmental data acquisition subunit is used to acquire ultrasonic images of obstacles detected by the endoscope during operation through an ultrasonic sensor installed on the endoscope. The third environmental data acquisition subunit is used to acquire infrared environmental images of the environment in which the endoscope is located through infrared sensors installed on the endoscope.

2. The automatic control device for an endoscope according to claim 1, characterized in that, The types of obstacles detected during endoscopic procedures include: protrusions, cavity walls, air bubbles, mucus, and tumors; The operation control unit includes: The first operation control subunit is used to determine the first position of the protrusion or cavity wall and the first distance between the protrusion or cavity wall and the endoscope according to SLAM technology when the obstacle is a protrusion or cavity wall, and to control the endoscope to move around the protrusion or cavity wall according to the first position and the second distance. The second operation control subunit is used to control the endoscope to move through the bubble when the obstacle is a bubble; The third operation control subunit is used to control the suction seat on the endoscope to adsorb the mucus when the obstacle is mucus, so that the mucus is removed from the internal cavity environment; The fourth operation control subunit is used to determine the second distance between the tumor and the endoscope when the obstacle is a tumor, and to control the endoscope to perform setting operations according to the second distance.

Images