Security protection method, system, computer equipment and storage medium

Abstract

The present invention discloses a safety protection method, system, computer equipment and storage medium. The safety protection method comprises: determining an area of ​​interest based on image information in a monitoring area of ​​a target operating terminal collected in real time; detecting whether a non-operating person has invaded the area of ​​interest based on a target detection algorithm, and if so, triggering corresponding early warning actions and / or protection actions to avoid major risks caused by non-operating persons accidentally hitting the operating terminal, thereby improving the safety of the operating terminal.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and in particular to a security protection method, system, computer equipment and storage medium. Background Art

[0002] With the popularization of laparoscopic surgical robots, they have been applied to multiple departments such as urology, obstetrics and gynecology, cardiac surgery, thoracic surgery, and hepatobiliary surgery. The scenarios they face are becoming more and more complex, and at the same time, more potential safety hazards are exposed.

[0003] When a doctor operates a surgical robot through the main operating end to perform surgery, if a non-operating person (such as a nurse or assistant) accidentally hits the doctor's main operating end, it will cause unexpected movement of the surgical robot, causing major surgical risks.

[0004] Therefore, studying the safety technology of surgical robots and establishing a safe and reliable surgical robot protection system are important links to ensure the smooth progress of surgery and the safety of patients, and are of vital significance. Summary of the invention

[0005] Based on this, it is necessary to provide a security protection method, system, computer device and storage medium to provide security protection for the target operating terminal in response to the problems in the above-mentioned background technology.

[0006] In order to solve the above technical problems, the first aspect of the present application proposes a security protection method, comprising:

[0007] Determine the region of interest based on the image information collected in real time within the monitoring area of ​​the target operation end;

[0008] Based on the target detection algorithm, it is detected whether there is any non-operating personnel intruding into the area of ​​interest. If so, the corresponding warning action and / or protection action is triggered.

[0009] In the security protection method provided in the above embodiment, the area of ​​interest is determined based on the image information in the monitoring area around the target operating terminal collected in real time; and based on the target detection algorithm, it is detected whether non-operating personnel have invaded the area of ​​interest. If so, the corresponding early warning action and / or protection action is triggered to avoid major operational risks caused by non-operating personnel accidentally colliding with the target operating terminal, thereby improving the safety of the target operating terminal.

[0010] In one embodiment, determining the region of interest based on the image information in the monitoring area of ​​the target operating terminal acquired in real time includes:

[0011] Preprocessing the image information to obtain an image to be processed;

[0012] The region of interest is determined in the image to be processed.

[0013] In one embodiment, the preprocessing includes at least one of noise elimination, de-jitter processing and image distortion elimination.

[0014] In one embodiment, the de-jittering process includes:

[0015] Acquire multiple frames of images contained in the image information;

[0016] Determine one of the multiple frames of images as a reference frame, and determine other images except the reference frame as target frames;

[0017] Performing grayscale processing, binarization processing and image projection calculation on the multiple frames of images in sequence;

[0018] Calculating a correlation coefficient between an image projection calculation result of the target frame and an image projection calculation result of the reference frame;

[0019] The target frame is de-jittered according to the correlation coefficient.

[0020] In one embodiment, the step of obtaining and determining the region of interest in the image to be processed includes:

[0021] According to the interactive instruction input by the user, the region of interest is determined in the image to be processed.

[0022] In one embodiment, the method for detecting whether a non-operating person has intruded into the region of interest based on a target detection algorithm further includes:

[0023] It is determined whether the target operating terminal is in the working mode. If so, it is detected based on the target detection algorithm whether a non-operating person enters the area of ​​interest.

[0024] In one embodiment, the detecting whether a non-operating person has intruded into the region of interest based on a target detection algorithm includes:

[0025] Performing feature extraction on the image information in the region of interest based on a feature extraction network to obtain a feature extraction result;

[0026] The feature extraction result is processed by a preset classifier to determine whether there is a non-operating person in the area of ​​interest.

[0027] In one embodiment, the area of ​​interest includes a first warning area and a second warning area located inside the first warning area; and triggering corresponding warning actions and / or protection actions includes:

[0028] If it is detected that a non-operating person is in the first warning area, triggering a corresponding first warning action;

[0029] If it is detected that a non-operating person is in the second warning area, the corresponding second warning action and the protection action are triggered.

[0030] In one of the embodiments, the first warning action includes triggering a first warning light to flash, and / or issuing a voice prompt of a first trigger frequency;

[0031] The second warning action includes triggering a second warning light to flash, and / or issuing a voice prompt at a second trigger frequency;

[0032] The protection action includes triggering the target operating end to lock.

[0033] In one embodiment, it also includes:

[0034] Detect whether the light curtain pre-set around the target operating end is blocked, and if so, trigger the third warning action and the protection action.

[0035] In one embodiment, the third warning action includes triggering a third warning light to flash, and / or issuing a voice prompt of a third trigger frequency;

[0036] The protection action includes triggering the target operating end to lock.

[0037] The second aspect of the present application provides a security protection system, comprising:

[0038] The image monitoring module is configured as follows:

[0039] Determine the region of interest based on the image information collected in real time within the monitoring area of ​​the target operation end;

[0040] Detecting whether a non-operating person has invaded the region of interest based on a target detection algorithm;

[0041] The execution module is connected to the image monitoring module and is used to trigger corresponding warning actions and / or protection actions when a non-operating person intrudes into the area of ​​interest.

[0042] The security protection system provided in the above embodiment includes an image monitoring module and an execution module connected to the image monitoring module; the image monitoring module determines the area of ​​interest based on the image information in the monitoring area around the target operating terminal collected in real time; and detects whether there is a non-operating personnel intruding into the area of ​​interest based on the target detection algorithm; when the non-operating personnel intrudes into the area of ​​interest, the execution module triggers the corresponding early warning action and / or protection action to avoid major risks caused by the non-operating personnel accidentally colliding with the target operating terminal, thereby improving the safety of the operating terminal.

[0043] In one embodiment, the image monitoring module includes:

[0044] The image acquisition unit is configured to: acquire image information in the monitoring area of ​​the target operation terminal in real time, and pre-process the image information to obtain an image to be processed;

[0045] An image processing unit is connected to the image acquisition unit and the execution module, and is configured as follows:

[0046] Determining the region of interest in the image to be processed;

[0047] Performing feature extraction on the image information in the region of interest based on a feature extraction network to obtain a feature extraction result;

[0048] The feature extraction result is processed by a preset classifier to determine whether the non-operating personnel is in the region of interest.

[0049] In one embodiment, the system further comprises:

[0050] A light curtain is arranged around the target operation end;

[0051] The light curtain monitoring module is connected to the execution module and is configured to detect whether the light curtain is blocked, and if so, trigger the execution module to perform corresponding warning actions and / or protection actions.

[0052] A third aspect of the present application proposes a surgical robot system, comprising a main operating end and an operating trolley, wherein the main operating end is a target operating end, and the system also includes a safety protection system as described above.

[0053] A fourth aspect of the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0054] A fifth aspect of the present application provides a storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0055] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, drawings of other embodiments can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 A schematic diagram of a use scenario of a surgical robot system provided in one embodiment of the present application;

[0058] Figure 2 This is a schematic diagram of the structure of a doctor's main operating terminal provided in one embodiment of the present application;

[0059] Figure 3 A schematic diagram of a safety protection method provided in an embodiment of the present application;

[0060] Figure 4 A partial flow chart of a security protection method provided in one embodiment of the present application;

[0061] Figure 5 A schematic diagram of a process of image de-shaking provided in one embodiment of the present application;

[0062] Figure 6 A schematic diagram of a process of image de-shaking provided in another embodiment of the present application;

[0063] Figure 7 A schematic diagram of a process for obtaining a region of interest provided in an embodiment of the present application;

[0064] Figure 8 A partial flow chart of a security protection method provided in another embodiment of the present application;

[0065] Fig. 9 A schematic diagram of the principle of a convolutional feature extractor provided in one embodiment of the present application;

[0066] Fig.10 A K-means algorithm implementation flow chart provided in one embodiment of the present application;

[0067] Fig.11 A partial flow chart of a security protection method provided in another embodiment of the present application;

[0068] Fig.12 A partial flow chart of a security protection method provided in another embodiment of the present application;

[0069] Fig.13 A schematic diagram of the structure of a security protection system provided in one embodiment of the present application;

[0070] Fig.14 This is a schematic diagram of the structure of a security protection system provided in another embodiment of the present application;

[0071] Fig.15 A security protection scenario diagram provided in an embodiment of the present application;

[0072] Fig.16 A schematic diagram of locking protection of a doctor's main operating terminal provided in an embodiment of the present application;

[0073] Fig.17 A schematic diagram of an interactive interface provided in an embodiment of the present application;

[0074] Fig.18 A schematic diagram of the structure of a security protection system provided in another embodiment of the present application;

[0075] Fig.19 A protection scenario schematic diagram of a monitoring module provided in an embodiment of the present application;

[0076] Fig. 20 A schematic diagram of the structure of a computer device provided in one embodiment of the present application.

[0077] Description of reference numerals: 100, doctor's main operating terminal; 110, adjustment component; 120, manipulation arm; 130, trolley component; 140, imaging component; 200, operating trolley; 201, mechanical arm; 300, imaging trolley; 400, tool trolley;

[0078] 10. Image monitoring module; 11. Image acquisition unit; 12. Image processing unit; 20. Execution module; 30. Light curtain monitoring module; 31. Transceiver unit; 31. Monitoring unit. DETAILED DESCRIPTION

[0079] In order to facilitate the understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. The preferred embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive.

[0080] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0081] In the case of using “including”, “having”, and “comprising” described herein, another component may be added unless a clear limiting term such as “only”, “consisting of”, etc. is used. Unless mentioned otherwise, a term in the singular form may include a plural form and should not be understood as being one in number.

[0082] It should be understood that although the terms "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the present application.

[0083] In this application, unless otherwise clearly specified and limited, the terms "connected", "connection" and the like should be understood in a broad sense, for example, it can be directly connected or indirectly connected through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0084] In order to protect the safety of the target operating end, in traditional technology, one solution is to install a fixed mechanical guardrail around the target operating end to prevent non-operating personnel from approaching. However, since the mechanical guardrail is large in size, inconvenient to carry, and occupies space in the operating room, it is not possible to flexibly adjust the safety protection area according to different surgical scenarios, and is inconvenient to use.

[0085] Therefore, the present application proposes a safety protection method, system, computer equipment and storage medium, which monitor the monitoring area of ​​the target operating end in real time, and determine the area of ​​interest based on the image information in the monitoring area of ​​the target operating end collected in real time, and then detect whether there is any non-operating personnel invading the said area of ​​interest based on the target detection algorithm. If so, the corresponding early warning action and / or protection action is triggered, so as to avoid non-operating personnel accidentally colliding with the target operating end and causing unexpected movement of the target operating end, causing major surgical risks, and improving the safety of the target operating end. At the same time, the area of ​​interest can be adjusted in real time according to the actual scene needs, saving the occupied operating room space compared with mechanical guardrails.

[0086] The security protection method, system, computer device and storage medium provided in this application can be applied to Figure 1The schematic diagram of the use scenario of the surgical robot system shown in FIG. The surgical robot system includes a doctor's main operation terminal 100, an operating trolley 200, an image trolley 300 and a tool trolley 400, and a master operator is provided on the doctor's main operation terminal 100. The operating trolley 200 has at least two mechanical arms 201, and surgical instruments and endoscopes can be mounted on the mechanical arms respectively. Minimally invasive surgical treatment of patients on the bed is achieved through remote operation by the doctor's main operation terminal 100 and the master operator. Among them, the master operator and the mechanical arm 201 and the surgical instrument form a master-slave control relationship. The doctor's main operation terminal 100 has a display device, which is connected to the endoscope mounted on the mechanical arm of the operating trolley 200 for communication, and can receive and display the image collected by the endoscope. The operator controls the movement of the mechanical arm and the surgical instrument through the master operator according to the image displayed by the display device on the doctor's main operation terminal 100. The endoscope and the surgical instrument enter the patient's position through the wound on the patient's body respectively.

[0087] Figure 2 The figure is a schematic diagram of the specific structure of the main operation terminal 100 of the surgical robot. The main operation terminal 100 of the doctor includes an adjustment component 110, a manipulation arm 120, a trolley component 130 and an image component 140. The two manipulation arms 120 detect the operator's hand motion information through the control handles at their ends as the motion control input of the entire system; the trolley part 130 is a basic bracket for installing other components, and the main operation terminal vehicle has movable casters and can be moved or fixed as needed; the adjustment component 110 can electrically adjust the position of the manipulation arm, image component, operator armrest and other devices, that is, the human-machine parameter adjustment function. The image component 140 can provide the operator with a stereoscopic image detected from the image system, providing the operator with reliable image information for performing surgical operations.

[0088] It should be noted that the security protection method and system of this embodiment can be applied not only to the above Figure 1 The usage scenario of the surgical robot system shown can also be applied to other operating ends that require real-time monitoring, not limited to operating room scenarios.

[0089] In a security protection method provided in one embodiment of the present application, as Figure 3 As shown, the following steps are included:

[0090] Step S10: determining an area of ​​interest based on image information collected in real time within the monitoring area of ​​the target operating terminal;

[0091] Step S20: Based on the target detection algorithm, detect whether there is any non-operating personnel intruding into the area of ​​interest, and if so, trigger corresponding warning actions and / or protection actions.

[0092] When the present invention is applied to Figure 1In the case of the surgical robot system shown in FIG. 1 , the target operation end in step S10 refers to Figure 1 The main operation terminal 100 in the embodiment. In the safety protection method provided in the above embodiment, the region of interest is determined based on the image information in the monitoring area around the doctor's main operation terminal collected in real time; and based on the target detection algorithm, it is detected whether non-operating personnel invade the region of interest. If so, the corresponding warning action and / or protection action is triggered to avoid major surgical risks caused by non-operating personnel accidentally hitting the main operation terminal, thereby improving the safety of the doctor's main operation terminal.

[0093] As an example, the size of the monitoring area of ​​the main operating terminal is determined according to the detection area of ​​the monitoring camera, and an area of ​​interest is selected within the monitoring area; the area of ​​the area of ​​interest is less than or equal to the detection area of ​​the monitoring camera.

[0094] As an example, the region of interest (ROI) includes a closed area, and the shape of the region of interest includes a closed area of ​​a preset shape, which can be a circle, a polygon, an ellipse, etc.; it can be a regular shape or an irregular shape; the occupied area of ​​the region of interest can be changed according to the size of the operating room space and different surgical scenarios.

[0095] As an example, the protection action includes triggering the doctor's main operating terminal 100 to lock the protection; the warning action includes voice prompts and / or flashing warning lights with different levels of trigger frequency.

[0096] In one embodiment, Figure 4 As shown, step S10: determining the region of interest based on the image information in the monitoring area of ​​the target operating terminal acquired in real time, includes the following steps:

[0097] Step S11: pre-processing the image information to obtain an image to be processed;

[0098] Step S12: determining a region of interest in the image to be processed.

[0099] Specifically, the preprocessing includes at least one of noise elimination, de-jitter processing and image distortion elimination.

[0100] As an example, the acquired image information is subjected to noise elimination to remove high-frequency and burr image points in the image information; image distortion elimination can be achieved by calibrating the surveillance camera to obtain camera intrinsic parameters including distortion coefficients, and then a dedistorted image is obtained based on the camera intrinsic parameters.

[0101] In one embodiment, a surveillance camera is used to collect image information of the surveillance area, and the image information can be stored in a hard disk. Due to the instability of the external environment, the image is prone to shaking, which is not conducive to subsequent operation and processing. The image information needs to be de-jittered to eliminate the shaking caused by the shaking of the device.

[0102] Specifically, Figure 5 As shown, the de-jitter process includes the following steps:

[0103] Step S111: acquiring multiple frames of images contained in the image information;

[0104] Step S112: determining one of the multiple frames of images as a reference frame (for example, taking the first frame as the reference frame), and determining the other images except the reference frame as target frames;

[0105] Step S113: performing grayscale processing, binarization processing and image projection calculation on multiple frames of images in sequence;

[0106] Step S114: calculating the correlation coefficient between the image projection calculation result of the target frame and the image projection calculation result of the reference frame;

[0107] Step S115: performing de-jitter processing on each target frame according to the calculated correlation coefficient.

[0108] Specifically, the correlation coefficient includes a correlation coefficient of a vertical projection and a correlation coefficient of a horizontal projection; starting from the first frame image, the frame number is incremented by 1 each time the image is processed until all the image processing is completed, and multiple frames of images are obtained, and the reference frame and the target frame are determined; firstly, grayscale processing, binarization processing and image horizontal projection calculation are performed on the multiple frames of images in sequence, and the horizontal projection calculation results of the target frame and the reference frame are calculated respectively, and the area with the largest correlation coefficient with the horizontal calculation result of the reference frame is determined, and the area with the largest horizontal projection correlation coefficient is obtained, which can be multiple target frame horizontal projection images, thereby eliminating the jitter in the horizontal direction of the surveillance camera; from the multiple target frame horizontal projection images contained in the area with the largest horizontal projection correlation coefficient, one of the frames is selected as the vertical reference frame and the vertical target frame, and the vertical projection calculation is performed on the vertical reference frame and the vertical target frame, and the vertical target frame projection calculation results and the vertical reference frame projection calculation results are calculated respectively, and the area with the largest correlation coefficient with the vertical reference frame projection calculation result is determined, and the area with the largest vertical projection correlation coefficient is obtained, so as to eliminate the jitter in the vertical direction of the surveillance camera, and the area with the largest vertical projection correlation coefficient is the image to be processed. In this embodiment, the aforementioned correlation coefficient may refer to the Pearson correlation coefficient.

[0109] As an example, see Figure 6 (exist Figure 6The upper row in the middle shows the process steps of de-jittering, and the lower row shows the specific implementation methods of the corresponding steps). The grayscale processing is implemented by the weighted average method; the binarization processing includes adaptive local binarization; the projection calculation includes column summation of the image data; the Pearson correlation coefficient is used to calculate the correlation coefficient between the image projection calculation results of the target frame and the image projection calculation results of the reference frame, so as to repair the image anomalies caused by jitter.

[0110] In one embodiment, step S12: determining a region of interest in the image to be processed, includes: determining the region of interest in the image to be processed according to an interactive instruction input by a user.

[0111] Specifically, the interaction instructions input by the user include mouse action instructions and / or touch screen instructions.

[0112] Take getting mouse action instructions as an example, Figure 7 As shown, determining the region of interest in step S12 includes:

[0113] Step S121: obtaining mouse action instructions;

[0114] Step S122: judging whether the left button is clicked according to the mouse action instruction;

[0115] Step S123: if the left button is clicked, determine whether it is the first time to click the left button;

[0116] Step S124: if it is the first time to click the left button, the position of the first click of the left button is recorded as the starting vertex of the region of interest and marked in the image to be processed; if it is not the first time to click the left button, the corresponding current position is recorded as the vertex of the region of interest, and the current position is connected to the previous vertex of the region of interest, and it is continued to determine whether the left button is clicked;

[0117] Step S125: If the left button is not clicked, determine whether the right button is clicked;

[0118] Step S126: if the right button is clicked, the last point where the left button is clicked is connected to the starting vertex of the region of interest to obtain the region of interest; if the right button is not clicked, whether the mouse is moved is determined;

[0119] Step S127: If the mouse moves, determine the current coordinate value of the mouse on the image to be processed; if the mouse does not move, continue to determine whether the left button is clicked.

[0120] Optionally, several graphics can be pre-configured. After selecting the required graphic box, the position can be determined by dragging the graphic box, and the size of the graphic can be adjusted by pulling the image border to determine the area of ​​interest. Of course, the area of ​​interest can also be determined in other ways, and this embodiment does not make specific limitations on this.

[0121] In one embodiment, the method for detecting whether a non-operating person has invaded the region of interest based on a target detection algorithm in step S20 further includes:

[0122] Step S101: determine whether the target operating terminal 100 is in working mode. If so, detect whether a non-operating person enters the area of ​​interest based on the target detection algorithm. Otherwise, do not detect whether a non-operating person enters the area of ​​interest.

[0123] When the target operating end 100 is not in working mode, there is no need to prohibit non-operating personnel from entering the vicinity of the target operating end, and thus the step of detecting whether non-operating personnel have invaded the area of ​​interest according to the target detection algorithm is not performed, so as to avoid triggering warning actions and / or protection actions to disturb the staff.

[0124] In one embodiment, Figure 8 As shown, step S20: detecting whether a non-operating person has invaded the region of interest based on a target detection algorithm includes the following steps:

[0125] Step S21: extracting features of image information in the region of interest based on a feature extraction network to obtain feature extraction results;

[0126] Step S22: Process the feature extraction result through a preset classifier to determine whether there is a non-operating person in the area of ​​interest.

[0127] As an example, the target detection algorithm includes a feature extraction network and a preset classifier; the feature extraction network realizes feature extraction of image information within the region of interest, and does not extract images outside the region of interest, thereby reducing image processing time. The feature extraction network includes a convolutional feature extractor and a Darkne-53 network of YOLOv3; Fig.10 The figure is a schematic diagram of the convolutional feature extractor. The convolutional feature extractor is mainly achieved by alternating 1*1 and 3*3 convolution kernels. The 1*1 convolution kernel is used for dimensionality reduction, and the 3*3 convolution kernel is used for feature extraction. The Darknet-53 network consists of a series of 1*1 and 3*3 convolution layers. Each convolution layer is followed by a BN layer and a LeakyReLU layer. The entire network has a total of 53 layers, including the final fully connected layer, to improve detection accuracy.

[0128] As an example, Fig.11 The flowchart for the K-means algorithm is shown in Figure 1. The classifier classification is implemented using the K-means algorithm, which includes the following steps:

[0129] Step S211: Generate K random positions as initial centroids, each centroid replaces a cluster center;

[0130] Step S212: Calculate the Euclidean distance from each object to the cluster center, and classify the objects to the nearest cluster center according to the distance.

[0131] Step S213: accumulating and calculating the mean of the objects in each class as a new cluster center;

[0132] Step S214: determining whether the new cluster center achieves a smaller distance loss;

[0133] If yes, execute step S212;

[0134] If not, output the result.

[0135] The above classifier classification K-means algorithm improves the accuracy of judging whether the non-operating personnel are in the area of ​​interest, and can accurately distinguish whether the non-operating personnel are in the first warning area or in the second warning area.

[0136] In one embodiment, Fig.11 As shown, the area of ​​interest includes a first warning area and a second warning area located inside the first warning area; step S20: triggering corresponding warning actions and / or protection actions, including:

[0137] Step S23: If it is detected that a non-operating person is in the first warning area, triggering a corresponding first warning action;

[0138] Step S24: If it is detected that a non-operating person is in the second warning area, the corresponding second warning action and protection action are triggered.

[0139] As an example, the first warning action includes triggering the first warning light to flash, and / or issuing a voice prompt with a first trigger frequency; the second warning action includes triggering the second warning light to flash, and / or issuing a voice prompt with a second trigger frequency; after the non-operating personnel exits the area of ​​interest, the first warning action and the second warning action are stopped; the protection action includes triggering the target operating end to lock, and after the non-operating personnel exits the second warning area, the locking protection is stopped.

[0140] As an example, the second trigger frequency is greater than the first trigger frequency; the first warning light and the second warning light may have different colors, and may have different brightness for triggering flashing.

[0141] It should be noted that the region of interest can also be divided into multiple warning areas, and the number of the multiple warning areas is greater than or equal to 3; it can be adjusted according to the size of the operating room and the surgical scene, and this application does not limit this.

[0142] The security protection method provided in the above embodiment triggers the corresponding warning action by monitoring which warning area the non-operating personnel is in. When the non-operating personnel is in the first warning area, the first warning action is triggered; when the non-operating personnel is in the second warning area, the second warning action is triggered. The priority of the second warning action is greater than the first warning action, forming a graded warning mechanism with different risk levels to remind non-operating personnel to stay away from the area of ​​interest and improve the safety of the target operating end.

[0143] In one embodiment, Fig.12 As shown, the security protection method also includes the following steps:

[0144] Step S30: Detect whether the light curtain pre-set around the target operating end is blocked, and if so, trigger the third warning action and protection action.

[0145] Specifically, the third warning action includes triggering the third warning light to flash, and / or issuing a voice prompt with a third trigger frequency, and stopping the flashing of the third color light and the voice prompt of the third trigger frequency when the light curtain is not blocked; the protection action includes triggering the main target operating end 100 to lock, and when the non-operating personnel exit the light curtain area, the target operating end 100 stops locking protection and returns to normal operation.

[0146] As an example, the third trigger frequency is greater than the second trigger frequency; the color and / or brightness of the third warning light are different from the first warning light and the second warning light.

[0147] As an example, the area of ​​the light curtain is smaller than or equal to the area of ​​the second warning area; the light curtain is located inside the second warning area.

[0148] As an example, the formed light curtain includes an electronic fence formed by visible rays that can be blocked by non-operating personnel, wherein the visible rays include laser or infrared and the like.

[0149] In the light curtain monitoring provided in the above embodiment, when the image monitoring cannot monitor in real time whether the non-operating personnel enter the area of ​​interest due to external factors, the third warning action and the protection action are triggered through real-time monitoring by the light curtain to improve the safety of the target operating end; the priority of the light curtain monitoring is higher than the priority of the image monitoring. The image monitoring and the light curtain monitoring together form a safety protection mechanism for the target operating end.

[0150] In one embodiment of the present application, Fig.13As shown, a security protection system is provided to execute the security protection method as described above, and the security protection system includes an image monitoring module 10 and an execution module 20 connected to the image monitoring module. The image monitoring module 10 is configured to: determine the region of interest based on the image information in the monitoring area of ​​the target operation end collected in real time; detect whether there is a non-operating person invading the region of interest based on the target detection algorithm; the execution module 20 is used to trigger corresponding warning actions and / or protection actions when the non-operating person intrudes the region of interest.

[0151] In the safety protection system provided in the above embodiment, there is an image monitoring module and an execution module connected to the image monitoring module; the image monitoring module provides image information in the monitoring area around the doctor's target operating terminal collected in real time to determine the area of ​​interest; and detects whether there is a non-operating personnel invading the area of ​​interest based on the target detection algorithm; when a non-operating personnel intrudes into the area of ​​interest, the execution module triggers corresponding warning actions and / or protection actions to avoid major risks caused by non-operating personnel accidentally hitting the target operating terminal, thereby improving the safety of the target operating terminal.

[0152] As an example, the execution module 20 includes an audible and visual alarm for executing the early warning action.

[0153] In one embodiment, the area of ​​interest includes a first warning area and a second warning area located inside the first warning area; the execution module 20 includes a locking protection unit for triggering the target operating end 100 to perform a locking protection action when a non-operating person invades the second warning area.

[0154] In one embodiment, Fig.14 As shown, the image monitoring module 10 includes an image acquisition unit 11 and an image processing unit 12. The image acquisition unit 11 is configured to: acquire image information in the monitoring area in real time, and pre-process the image information to obtain an image to be processed; the image processing unit 12 is connected to both the image acquisition unit 11 and the execution module 20, and the image processing unit 12 is configured to: determine an area of ​​interest in the image to be processed; extract features of the image information in the area of ​​interest based on a feature extraction network to obtain a feature extraction result; process the feature extraction result through a preset classifier to determine whether there is a non-operating person in the area of ​​interest.

[0155] Specifically, Fig.15 As shown, if the image processing unit 12 detects that a non-operating person is in the first warning area, the execution module 20 is triggered to execute the corresponding first warning action; if the non-operating person is detected to be in the second warning area, the execution module 20 is triggered to execute the corresponding second warning action and protection action.

[0156] Specifically, the execution module 20 is further configured to: trigger the first warning light to flash, and / or issue a voice prompt of a first trigger frequency; or, trigger the second warning light to flash, and / or issue a voice prompt of a second trigger frequency, until the non-operating personnel exit the second warning area and stop the flashing of the second warning light and the voice prompt of the second trigger frequency, and the non-operating personnel exit the first warning area and stop the flashing of the first warning light and the voice prompt of the first trigger frequency; the execution module 20 is configured to: trigger the locking protection of the target operating end according to the second warning information, such as Fig.16 As shown, the locking protection is stopped until the non-operating personnel exit the second warning area and trigger the main operating terminal 100 to stop.

[0157] As an example, the security protection system also includes a UPS power supply, which provides a stable and uninterrupted power supply for the security protection system. The image acquisition unit 11 includes a monitoring camera and a hard disk, and the camera acquires image information, transmits it via an internal video line, and stores it in the hard disk.

[0158] In one embodiment, Fig.17 As shown, an interactive interface is provided on the image processing module 10; the interactive interface is configured to: execute corresponding preset actions according to received control instructions; the control instructions include at least one of reference frame selection instructions, jitter removal instructions, region of interest selection instructions and target detection instructions; the preset actions include at least one of reference frame selection, jitter removal, region of interest selection and target detection.

[0159] In one embodiment, the interactive interface is further configured such that when the image processing module 10 detects a non-operating person intruding into the area of ​​interest, the display frame of the interactive interface flashes and displays a prompt message; wherein the interactive interface is electrically connected to the image processing unit 12 .

[0160] As an example, please refer to Fig.17 The interactive interface also includes a "Confirm" button and a "Cancel" button; after receiving the control instruction, the interactive interface triggers the "Confirm" button before executing the corresponding preset action; if you want to cancel or interrupt any preset action, trigger the "Cancel" button.

[0161] In one embodiment, Fig.18 As shown, the safety protection system further includes: a light curtain and a light curtain monitoring module 30. The light curtain (not shown) is arranged around the target operation end (such as the main operation end 100); the light curtain monitoring module 30 is connected to the execution module 20 and is configured to: detect whether the light curtain pre-arranged around the target operation end is blocked, and if so, trigger the execution module 20 to execute the corresponding third warning action and / or protection action.

[0162] Specifically, Fig. 20As shown, the light curtain monitoring module 30 includes a transceiver unit 31 and a monitoring unit 32. The transceiver unit 31 is arranged on the target operation end, and is used to transmit and receive visible rays to form a light curtain; the monitoring unit 32 is integrated in the target operation end and connected to the execution module 20, and is used to monitor whether the light curtain is blocked, and generate a monitoring signal when it is detected that the light curtain is blocked.

[0163] As examples, visible rays include laser or infrared, etc.; light curtains include electronic fences.

[0164] In one embodiment, the execution module 20 is also used to trigger the flashing of the third warning light according to the monitoring signal, and to issue a voice prompt of the third trigger frequency, and to stop the flashing of the third warning light and the voice prompt of the third trigger frequency when the light curtain is not blocked; the execution module 20 is also used to trigger the locking protection of the target operating end according to the monitoring signal, and to control the locking protection of the target operating end when the light curtain is not blocked.

[0165] In one embodiment of the present application, a surgical robot system is also proposed, including a main operating end, a surgical trolley, and the safety protection system as described above, wherein the main operating end serves as a target operating end protected by the safety protection system. The surgical robot system may be, for example, Figure 1 The surgical robot system shown in the figure has a main operating terminal such as Figure 1 Of course, the present application does not limit the specific form of the surgical robot.

[0166] In one embodiment of the present application, a computer device is also provided, such as Fig. 20 As shown, it includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above method are implemented.

[0167] Those skilled in the art will understand that Fig. 20 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0168] In one embodiment of the present application, a storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0169] It should be understood that, unless otherwise specified herein, the execution of the steps described is not strictly limited in order, and the steps may be executed in other orders. Moreover, at least a portion of the steps described may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0170] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0171] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0172] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A security protection method, characterized in that: include: Determine the region of interest based on the image information in the monitoring area of ​​the target operating terminal acquired in real time, wherein the target operating terminal is the doctor's main operating terminal; Detecting whether a non-operating person has invaded the area of ​​interest based on a target detection algorithm, and if so, triggering corresponding warning actions and / or protection actions; When it is impossible to detect whether a non-operating person has invaded the area of ​​interest based on the target detection algorithm, detecting whether a light curtain pre-set around the target operating end is blocked, and if so, triggering a third warning action and the protection action; The detecting whether a non-operating person has invaded the region of interest based on a target detection algorithm includes: Performing feature extraction on the image information in the region of interest based on a feature extraction network to obtain a feature extraction result; The feature extraction result is processed by a preset classifier to determine whether there is a non-operating person in the area of ​​interest, including: generating K random positions as initial centroids, each centroid replacing a cluster center; calculating the Euclidean distance from each object to the cluster center, and classifying it to the nearest cluster center according to the length of the distance; accumulating the objects in each class and calculating the mean as a new cluster center; judging whether the new cluster center achieves a smaller distance loss; if so, executing: calculating the Euclidean distance from each object to the cluster center, and classifying it to the nearest cluster center according to the length of the distance; if not, outputting the result.

2. The method according to claim 1, characterized in that Determining the region of interest based on the image information in the monitoring area of ​​the target operating terminal acquired in real time includes: Preprocessing the image information to obtain an image to be processed; The region of interest is determined in the image to be processed.

3. The method according to claim 2, characterized in that The preprocessing includes at least one of noise elimination, de-jitter processing and image distortion elimination.

4. The method according to claim 3, characterized in that The de-jittering process includes: Acquire multiple frames of images contained in the image information; Determine one of the multiple frames of images as a reference frame, and determine other images except the reference frame as target frames; Performing grayscale processing, binarization processing and image projection calculation on the multiple frames of images in sequence; Calculating a correlation coefficient between an image projection calculation result of the target frame and an image projection calculation result of the reference frame; The target frame is de-jittered according to the correlation coefficient.

5. The method according to claim 2, characterized in that: The step of determining the region of interest in the image to be processed includes: According to the interactive instruction input by the user, the region of interest is determined in the image to be processed.

6. The method according to any one of claims 1 to 5, characterized in that: The method for detecting whether a non-operating person has invaded the region of interest based on a target detection algorithm further includes: It is determined whether the target operating terminal is in the working mode. If so, it is detected based on the target detection algorithm whether a non-operating person enters the area of ​​interest.

7. The method according to any one of claims 1 to 5, characterized in that: The area of ​​interest includes a first warning area and a second warning area located inside the first warning area; The triggering of corresponding warning actions and / or protection actions includes: If it is detected that a non-operating person is in the first warning area, triggering a corresponding first warning action; If it is detected that a non-operating person is in the second warning area, the corresponding second warning action and the protection action are triggered.

8. The method according to claim 7, characterized in that The first warning action includes triggering a first warning light to flash, and / or issuing a voice prompt at a first trigger frequency; The second warning action includes triggering a second warning light to flash, and / or issuing a voice prompt at a second trigger frequency; The protection action includes triggering the target operating end to lock.

9. The method according to claim 1, characterized in that: The third warning action includes triggering the third warning light to flash, and / or issuing a voice prompt with a third trigger frequency; The protection action includes triggering the target operating end to lock.

10. A safety protection system, characterized in that: include: The image monitoring module is configured as follows: Determine the region of interest based on the image information in the monitoring area of ​​the target operating terminal acquired in real time, wherein the target operating terminal is the doctor's main operating terminal; Detecting whether a non-operating person has invaded the region of interest based on a target detection algorithm includes: extracting features from the image information in the region of interest based on a feature extraction network to obtain a feature extraction result; processing the feature extraction result through a preset classifier to determine whether a non-operating person is in the region of interest, including: generating K random positions as initial centroids, each centroid replacing a cluster center; calculating the Euclidean distance from each object to the cluster center, and classifying to the nearest cluster center according to the length of the distance; accumulating and calculating the mean of the objects in each class as a new cluster center; judging whether the new cluster center has a smaller distance loss; if so, executing: calculating the Euclidean distance from each object to the cluster center, and classifying to the nearest cluster center according to the length of the distance; if not, outputting the result; An execution module, connected to the image monitoring module, for triggering corresponding warning actions and / or protection actions when a non-operating person intrudes into the area of ​​interest; A light curtain is arranged around the target operating end; The light curtain monitoring module is connected to the execution module and is configured to: when it is impossible to detect whether a non-operating person has intruded into the area of ​​interest based on the target detection algorithm, detect whether the light curtain is blocked, and if so, trigger the execution module to perform corresponding warning actions and / or protection actions.

11. The system according to claim 10, characterized in that The image monitoring module comprises: The image acquisition unit is configured to: acquire image information in the monitoring area of ​​the target operation terminal in real time, and pre-process the image information to obtain an image to be processed; An image processing unit is connected to the image acquisition unit and the execution module, and is configured as follows: The region of interest is determined in the image to be processed.

12. A surgical robot system, comprising a main operating end and a surgical trolley, wherein: The main operation terminal is a target operation terminal, characterized in that it also includes the safety protection system as described in claim 10 or 11.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

14. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images