Personnel operation behavior identification method and system based on computer vision

Abstract

The invention relates to the technical field of image processing, and discloses a personnel operation behavior recognition method and system based on computer vision, and the system comprises an image data collection module, an image data analysis module, and a personnel operation behavior recognition early warning module. The image data acquisition module comprises an image definition judgment unit, a wearing behavior recognition unit, a cleaning behavior recognition unit and an operation behavior recognition unit. Wearing image data are acquired by reading a camera arranged in a wearing area, cleaning image data are acquired by a camera arranged in a cleaning area, and operation image data are acquired by reading a camera arranged in an operation area, so that behaviors of personnel are identified; whether the first early-warning instruction is executed or not is judged based on the cleaning image data, whether the second early-warning instruction is executed or not is judged based on the wearing image data, whether the third early-warning instruction is executed or not is judged based on the operation image data, whether the medical staff is standard or not and the fatigue degree can be accurately judged according to operation behaviors of the medical staff, and safe operation is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and system for identifying human operation behavior based on computer vision. Background Art

[0002] The operating room (OR) is a place for patients to undergo surgery and rescue, and is an important technical department of the hospital. The operating room should be connected to the surgical department, and should also be close to the blood bank, intensive care unit, anesthesia recovery room, etc. The four ways of surgical wound infection should be managed well, namely: the air in the operating room; the items required for the operation; the fingers of doctors and nurses and the skin of patients to prevent infection and ensure the success rate of the operation. It is required to have a reasonable design, complete equipment, and nurses who are responsive and quick, and have high work efficiency. The operating room must have a set of strict and reasonable rules and regulations and aseptic operation specifications. With the rapid development of surgical technology, the work of the operating room is becoming more and more modern. At present, there are still problems such as relatively subjective evaluation of the fatigue level of doctors and nurses during surgery and untimely monitoring of operation specifications. Summary of the invention

[0003] 1. Technical issues to be resolved In view of the deficiencies in the prior art, the present invention provides a method and system for identifying personnel operation behaviors based on computer vision, which has the advantages of identifying personnel operation behaviors and issuing early warnings, thereby solving the above-mentioned technical problems.

[0004] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: A personnel operation behavior recognition system based on computer vision, comprising an image data acquisition module, an image data analysis module and a personnel operation behavior recognition and warning module; The image data acquisition module includes an image clarity judgment unit, a wearing behavior recognition unit, a cleaning behavior recognition unit and an operation behavior recognition unit. The wearing behavior recognition unit obtains wearing image data by reading a camera set in a wearing area, the cleaning behavior recognition unit obtains cleaning image data by reading a camera set in a cleaning area, and the operation behavior recognition unit obtains operation image data by reading a camera set in an operation area. The image clarity judgment unit is used to read each camera and calculate the clarity, and then judge whether to execute the clarity warning instruction according to the clarity, and send the wearing image data, cleaning image data and operation image data to the image data analysis module when the clarity warning instruction is not executed; The image data analysis module determines whether to execute the first warning instruction based on the cleaning image data, the image data analysis module determines whether to execute the second warning instruction based on the wearing image data, and the image data analysis module determines whether to execute the third warning instruction based on the operation image data. When the first warning instruction, the second warning instruction and the third warning instruction are executed, they jump to the personnel operation behavior recognition warning module, and the execution order of the first warning instruction, the second warning instruction and the third warning instruction is to execute them in sequence; The personnel operation behavior identification and warning module respectively issues warnings according to the clarity warning instruction, the first warning instruction, the second warning instruction and the third warning instruction.

[0005] As a preferred technical solution of the present invention, the specific expression of the cleaning behavior recognition unit obtaining cleaning image data by reading the camera set in the cleaning area is as follows: ; in, Represents a clear data set, They represent the first image of the person doing the cleaning, ..., the first image of the person doing the cleaning, taken by the camera set in the cleaning area. The image of the person being cleaned, ..., Person images to be cleaned, and the Images of people doing cleaning , Respectively represent The first cleaning step is performed by the cleaning personnel, ..., cleaning step, ..., The cleaning steps are identified and labeled by OpenPose. Respectively represent The first cleaning step by the cleaning personnel takes time, ..., The cleaning step takes time, ... The cleaning step is time consuming.

[0006] As a preferred technical solution of the present invention, the expression for the image data analysis module to judge whether to execute the first warning instruction based on the cleaned image data is as follows: ; in, represents the cleaning judgment coefficient, Indicates that the first warning instruction will not be executed. Indicates execution of the first warning instruction. Indicates the minimum value of the cleaning step, Indicates the minimum cleaning time. express The total time for each cleaning step.

[0007] As a preferred technical solution of the present invention, the specific expression of the wearing behavior recognition unit obtaining the wearing image data by reading the camera set in the wearing area is as follows: ; in, represents the wearable image dataset, They respectively represent the first image of the person who has completed wearing the clothing, ..., the first image of the person who has completed wearing the clothing, taken by the camera set in the wearing area. The image of the person who has finished wearing the clothes, ... The image of the person who has finished wearing the clothes, among which Completed wearing personnel images , Indicates the image data taken just after entering the wearable area. Indicates the image data captured when the donning is completed and the wearer enters the donning area.

[0008] As a preferred technical solution of the present invention, the expression for the image data analysis module to judge whether to execute the second warning instruction based on the wearing image data is as follows: ; in, It indicates the area where the pixels change between the image data taken just after entering the wearing area and the image data taken after entering the wearing area. represents the area change rate, Represents the image area, when the area change rate Minimum area change When the area change rate is Minimum area change The second warning instruction will not be executed.

[0009] As a preferred technical solution of the present invention, the specific expression of the operation behavior recognition unit obtaining the operation image data by reading the camera set in the operation area is as follows: ; in, Represents the operation data set, They respectively represent the first image of the person performing the operation, ..., the first image of the person performing the operation, taken by the camera set in the wearing area. The image of the person performing the operation, ... images of people performing operations, among which , Indicates The current closing moment of the operator’s eyes, Indicates The Dlib model is used to identify the last time a person opened his eyes.

[0010] As a preferred technical solution of the present invention, the specific expression of the image data analysis module for judging whether to execute the third warning instruction based on the operation image data is as follows: ; in, represents the operation judgment coefficient, Indicates The operator closes his eyes for a moment. Indicates When the operator opens his eyes, represents the blink threshold, Determine and execute the third warning instruction. It is determined that the third warning instruction will not be executed.

[0011] As a preferred technical solution of the present invention, the image clarity judgment unit is used to read each camera and calculate the clarity, and then the specific steps of judging whether to execute the clarity warning instruction according to the clarity are as follows: Step A1: read the signal-to-noise ratio of the camera and determine whether the signal-to-noise ratio is lower than the signal-to-noise ratio threshold. If it is lower than the signal-to-noise ratio threshold, issue a clarity warning instruction. If it is higher than the signal-to-noise ratio threshold, execute step A2. Step A2: Calculate the image brightness value. If the brightness value is lower than the brightness threshold, issue a clarity warning instruction. If the brightness value is higher than the brightness threshold, no clarity warning is issued.

[0012] As a preferred technical solution of the present invention, the personnel operation behavior identification and warning module warns the supervisor according to the clarity warning instruction, the first warning instruction, the second warning instruction and the third warning instruction respectively, and marks the corresponding camera that issues the warning and transmits it to the supervisor.

[0013] The present invention also provides a method for identifying human operation behavior based on computer vision, based on the above-mentioned human operation behavior identification system based on computer vision, comprising the following steps: Step 1: acquiring wearing image data based on the camera set in the wearing area, acquiring cleaning image data based on the camera set in the cleaning area, and acquiring operating image data based on the camera set in the operating area; Step 2: Calculate the clarity of each camera, and then determine whether to execute the clarity warning instruction based on the clarity; Step 3: judging whether to execute the first warning instruction based on the cleaned image data; Step 4: Determine whether to execute the second warning instruction based on the wearable image data; Step 5: judging whether to execute the third warning instruction based on the operation image data; Step 6: Issue an early warning.

[0014] Compared with the prior art, the present invention provides a method and system for identifying human operation behavior based on computer vision, which has the following beneficial effects: The present invention identifies personnel behavior by reading a camera set in the wearing area to obtain wearing image data, a camera set in the cleaning area to obtain cleaning image data, and a camera set in the operation area to obtain operation image data, and determines whether to execute the first warning instruction based on the cleaning image data, whether to execute the second warning instruction based on the wearing image data, and whether to execute the third warning instruction based on the operation image data. In this way, the fatigue degree of medical staff and whether the operation behavior is standardized can be accurately judged according to their operation behavior, thereby ensuring the safe conduct of the operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the framework of the personnel operation behavior recognition system based on computer vision of the present invention; Figure 2 The figure is a flow chart of the method for identifying personnel operation behavior based on computer vision of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] See also Figure 1 , a personnel operation behavior recognition system based on computer vision, including an image data acquisition module, an image data analysis module and a personnel operation behavior recognition and warning module; The image data acquisition module includes an image clarity judgment unit, a wearing behavior recognition unit, a cleaning behavior recognition unit, and an operation behavior recognition unit. The wearing behavior recognition unit obtains the wearing image data by reading the camera set in the wearing area. The specific expression of the wearing behavior recognition unit obtaining the wearing image data by reading the camera set in the wearing area is as follows: ; in, represents the wearable image dataset, They respectively represent the first image of the person who has completed wearing the clothing, ..., the first image of the person who has completed wearing the clothing, taken by the camera set in the wearing area. The image of the person who has finished wearing the clothes, ... The image of the person who has finished wearing the clothes, among which Completed wearing personnel images , Indicates the image data taken just after entering the wearable area. Indicates the image data captured after the wearer enters the wear area. The cleaning behavior recognition unit obtains the cleaning image data by reading the camera set in the cleaning area. The specific expression of the cleaning behavior recognition unit obtaining the cleaning image data by reading the camera set in the cleaning area is as follows: ; in, Represents a clear data set, They represent the first image of the person doing the cleaning, ..., the first image of the person doing the cleaning, taken by the camera set in the cleaning area. The image of the person being cleaned, ..., Person images to be cleaned, and the Images of people doing cleaning , Respectively represent The first cleaning step is performed by the cleaning personnel, ..., cleaning step, ..., The cleaning steps are identified and labeled by OpenPose. Respectively represent The first cleaning step by the cleaning personnel takes time, ..., The cleaning step takes time, ... The cleaning step is time-consuming. The operation behavior recognition unit obtains the operation image data by reading the camera set in the operation area. The specific expression of the operation behavior recognition unit obtaining the operation image data by reading the camera set in the operation area is as follows: ; in, Represents the operation data set, They respectively represent the first image of the person performing the operation, ..., the first image of the person performing the operation, taken by the camera set in the wearing area. The image of the person performing the operation, ... images of people performing operations, among which , Indicates The current closing moment of the operator’s eyes, Indicates The Dlib model is used to identify the blink of an eye when the operator last opened his eyes. The Dlib model is a model based on OpenCV / Python / dlib, which is a prior art and will not be described in detail here. The image clarity judgment unit is used to read each camera and calculate the clarity, and then judge whether to execute the clarity warning instruction according to the clarity. The image clarity judgment unit is used to read each camera and calculate the clarity, and then judge whether to execute the clarity warning instruction according to the clarity. The specific steps are as follows: Step A1: read the signal-to-noise ratio of the camera and determine whether the signal-to-noise ratio is lower than the signal-to-noise ratio threshold. If it is lower than the signal-to-noise ratio threshold, issue a clarity warning instruction. If it is higher than the signal-to-noise ratio threshold, execute step A2. Step A2: Calculate the image brightness value. If the brightness value is lower than the brightness threshold, issue a clarity warning instruction. If the brightness value is higher than the brightness threshold, do not issue a clarity warning. When the clarity warning instruction is not executed, send the wearing image data, the cleaning image data, and the operating image data to the image data analysis module. The image data analysis module determines whether to execute the first warning instruction based on the cleaned image data, the image data analysis module determines whether to execute the second warning instruction based on the worn image data, and the image data analysis module determines whether to execute the third warning instruction based on the operation image data. When the first warning instruction, the second warning instruction and the third warning instruction are executed, they jump to the personnel operation behavior recognition warning module, and the execution order of the first warning instruction, the second warning instruction and the third warning instruction is sequentially executed. The expression for the image data analysis module to determine whether to execute the first warning instruction based on the cleaned image data is as follows: ; in, represents the cleaning judgment coefficient, Indicates that the first warning instruction will not be executed. Indicates execution of the first warning instruction. Indicates the minimum value of the cleaning step, Indicates the minimum cleaning time. express The total time taken for the cleaning steps, the expression for the image data analysis module to determine whether to execute the second warning instruction based on the wear image data is as follows: ; in, It indicates the area where the pixels change between the image data taken just after entering the wearing area and the image data taken after entering the wearing area. represents the area change rate, Represents the image area, when the area change rate Minimum area change When the area change rate is Minimum area change , the second warning instruction is not executed, and the specific expression for the image data analysis module to determine whether to execute the third warning instruction based on the operation image data is as follows: ; in, represents the operation judgment coefficient, Indicates The operator closes his eyes for a moment. Indicates When the operator opens his eyes, represents the blink threshold, Determine and execute the third warning instruction. It is determined that the third warning instruction will not be executed. The long blinking time here indicates that the doctor or nurse is at risk of eye fatigue, and thus a timely warning is required; The personnel operation behavior identification and warning module issues warnings based on the clarity warning instructions, the first warning instructions, the second warning instructions and the third warning instructions respectively. The personnel operation behavior identification and warning module issues warnings to the supervisors based on the clarity warning instructions, the first warning instructions, the second warning instructions and the third warning instructions respectively, and marks the corresponding camera that issues the warning and transmits it to the supervisors.

[0018] Embodiment 1: This embodiment records , , ,At this time, there was no warning information at the beginning and during the operation; Embodiment 2: This embodiment records At this time, an early warning that the cleaning condition does not meet the standard appears before the operation begins, and the management personnel and corresponding personnel are reminded that the cleaning condition does not meet the standard; Embodiment 3: This embodiment records , , , at this time, fatigue occurs at the beginning and during the operation. After the current operation is completed, the non-surgeon doctor needs to be rotated to remind the surgeon to rest; See also Figure 2 The present invention also provides a method for identifying human operation behavior based on computer vision. Based on the above-mentioned human operation behavior identification system based on computer vision, the method comprises the following steps: Step 1: acquiring wearing image data based on the camera set in the wearing area, acquiring cleaning image data based on the camera set in the cleaning area, and acquiring operating image data based on the camera set in the operating area; Step 2: Calculate the clarity of each camera, and then determine whether to execute the clarity warning instruction based on the clarity; Step 3: judging whether to execute the first warning instruction based on the cleaned image data; Step 4: Determine whether to execute the second warning instruction based on the wearable image data; Step 5: judging whether to execute the third warning instruction based on the operation image data; Step 6: Issue an early warning.

[0019] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A personnel operation behavior recognition system based on computer vision, characterized by: It includes image data acquisition module, image data analysis module and personnel operation behavior recognition and early warning module; The image data acquisition module includes an image clarity judgment unit, a wearing behavior recognition unit, a cleaning behavior recognition unit and an operation behavior recognition unit. The wearing behavior recognition unit obtains wearing image data by reading a camera set in a wearing area, the cleaning behavior recognition unit obtains cleaning image data by reading a camera set in a cleaning area, and the operation behavior recognition unit obtains operation image data by reading a camera set in an operation area. The image clarity judgment unit is used to read each camera and calculate the clarity, and then judge whether to execute the clarity warning instruction according to the clarity, and send the wearing image data, cleaning image data and operation image data to the image data analysis module when the clarity warning instruction is not executed; The image data analysis module determines whether to execute the first warning instruction based on the cleaning image data, the image data analysis module determines whether to execute the second warning instruction based on the wearing image data, and the image data analysis module determines whether to execute the third warning instruction based on the operation image data. When the first warning instruction, the second warning instruction and the third warning instruction are executed, they jump to the personnel operation behavior recognition warning module, and the execution order of the first warning instruction, the second warning instruction and the third warning instruction is to execute them in sequence; The personnel operation behavior identification and warning module respectively issues warnings according to the clarity warning instruction, the first warning instruction, the second warning instruction and the third warning instruction.

2. The human operation behavior recognition system based on computer vision according to claim 1, characterized in that: The specific expression of the cleaning behavior recognition unit obtaining cleaning image data by reading the camera set in the cleaning area is as follows: ； in, Represents a clear data set, They represent the first image of the person doing the cleaning, ..., the first image of the person doing the cleaning, taken by the camera set in the cleaning area. The image of the person being cleaned, ..., Person images to be cleaned, and the Images of people doing the cleaning , Respectively represent The first cleaning step is performed by the cleaning personnel, ..., The first cleaning step, ... The cleaning steps are identified and labeled by OpenPose. Respectively represent The first cleaning step by the cleaning personnel takes time, ..., The cleaning step takes time, ... The cleaning step is time consuming.

3. The human operation behavior recognition system based on computer vision according to claim 2, characterized in that: The expression for the image data analysis module to determine whether to execute the first warning instruction based on the cleaned image data is as follows: ； in, represents the cleaning judgment coefficient, Indicates that the first warning instruction will not be executed. Indicates execution of the first warning instruction. Indicates the minimum value of the cleaning step, Indicates the minimum cleaning time. express The total time for each cleaning step.

4. The human operation behavior recognition system based on computer vision according to claim 3 is characterized in that: The specific expression of the wearing behavior recognition unit obtaining the wearing image data by reading the camera set in the wearing area is as follows: ； in, represents the wearable image dataset, They respectively represent the first image of the person who has completed wearing the clothing, ..., the first image of the person who has completed wearing the clothing, taken by the camera set in the wearing area. The image of the person who has finished wearing the clothes, ... The image of the person who has finished wearing the clothes, among which Completed wearing personnel images , Indicates the image data taken just after entering the wearable area. Indicates the image data captured when the donning is completed and the wearer enters the donning area.

5. The human operation behavior recognition system based on computer vision according to claim 4 is characterized in that: The expression for the image data analysis module to determine whether to execute the second warning instruction based on the wearing image data is as follows: ； in, It indicates the area where the pixels change between the image data taken just after entering the wearing area and the image data taken after entering the wearing area. represents the area change rate, Represents the image area, when the area change rate Minimum area change When the area change rate is Minimum area change The second warning instruction will not be executed.

6. The human operation behavior recognition system based on computer vision according to claim 5, characterized in that: The specific expression of the operation behavior recognition unit obtaining the operation image data by reading the camera set in the operation area is as follows: ； in, Represents the operation data set, They respectively represent the first image of the person performing the operation, ..., the first image of the person performing the operation, taken by the camera set in the wearing area. The image of the person performing the operation, ... images of people performing operations, among which , Indicates The current closing moment of the operator’s eyes, Indicates The Dlib model is used to identify the last time a person opened his eyes.

7. The human operation behavior recognition system based on computer vision according to claim 6, characterized in that: The specific expression for the image data analysis module to determine whether to execute the third warning instruction based on the operation image data is as follows: ； in, represents the operation judgment coefficient, Indicates The operator closes his eyes for a moment. Indicates When the operator opens his eyes, represents the blink threshold, Determine and execute the third warning instruction. It is determined that the third warning instruction will not be executed.

8. The human operation behavior recognition system based on computer vision according to claim 7, characterized in that: The image clarity judgment unit is used to read each camera and calculate the clarity, and then judge whether to execute the clarity warning instruction according to the clarity. The specific steps are as follows: Step A1: read the signal-to-noise ratio of the camera and determine whether the signal-to-noise ratio is lower than the signal-to-noise ratio threshold. If it is lower than the signal-to-noise ratio threshold, issue a clarity warning instruction. If it is higher than the signal-to-noise ratio threshold, execute step A2. Step A2: Calculate the image brightness value. If the brightness value is lower than the brightness threshold, issue a clarity warning instruction. If the brightness value is higher than the brightness threshold, no clarity warning is issued.

9. The human operation behavior recognition system based on computer vision according to claim 8, characterized in that: The personnel operation behavior identification and warning module warns the supervisor respectively according to the clarity warning instruction, the first warning instruction, the second warning instruction and the third warning instruction, and marks the corresponding camera that issues the warning and transmits it to the supervisor.

10. A method for identifying human operation behavior based on computer vision, based on a system for identifying human operation behavior based on computer vision according to any one of claims 1 to 9, characterized in that: The following steps are involved: Step 1: acquiring wearing image data based on the camera set in the wearing area, acquiring cleaning image data based on the camera set in the cleaning area, and acquiring operating image data based on the camera set in the operating area; Step 2: Calculate the clarity of each camera, and then determine whether to execute the clarity warning instruction based on the clarity; Step 3: judging whether to execute the first warning instruction based on the cleaned image data; Step 4: Determine whether to execute the second warning instruction based on the wearable image data; Step 5: judging whether to execute the third warning instruction based on the operation image data; Step 6: Issue an early warning.

Images