Personnel behavior and sight line detection method in ship environment

Abstract

The invention provides a personnel behavior and sight line detection method in a ship environment, and the method comprises the steps: obtaining a human body posture data set, and randomly dividing the data set into a training set, a verification set and a test set; pre-training a target detection and posture estimation model according to the human body posture data set to obtain a human body characteristic estimation model; acquiring a real-time video stream in a ship environment, and positioning a personnel area and extracting key point information according to a human body characteristic estimation model; a Kalman filter is used for tracking a positioning personnel area and key points, and personnel position and key point information is updated in real time; according to the position of the person and the key point information, determining a sight path by taking the key points of the two ears as sight starting points and the nose tip as a sight path point; a skeleton identification network is used to process personnel positions and key point information, and specific behaviors of personnel are identified. According to the invention, personnel positioning, tracking, sight line determination and behavior identification can be carried out in a complex ship environment, and the safety and efficiency of offshore operation are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of offshore operations, and in particular to a method for detecting human behavior and sight lines in a ship environment. Background Art

[0002] As the complexity and safety requirements of maritime operations and ship operations continue to increase, how to monitor and identify the behavior and sight lines of people in the ship environment in real time and accurately has become an important research topic. Traditional human monitoring methods are difficult to meet the high standards of modern ship operations due to complex environments, poor real-time performance, and low recognition accuracy. Therefore, an efficient and accurate method for detecting human behavior and sight lines is urgently needed to improve the safety and efficiency of maritime operations. Summary of the invention

[0003] According to the technical problems raised above, a method for detecting personnel behavior and sight line in a ship environment is provided. The present invention mainly uses a human body characteristic estimation model to locate the personnel area and extract key point information, uses a Kalman filter to track the located personnel area and key points, updates the personnel position and key point information in real time, and determines the sight line path based on the personnel position and key point information, taking the binaural focus as the sight line starting point and the nose tip as the sight line path point; uses a skeleton recognition network to process the personnel position and key point information and identify the specific behavior of the personnel.

[0004] The technical means adopted by the present invention are as follows: A method for detecting personnel behavior and sight lines in a ship environment comprises: obtaining a human posture data set, and randomly dividing the data set into a training set, a verification set and a test set; pre-training a target detection and posture estimation model according to the human posture data set to obtain a human characteristic estimation model; obtaining a real-time video stream in the ship environment, locating a personnel area and extracting key point information according to the human characteristic estimation model; tracking the located personnel area and key points by using a Kalman filter, and updating the personnel position and key point information in real time; determining the sight line path according to the personnel position and key point information, taking the binaural focus as the sight line starting point and the nose tip as the sight line path point; and processing the personnel position and key point information by using a skeleton recognition network to identify the specific behavior of the personnel.

[0005] Furthermore, the human body characteristic estimation model is expressed as: , , in, Represents the human characteristics estimation model, matrix represents the input image, R represents a real matrix, represents the matrix size, and the image size is , is the image height, is the image width; the number of channels is , P represents the key point information of the human body, and the input image is: , The output is a set of coordinates of key points of the human body in the image; if the model needs to predict key points, the coordinates of each key point are , representing the output as information : , in, ; then personnel area information And human body key point information for: , , in, Indicates time and frame number Human body key point information in dimension, Indicates time and frame number Personnel area information in dimension, Represents a video stream.

[0006] Furthermore, the method of locating the personnel area and extracting key point information according to the human body characteristic estimation model specifically includes: for each video frame, each personnel area information is represented by four parameters: Upper left corner coordinates: ; Bottom right corner coordinates: ; The key point information includes at least: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left palm, right palm, left waist, right waist, left knee, right knee, left ankle and right ankle.

[0007] Furthermore, the use of the Kalman filter to track the location of personnel areas and key points, and to update personnel location and key point information in real time, specifically includes: defining the state vector Contains the bounding box and key point information of the person, as well as the movement speed; the Kalman filter includes the predicted state matrix and the predicted covariance matrix: , , in, represents the predicted state matrix, is the state transition matrix, represents the control input, represents the prediction covariance matrix, is the process noise covariance matrix; , in, Indicates a time interval; The update step of the Kalman filter includes calculating the Kalman gain, updating the state, and updating the covariance matrix: , , , , in, represents the Kalman gain, is the observation matrix, is the observation noise covariance matrix, Indicates the update status. is the current observation value, represents the updated covariance matrix, Represents the identity matrix.

[0008] Further, according to the position and key point information of the personnel, the eye point is taken as the starting point of the eye and the nose tip is taken as the eye path point to determine the eye path, specifically including: determining the coordinates of the midpoint of the ears and the nose tip: assuming that the coordinates of the left ear key point are , the coordinates of the key points of the right ear are , the coordinates of the nose tip key point are ; Calculate the midpoint coordinates of both ears : , The sight vector is obtained by the coordinate difference between the midpoint of the two ears and the tip of the nose : , Take the midpoint of the two ears as the starting point and the tip of the nose as the path point, and follow the line of sight It is expressed as: , in, is a free parameter that controls the position of the straight line points.

[0009] Furthermore, the skeleton recognition network is used to process the position and key point information of personnel and identify the specific behavior of personnel, specifically including: The key point set It is expressed as: , Construct skeleton information sequence, Construct a sequence of key points within the time period: , The skeleton information sequence is processed using the skeleton recognition network, and the skeleton recognition network is represented as a function , the input is a sequence of key points , the output is the behavior category : .

[0010] Compared with the prior art, the present invention has the following advantages: The method for detecting human behavior and sight line in a ship environment provided by the present invention obtains a human characteristic estimation model by constructing a human posture data set and pre-training a target detection and posture estimation model. This step ensures that the model can accurately extract human characteristics from a video stream. A real-time video stream is collected from a ship environment, and the video stream is processed using a pre-trained human characteristic estimation model to realize the positioning of personnel and the extraction of key point information. This includes the detection of a human body boundary box and key point coordinates in each video frame. A Kalman filter is used to track the positioned personnel and their key points. In each video frame, the tracking algorithm can smoothly predict and update the position and posture information of the personnel, reducing the detection error caused by image noise or other interference. Based on the area and key coordinate information of the personnel, a sight line path is determined with the midpoint of both ears as the starting point of the sight line and the tip of the nose as the sight line path point. This step is achieved by calculating the vector between the midpoint of both ears and the coordinates of the tip of the nose, which helps determine the sight line direction of each person. The skeleton recognition network is used to process the personnel area and key coordinate information at a frequency of 30 frames per second to identify the specific behavior of the personnel. The skeleton recognition network can analyze continuous video frames based on human skeleton information and accurately identify people's behavior types, such as walking, standing, sitting, operating equipment, etc.

[0011] The method for detecting personnel behavior and sight lines in a ship environment provided by the present invention has the advantages of efficient and accurate real-time processing capability, which can stably locate, track, determine sight lines and identify behavior of personnel in a complex ship environment, greatly improving the safety and efficiency of marine operations. By comprehensively and accurately monitoring the behavior and sight lines of crew members, the present invention helps prevent accidents, promptly discover and deal with potential dangers, and ensure the safety of ships and personnel.

[0012] Based on the above reasons, the present invention can be widely promoted in fields such as offshore operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0014] Figure 1 This is a flow chart of the method for detecting personnel behavior and sight lines in a ship environment in the present invention.

[0015] Figure 2 The figure shows the detection results of personnel behavior and sight lines in a closed cabin environment in an embodiment of the present invention.

[0016] Figure 3 The following are the detection results of human behavior and sight lines in a dark environment in an embodiment of the present invention.

[0017] Figure 4 The following are the detection results of abnormal personnel behavior and sight lines in the embodiment of the present invention. DETAILED DESCRIPTION

[0018] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0019] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0021] Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps described in these embodiments do not limit the scope of the present invention. At the same time, it should be clear that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to the actual proportional relationship. The technology, method and equipment known to those of ordinary skill in the relevant field may not be discussed in detail, but in appropriate cases, the technology, method and equipment should be regarded as a part of the specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0022] like Figure 1 As shown, the present invention provides a method for detecting human behavior and sight lines in a ship environment, which is characterized by comprising: obtaining a human posture data set, and randomly dividing the data set into a training set, a validation set, and a test set; pre-training a target detection and posture estimation model according to the human posture data set to obtain a human characteristic estimation model. In specific implementation, as a preferred embodiment of the present invention, the human characteristic estimation model is represented as: , , in, Represents the human characteristics estimation model, matrix represents the input image, R represents a real matrix, represents the matrix size, and the image size is , is the image height, is the image width; the number of channels is , P represents the key point information of the human body, and the input image is: , The output is a set of coordinates of key points of the human body in the image; if the model needs to predict key points, the coordinates of each key point are , representing the output as information : , in, ; then personnel area information And human body key point information for: , , in, Indicates time and frame number Human body key point information in dimension, Indicates time and frame number Personnel area information in dimension, In the implementation, a real-time video stream V in the ship environment is obtained; the video stream uses RGB data at 30 frames per second, and the video stream is: , Using the human body characteristics estimation method, it is corrected to: , , Among them, 2 represents the two-dimensional coordinates (x, y) of each key point. represents time in seconds; F is the frame rate, which is taken as a constant of 30 in the following text.

[0023] A real-time video stream in a ship environment is obtained, and the personnel area is located and key point information is extracted according to the human body characteristic estimation model; in specific implementation, as a preferred embodiment of the present invention, the personnel area is located and key point information is extracted according to the human body characteristic estimation model, specifically including: for each video frame, each personnel area information is represented by four parameters: Upper left corner coordinates: ; Bottom right corner coordinates: ; The key point information includes at least: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left palm, right palm, left waist, right waist, left knee, right knee, left ankle and right ankle.

[0024] The Kalman filter is used to track the positioning personnel area and key points, and the personnel position and key point information is updated in real time; for the personnel positioning and key point tracking in the human posture estimation, the Kalman filter can help smooth and predict the position at each moment. In specific implementation, as a preferred embodiment of the present invention, the use of the Kalman filter to track the positioning personnel area and key points, and update the personnel position and key point information in real time, specifically includes: defining the state vector Contains the bounding box and key point information of the person, as well as the movement speed; the Kalman filter includes the predicted state matrix and the predicted covariance matrix: , , in, represents the predicted state matrix, is the state transition matrix, represents the control input, represents the prediction covariance matrix, is the process noise covariance matrix, , in, Indicates a time interval; The update step of the Kalman filter includes calculating the Kalman gain, updating the state, and updating the covariance matrix: , , , , in, represents the Kalman gain, is the observation matrix, is the observation noise covariance matrix, Indicates the update status. is the current observation value, represents the updated covariance matrix, Represents the identity matrix.

[0025] According to the position and key point information of the personnel, the binaural focus is used as the starting point of the line of sight, and the tip of the nose is used as the line of sight path point to determine the line of sight path; in specific implementation, as a preferred embodiment of the present invention, according to the position and key point information of the personnel, the binaural focus is used as the starting point of the line of sight, and the tip of the nose is used as the line of sight path point to determine the line of sight path, specifically including: Determine the coordinates of the midpoints of both ears and the tip of the nose: Assume the coordinates of the left ear key point are , right ear key point coordinates , the coordinates of the nose tip key point are ; Calculate the midpoint coordinates of both ears : , The sight vector is obtained by the coordinate difference between the midpoint of the two ears and the tip of the nose : , Take the midpoint of the two ears as the starting point and the tip of the nose as the path point, and follow the line of sight It is expressed as: , in, is a free parameter that controls the position of the straight line points.

[0026] The skeleton recognition network is used to process the position and key point information of personnel at a frequency of 30 frames per second to identify the specific behavior of personnel. In specific implementation, as a preferred embodiment of the present invention, the skeleton recognition network is used to process the position and key point information of personnel to identify the specific behavior of personnel, specifically including: The key point set It is expressed as: , Construct skeleton information sequence, Construct a sequence of key points within the time period: , The skeleton information sequence is processed using the skeleton recognition network, and the skeleton recognition network is represented as a function , the input is a sequence of key points , the output is the behavior category : .

[0027] Example like Figure 2 As shown in the figure, this embodiment provides the detection results of personnel behavior and sight lines in a closed cabin environment. The green box is the personnel position information, the key point connection line represents the human skeleton, the text in the upper left corner of the detection box is the specific behavior, and the red line is the sight line direction. It can be seen from the experimental effect diagram that the tasks and specific behaviors in the closed environment of the ship are accurately identified, which are playing with mobile phones and not looking out. The method of the present invention is more accurate in detecting and identifying people.

[0028] like Figure 3 As shown, this embodiment provides the detection results of human behavior and sight line in a dark light environment. The green box is the person's location information, the key point connection line represents the human skeleton, the text in the upper left corner of the detection box is the specific behavior, and the red line is the sight line direction. It can be seen from the experimental effect diagram that the method of the present invention is more accurate in detecting people in a dark light environment, can better identify the specific posture and behavior of people, and is more accurate in detecting sight lines.

[0029] like Figure 4 As shown, this embodiment provides the detection results of abnormal person behavior and line of sight. The green box is the person's location information, the key point connection line represents the human skeleton, the text in the upper left corner of the detection box is the specific behavior, and the red line is the line of sight. It can be seen from the experimental effect diagram that the method of the present invention is more accurate in detecting abnormal people, and correctly detects the fallen person and his corresponding line of sight after falling to the ground.

[0030] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting human behavior and sight lines in a ship environment, characterized in that: include: Obtain a human posture dataset and randomly divide the dataset into a training set, a validation set, and a test set; Pre-training the target detection and posture estimation model according to the human posture dataset to obtain a human characteristic estimation model; Obtain real-time video streams in the ship environment, locate personnel areas based on the human body characteristic estimation model, and extract key point information; Use Kalman filter to track the location of personnel areas and key points, and update personnel location and key point information in real time; According to the position and key point information of the personnel, the sight path is determined by taking the focus of both ears as the starting point of the sight and the tip of the nose as the sight path point; The skeleton recognition network is used to process the location and key point information of personnel and identify their specific behaviors.

2. The method for detecting human behavior and sight lines in a ship environment according to claim 1, characterized in that: The human body characteristic estimation model is expressed as: ， ， in, Represents the human characteristics estimation model, matrix represents the input image, R represents a real matrix, represents the matrix size, and the image size is ,in is the image height, is the image width and the number of channels is , P represents the key point information of the human body, and the input image is: ， The output is a set of coordinates of key points of the human body in the image; if the model needs to predict key points, the coordinates of each key point are , representing the output as information : ， in, ; then personnel area information And human body key point information for: ， ， in, Indicates time and frame number Human body key point information in dimension, Indicates time and frame number Personnel area information in dimension, Represents a video stream.

3. The method for detecting human behavior and sight lines in a ship environment according to claim 1, characterized in that: The method of locating the personnel area and extracting key point information based on the human body characteristic estimation model specifically includes: For each video frame, each person region information is represented by four parameters: Upper left corner coordinates: ; Bottom right corner coordinates: ; The key point information includes at least: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left palm, right palm, left waist, right waist, left knee, right knee, left ankle and right ankle.

4. The method for detecting human behavior and sight lines in a ship environment according to claim 1, characterized in that: The use of the Kalman filter to track the location of personnel areas and key points, and to update personnel location and key point information in real time, specifically includes: Define the state vector Contains the bounding box and key point information of the person, as well as the movement speed; the Kalman filter includes the predicted state matrix and the predicted covariance matrix: ， ， in, represents the predicted state matrix, is the state transition matrix, represents the control input, represents the prediction covariance matrix, is the process noise covariance matrix; ， in, Indicates a time interval; The update step of the Kalman filter includes calculating the Kalman gain, updating the state, and updating the covariance matrix: ， ， ， ， in, represents the Kalman gain, is the observation matrix, is the observation noise covariance matrix, Indicates the update status. is the current observation value, represents the updated covariance matrix, Represents the identity matrix.

5. The method for detecting human behavior and sight lines in a ship environment according to claim 1, characterized in that: The method of determining the sight path according to the position and key point information of the person, taking the two ear points as the sight starting points and the nose tip as the sight path point, specifically includes: Determine the coordinates of the midpoints of both ears and the tip of the nose: Assume the coordinates of the left ear key point are , the coordinates of the key points of the right ear are , the coordinates of the nose tip key point are ; Calculate the midpoint coordinates of both ears : ， The sight vector is obtained by the coordinate difference between the midpoint of the two ears and the tip of the nose : ， Take the midpoint of the two ears as the starting point and the tip of the nose as the path point, and follow the line of sight It is expressed as: ， in, is a free parameter that controls the position of the straight line points.

6. The method for detecting human behavior and sight lines in a ship environment according to claim 1, characterized in that: The skeleton recognition network is used to process the position and key point information of personnel and identify the specific behavior of personnel, specifically including: Time The key point set It is expressed as: ， Construct skeleton information sequence, Construct a sequence of key points within the time period: ， The skeleton information sequence is processed using the skeleton recognition network, and the skeleton recognition network is represented as a function , the input is a sequence of key points , the output is the behavior category : 。

Images