A method and system for detecting non-standard behavior in the cockpit of a ship.

Abstract

The present invention proposes a method and system for detecting non-standard behaviors in a ship's cockpit. The method includes acquiring a picture of the ship's cockpit through an image acquisition device and segmenting different driving areas and drivers; identifying the driver, and using a skeleton recognition network to detect key points of the driver, generating a key point action timing diagram; designing a sequential automaton recognition model for the driver's behavior, performing time-series matching on the key point action timing diagram, and outputting the matched behavior in real time. Based on the detection method, a non-standard behavior detection system in a ship's cockpit is also proposed. The present invention is based on the problem of identifying driving behaviors in a ship's cockpit based on a logical sequence, which is conducive to real-time early warning and prevention of non-standard behaviors in a ship's cockpit, overcomes the dilemma of traditional action recognition systems relying on key scenes and key objects, and can track and recognize actions of multiple drivers in the same scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of ship driving behavior detection, and in particular relates to a method and system for detecting non-standard behavior in a ship's cockpit. Background Art

[0002] The detection of irregular behavior in the cockpit of a ship has great uses in both military and civilian fields. A good algorithm for detecting irregular behavior in the cockpit can detect and warn the behavior of the driver in the cockpit in a timely manner. Through the real-time output and warning of the algorithm, the driving safety of the ship can be greatly enhanced. At the same time, it can also ensure the personal safety of the driver and all crew members in the cabin to a certain extent.

[0003] Common abnormal behavior recognition algorithms in the cabin in the prior art can be divided into two types: strong category behavior recognition algorithms and strong logic behavior recognition algorithms. The recognition of strong category behavior is often accompanied by the appearance of key people, objects, clothing, etc. in the cockpit scene, and can be recognized through traditional image semantic understanding, so it is relatively easy to solve. Existing strong logic behavior recognition algorithms do not have key identification objects in the recognition process, and most of them rely on empirical knowledge of actions, which increases the difficulty of identifying standardized and non-standard behaviors in the cockpit. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a method and system for detecting non-standard behavior in the ship's cockpit, which solves the problem of identifying driving behavior in the ship's cockpit based on logical sequences, and is conducive to real-time warning and prevention of non-standard behavior in the ship's cockpit.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for detecting non-standard behavior in a ship's cockpit comprises the following steps:

[0007] The image of the ship's cab is acquired through image acquisition equipment and different driving areas and drivers are segmented;

[0008] Identify the driver and use the skeleton recognition network to detect the key points of the driver and generate a key point action sequence diagram;

[0009] Design a sequential automaton recognition model for driver behavior, perform sequential matching on the key point action timing diagrams, and output the matched behavior in real time.

[0010] Furthermore, the process of acquiring the ship's cab image through the image acquisition device and segmenting different driving areas and drivers includes:

[0011] The image of the ship's wheelhouse and the image of the driver are obtained by using an image acquisition device installed in the ship's wheelhouse;

[0012] The cab image is divided into different driving areas by predefined area coordinates, and the driver's outline is segmented by a pedestrian segmentation algorithm.

[0013] Furthermore, the process of identifying the driver and detecting the key points of the driver using the skeleton recognition network is as follows:

[0014] Use pedestrian re-identification algorithm to match drivers and determine the required standard behavior recognition model set and non-standard behavior recognition model set for driving;

[0015] The human skeleton recognition algorithm is used to identify the skeleton models of different drivers, and key points are extracted from the skeleton models to construct multiple human key point sets.

[0016] Furthermore, the process of generating the key point action timing diagram is as follows:

[0017] The key point sets of different drivers are grouped with the prior key point sets by using the personnel matching information, and the prior key point sequence information is matched with the current multiple human key point sets; the key point information of each ship driver is fused with the prior key point sequence information of the current person to generate the key point information sequence diagram of the driver at the current moment.

[0018] Furthermore, the priori key point sequence information is a human body key point information database.

[0019] The method for detecting non-standard behavior in a ship cockpit according to claim 1 is characterized in that the process of designing a temporal automaton recognition model of the driver's behavior is:

[0020] Decompose non-standard actions into a set of sub-action sequences where p i represents the number of the key point; j represents the sub-action number; the sub-action is a combination of any position and any atomic behavior; the atomic behavior set A = {a 1 ,a 2 ...a n};

[0021] The state transition function of each action sequential automaton is obtained by constructing a set of sub-action sequences for each action. Where S x and S y There are two states, belonging to two elements in the state set S.

[0022] The present invention also proposes a non-standard behavior detection system in a ship cockpit, including a segmentation module, a recognition detection module and a design matching module;

[0023] The segmentation module is used to obtain the ship's cab image through an image acquisition device and segment different driving areas and drivers;

[0024] The detection module is used to identify the driver and use a skeleton recognition network to detect key points of the driver and generate a key point action sequence diagram;

[0025] The matching module is used to design a sequential automaton recognition model of the driver's behavior, perform sequential matching on the key point action sequential diagram, and output the matched behavior in real time.

[0026] Furthermore, the process performed by the segmentation module is: obtaining the ship cab image and the driver image through an image acquisition device installed in the ship cab; dividing the cab image into different driving areas through predefined area coordinates, and segmenting the driver's outline through a pedestrian segmentation algorithm.

[0027] Further, the identification and detection module includes an identification module and a detection module;

[0028] The recognition module is used to match drivers using a pedestrian re-identification algorithm, determine the required standard behavior recognition model set and non-standard behavior recognition model set for driving; use a human skeleton recognition algorithm to identify skeleton models of different drivers, extract key points from the skeleton models to construct multiple human key point sets.

[0029] The detection module is used to use personnel matching information to group the key point sets of different drivers with the prior key point sets, match the prior key point sequence information with the current multiple human body key point sets; merge the key point information of each ship driver with the prior key point sequence information of the current person, and generate a key point information sequence diagram of the driver at the current moment.

[0030] Further, the design matching module includes a design module and a matching module;

[0031] The design module is used to split the non-standard action into sub-action sequence sets and where p i represents the number of the key point; j represents the sub-action number; the sub-action is a combination of any position and any atomic behavior; the atomic behavior set A = {a 1 ,a 2 ...a n}; Obtain the state transition function of each action sequential automaton by constructing a set of sub-action sequences for each action Where S x and S y There are two states, belonging to two elements in the state set S.

[0032] The matching module is used to perform time sequence matching on the key point action timing diagram and output the matched behavior in real time.

[0033] The effects provided in the content of the invention are only the effects of the embodiments, not all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects:

[0034] The present invention proposes a method and system for detecting non-standard behaviors in a ship's cockpit. The method includes acquiring a picture of the ship's cockpit through an image acquisition device and segmenting different driving areas and drivers; identifying the driver, and using a skeleton recognition network to detect key points of the driver, generating a key point action timing diagram; designing a sequential automaton recognition model for the driver's behavior, performing time-series matching on the key point action timing diagram, and outputting the matched behavior in real time. Based on a method for detecting non-standard behaviors in a ship's cockpit, a system for detecting non-standard behaviors in a ship's cockpit is also proposed. The present invention is based on the problem of identifying driving behaviors in a ship's cockpit based on a logical sequence, which is conducive to real-time early warning and prevention of non-standard behaviors in a ship's cockpit, and overcomes the dilemma of traditional action recognition systems relying on key scenes and key objects, and can track and identify actions of multiple drivers in the same scene.

[0035] The present invention extracts the associated information from the input current key point information and historical key point sequence diagram of the driver in the cabin, and establishes an adjustable logical association function for each action. Therefore, the result can be self-optimized as the input data increases, thereby improving the accuracy of non-standard action recognition.

[0036] The present invention also supports real-time tracking and identification of multiple persons in the ship's cockpit at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] like Figure 1 This is a flow chart of a method for detecting non-standard behavior in a ship's cockpit according to Embodiment 1 of the present invention;

[0038] like Figure 2 A schematic diagram of a non-standard action recognition model based on a temporal automaton in Embodiment 1 of the present invention;

[0039] like Figure 3 This is a schematic diagram of a non-standard behavior detection system in a ship cockpit according to Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0040] In order to clearly illustrate the technical features of the present solution, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings. The disclosure below provides many different embodiments or examples for realizing different structures of the present invention. In order to simplify the disclosure of the present invention, the components and settings of specific examples are described below. In addition, the present invention may repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed. It should be noted that the components illustrated in the accompanying drawings are not necessarily drawn to scale. The present invention omits the description of known components and processing techniques and processes to avoid unnecessary limitations on the present invention.

[0041] Example 1

[0042] Embodiment 1 of the present invention proposes a method for detecting non-standard behavior in a ship's cockpit. The method is based on a logical sequence of the problem of identifying driving behavior in a ship's cockpit, and is conducive to real-time warning and prevention of non-standard behavior in a ship's cockpit.

[0043] like Figure 1 A flow chart of a method for detecting non-standard behavior in a ship's cockpit is provided in accordance with Embodiment 1 of the present invention;

[0044] Firstly, the image of the ship's cab is obtained through image acquisition equipment and different driving areas and drivers are segmented.

[0045] The image of the ship's wheelhouse and the image of the driver are obtained by using an image acquisition device installed in the ship's wheelhouse;

[0046] The cab image is divided into different driving areas by predefined area coordinates, and the outline of the driver is segmented by a pedestrian segmentation algorithm. The specific process is: using a pedestrian re-identification algorithm to match the driver, determine the required standard behavior recognition model set and non-standard behavior recognition model set for driving; using a human skeleton recognition algorithm to identify the skeleton models of different drivers, extract key points from the skeleton models to construct multiple human key point sets.

[0047] Secondly, the driver’s identity is recognized, and the skeleton recognition network is used to detect the driver’s key points and generate a key point action timing diagram.

[0048] Use pedestrian re-identification algorithm to match drivers and determine the required standard behavior recognition model set and non-standard behavior recognition model set for driving;

[0049] The human skeleton recognition algorithm is used to identify the skeleton models of different drivers, and key points are extracted from the skeleton models to construct multiple human key point sets.

[0050] The process of generating the key point action timing diagram is:

[0051] The key point sets of different drivers are grouped with the prior key point sets using the personnel matching information, and the prior key point sequence information is matched with the current multiple human key point sets; the key point information of each ship driver is fused with the prior key point sequence information of the current person to generate the current driver key point information sequence diagram. The prior key point sequence information is the human key point information database.

[0052] Secondly, a sequential automaton recognition model of driver behavior is designed to match the key point action sequence diagrams in time sequence and output the matched behaviors in real time. Figure 2 A schematic diagram of a non-standard action recognition model based on a temporal automaton in Example 1 of the present invention is given.

[0053] Construct a temporal automatic recognition model for different behaviors (including normative behaviors and non-normative behaviors)

[0054] For any definable non-standard action, decompose the non-standard action into a set of sub-action sequences where p i represents the number of the key point; j represents the sub-action number; it can be any human landmark position such as the left eye, right palm, left shoulder, etc. A sub-action can be a combination of any position and any atomic action. Atomic action set A = {a 1 ,a 2 ...a n}; Each element in the set can be any atomic action such as "interval

[0055]

[0056] By constructing a set of sub-action sequences for each action, the state transition function of the action timing automaton can be obtained. By constructing a set of sub-action sequences for each action, the state transition function of the action timing automaton can be obtained. Where S x and S y There are two states, belonging to two elements in the state set S. There are two special elements S in the state set S cor and S err They respectively represent whether the current state of the ship's driver is a standard state or a non-standard state.

[0057] For the scene in the ship's cockpit, we take the "regularly check the dashboard" action sequence as an example to illustrate the algorithm:

[0058] Defining the inspection dashboard is mainly divided into three action logic sequences:

[0059] After the cabin driver lowers his head to observe the equipment, he must immediately raise his head to look far away.

[0060] If the cabin driver lowers his head for a long time without raising it, it will be considered a violation. This time can be configured

[0061] If the cabin driver lowers his head to observe the equipment, then looks around without keeping his eyes forward, it will be considered a violation.

[0062] In the algorithm implementation, we split the action sequence into three sub-actions: looking at the device, looking forward, and looking in other directions. A normal action logic sequence is that after the "looking at the device" appears, only a series of "looking forward" actions can appear immediately. If it is a "looking in other directions" action, it is considered a violation.

[0063] Finally, the key point information sequence diagrams of different drivers at that moment are input into different sequential automaton recognition models respectively, and the current behavior states of different ship drivers are output.

[0064] A method for detecting non-standard behavior in a ship's cockpit proposed in Example 1 of the present invention solves the problem of identifying driving behavior in a ship's cockpit based on a logical sequence, which is conducive to real-time warning and prevention of non-standard behavior in a ship's cockpit, overcomes the dilemma of traditional action recognition systems relying on key scenes and key objects, and can track and identify actions of multiple drivers in the same scene separately.

[0065] The method for detecting non-standard behaviors in a ship's cockpit proposed in Example 1 of the present invention extracts related information from the input current key point information and historical key point sequence diagram of the driver in the cabin, and establishes an adjustable logical association function for each action. Therefore, the result can be self-optimized as the input data increases, thereby improving the accuracy of non-standard action recognition.

[0066] The method for detecting non-standard behaviors in a ship's cockpit proposed in Embodiment 1 of the present invention supports real-time tracking and identification of multiple persons in the ship's cockpit at the same time.

[0067] Example 2

[0068] Based on the method for detecting non-standard behavior in a ship cockpit proposed in Example 1 of the present invention, Example 2 of the present invention further proposes a system for detecting non-standard behavior in a ship cockpit. Figure 3 This is a schematic diagram of a non-standard behavior detection system in a ship cockpit according to embodiment 2 of the present invention, the system comprising a segmentation module, a recognition and detection module and a design matching module;

[0069] The segmentation module is used to obtain the ship's cab image through the image acquisition device and segment different driving areas and drivers;

[0070] The detection module is used to identify the driver and use the skeleton recognition network to detect the key points of the driver and generate a key point action sequence diagram;

[0071] The matching module is used to design a sequential automaton recognition model for driver behavior, perform sequential matching on the key point action timing diagram, and output the matched behavior in real time.

[0072] The process performed by the segmentation module is: obtaining the ship's cab image and the driver's image through an image acquisition device installed in the ship's cab; dividing the cab image into different driving areas through predefined area coordinates, and segmenting the driver's outline through a pedestrian segmentation algorithm.

[0073] The identification and detection module includes an identification module and a detection module;

[0074] The recognition module is used to match drivers using a pedestrian re-identification algorithm, determine the required set of standard driving behavior recognition models and non-standard driving behavior recognition models; use the human skeleton recognition algorithm to identify the skeleton models of different drivers, extract key points from the skeleton models to construct multiple human key point sets.

[0075] The detection module is used to use personnel matching information to group the key point sets of different drivers with the prior key point sets, and to match the prior key point sequence information with the current multiple human key point sets; the key point information of each ship driver is merged with the prior key point sequence information of the current person to generate a key point information sequence diagram of the driver at the current moment.

[0076] The design matching module includes a design module and a matching module;

[0077] The design module is used to split non-standard actions into sub-action sequence sets and where p i represents the number of the key point; j represents the sub-action number; the sub-action is a combination of any position and any atomic behavior; the atomic behavior set A = {a 1 ,a 2 ...a n}; Obtain the state transition function of each action sequential automaton by constructing a set of sub-action sequences for each action Where S x and S y There are two states, belonging to two elements in the state set S.

[0078] The matching module is used to perform timing matching on the key point action timing diagram and output the matched behavior in real time.

[0079] The non-standard behavior detection system in the ship's cockpit proposed in Example 2 of the present invention solves the problem of identifying driving behaviors in the ship's cockpit based on logical sequences, which is conducive to real-time warning and prevention of non-standard behaviors in the ship's cockpit, overcomes the dilemma of traditional action recognition systems relying on key scenes and key objects, and can track and identify actions of multiple drivers in the same scene separately.

[0080] The non-standard behavior detection system in the ship cockpit proposed in Example 2 of the present invention extracts related information from the current key point information and historical key point sequence diagram of the driver in the cabin, and establishes an adjustable logical association function for each action. Therefore, the result can be self-optimized as the input data increases, thereby improving the accuracy of non-standard action recognition.

[0081] The non-standard behavior detection system in the ship cockpit proposed in Example 2 of the present invention supports real-time tracking and identification of multiple people in the ship cockpit at the same time.

[0082] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment that includes a series of elements are inherent to the elements. In the absence of more restrictions, the elements limited by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or equipment that includes the elements. In addition, the above-mentioned technical solution provided in the embodiment of the present application is consistent with the corresponding technical solution in the prior art in principle, and the part is not described in detail, so as not to repeat too much.

[0083] Although the above describes the specific implementation of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. For those skilled in the art, other different forms of modifications or deformations can be made on the basis of the above description. It is not necessary and impossible to list all the implementation methods here. On the basis of the technical solution of the present invention, various modifications or deformations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A method for detecting non-standard behavior in a ship's cockpit, characterized in that: The following steps are involved: The image of the ship's cab is acquired through image acquisition equipment and different driving areas and drivers are segmented; The driver is identified, and then the skeleton recognition network is used to detect the key points of the identified driver to generate a key point action sequence diagram; the process of generating the key point action sequence diagram is: using the personnel matching information to group the key point sets of different drivers with the prior key point sets, and matching the prior key point sequence information with the current multiple human body key point sets; The key point information of each ship driver is integrated with the prior key point sequence information of the current driver to generate a key point information sequence diagram of the driver at the current moment; The priori key point sequence information is a human body key point information database; Design a sequential automaton recognition model for driver behavior, match the key point action sequence diagrams sequentially, and output the matched behavior in real time; The process of designing the temporal automaton recognition model of driver behavior is as follows: decomposing the non-standard action into a set of sub-action sequences where p i represents the number of the key point; j represents the sub-action number; the sub-action is a combination of any position and any atomic behavior; the atomic behavior set A = {a1, a2...a n }; Obtain the state transition function of each action sequential automaton by constructing a set of sub-action sequences for each action Where S x and S y There are two states, belonging to two elements in the state set S.

2. A method for detecting non-standard behavior in a ship cockpit according to claim 1, characterized in that: The process of acquiring the ship's cab image through an image acquisition device and segmenting different driving areas and drivers includes: The image of the ship's wheelhouse and the image of the driver are obtained by using an image acquisition device installed in the ship's wheelhouse; The cab image is divided into different driving areas by predefined area coordinates, and the driver's outline is segmented by a pedestrian segmentation algorithm.

3. A method for detecting non-standard behavior in a ship cockpit according to claim 2, characterized in that: The process of identifying the driver and detecting the key points of the driver using the skeleton recognition network is as follows: Use pedestrian re-identification algorithm to match drivers and determine the driver's standard behavior recognition model set and non-standard behavior recognition model set; The human skeleton recognition algorithm is used to identify the skeleton models of different drivers, and key points are extracted from the skeleton models to construct multiple human key point sets.

4. A non-standard behavior detection system in a ship's cockpit, characterized in that: It includes segmentation module, recognition and detection module and design matching module; The segmentation module is used to obtain the ship's cab image through an image acquisition device and segment different driving areas and drivers; The identification detection module is used to identify the driver, and then use the skeleton recognition network to detect the key points of the identified driver to generate a key point action sequence diagram; the process of generating the key point action sequence diagram is: using the personnel matching information to group the key point sets of different drivers with the prior key point sets, and matching the prior key point sequence information with the current multiple human body key point sets; The key point information of each ship driver is integrated with the prior key point sequence information of the current driver to generate a key point information sequence diagram of the driver at the current moment; The priori key point sequence information is a human body key point information database; The design matching module is used to design a sequential automaton recognition model for driver behavior, perform sequential matching on the key point action sequential diagram, and output the matched behavior in real time; the design matching module includes a design module and a matching module; the design module is used to split the non-standard action into sub-action sequence sets and where p i represents the number of the key point; j represents the sub-action number; the sub-action is a combination of any position and any atomic behavior; the atomic behavior set A = {a1, a2...a n }; Obtain the state transition function of each action sequential automaton by constructing a set of sub-action sequences for each action Where S x and S y There are two states, which are two elements in the state set S; the matching module is used to perform time sequence matching on the key point action timing diagram and output the matched behavior in real time.

5. A non-standard behavior detection system in a ship cockpit according to claim 4, characterized in that: The process performed by the segmentation module is: obtaining the ship cab image and the driver image through the image acquisition device installed in the ship cab; dividing the cab image into different driving areas through predefined area coordinates, and segmenting the driver's outline through a pedestrian segmentation algorithm.

6. A non-standard behavior detection system in a ship cockpit according to claim 4, characterized in that: The identification and detection module includes an identification module and a detection module; The recognition module is used to match the driver with a pedestrian re-identification algorithm, determine the driver's standard behavior recognition model set and non-standard behavior recognition model set; use the human skeleton recognition algorithm to identify the skeleton models of different drivers, extract key points from the skeleton models to construct multiple human key point sets; The detection module is used to group the key point sets of different drivers with the prior key point sets by using the personnel matching information, and match the prior key point sequence information with the current multiple human body key point sets; The key point information of each ship driver is fused with the prior key point sequence information of the current driver to generate a key point information sequence diagram of the driver at the current moment.

Images