Service-oriented robot behavior recognition method based on PAF vector field algorithm and depth map

Abstract

The invention discloses a service-oriented robot behavior recognition method based on a PAF vector field algorithm and a depth map, and relates to the technical field of artificial intelligence and robots, and the method comprises the steps: employing a depth camera to obtain a depth image; human body information in the foreground of the depth image is separated from the background by adopting a foreground and background separation algorithm, and a background image is output; 3, processing the depth image by using a PAF vector field algorithm, extracting skeleton key points of a human body, generating skeleton information, and obtaining a human body posture according to the skeleton information; 4, calling the background image by adopting a privacy protection module, and placing the skeleton information in the background image; and 5, detecting whether a falling event or an abnormal waving action exists or not by analyzing the extracted human body posture, and if so, triggering an alarm mechanism. The human body posture in the service area is detected in real time through the PAF vector field and the depth camera, high real-time performance and high accuracy are achieved, and falling detection requirements in various complex environments can be effectively met.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and robotics, and in particular to a service robot behavior recognition method based on a PAF vector field algorithm and a depth map. Background Art

[0002] With the rapid development of artificial intelligence, computer vision and robotics, inspection robots are increasingly used in various places. Inspection robots can replace manual work to complete tasks such as equipment inspection, environmental monitoring, and security patrols, which improves work efficiency, reduces labor costs, and ensures human safety in dangerous environments. However, existing technologies still have certain limitations in environmental perception and safety monitoring in complex environments, especially in the accuracy of detecting fall events, which needs to be improved. In addition, the system usually requires real-time processing and response to detection, which places high demands on the calculation speed of the algorithm and the processing power of the system. Existing posture recognition technology often finds it difficult to balance high real-time and high accuracy, and does not take into account the privacy protection of people in the inspection area, resulting in unsatisfactory performance when dealing with complex scenarios.

[0003] Therefore, technicians in this field are committed to developing a service robot behavior recognition method based on PAF vector field algorithm and depth map, which has high real-time and high accuracy and can effectively meet the fall detection needs in various complex environments. Summary of the invention

[0004] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is how to improve the accuracy and real-time performance of behavior detection.

[0005] To achieve the above object, the present invention provides a service robot behavior recognition method based on PAF vector field algorithm and depth map, the method comprising the following steps:

[0006] Step 1: Use a depth camera to obtain a depth image;

[0007] Step 2: Use the foreground and background separation algorithm to process the depth image, separate the human body information in the foreground of the depth image from the background, and output the background image;

[0008] Step 3: Use the PAF vector field algorithm to process the depth image, extract the key points of the human skeleton, generate skeleton information, and obtain the human posture based on the skeleton information;

[0009] Step 4, using a privacy protection module to call the background image obtained in step 2, and placing the skeleton information obtained in step 3 in the background image;

[0010] Step 5: Detect whether there is a fall event or abnormal waving action by analyzing the extracted human posture;

[0011] Step 6: If a fall event or abnormal waving action is detected, the alarm mechanism is triggered.

[0012] Furthermore, the step 1 further comprises:

[0013] Step 1.1, use a depth camera to obtain a depth image of the inspection area, and compress the depth image value by 1 / 16; before starting the inspection task, use the depth image taken by the depth camera for the first time as the background depth image;

[0014] Step 1.2: preprocess the depth image, including extracting the maximum depth value information in the image to limit the detection range and generate a three-channel image.

[0015] Furthermore, the step 2 further comprises:

[0016] Step 2.1, calculate the infrared image difference between the current real-time acquired depth image and the background depth image, if the difference is less than the threshold, set the pixel value of the depth image to 0;

[0017] Step 2.2: Use morphological operations to remove noise from the depth image and retain the human body information in the foreground;

[0018] Step 2.3: Use the privacy protection module to filter out the human body, supplement it with the background, and output the background image.

[0019] Furthermore, the morphological operation includes: performing an opening operation to remove small white noise points, and performing a closing operation to fill black points.

[0020] Furthermore, the step 3 also includes:

[0021] Step 3.1, extract features from the depth image through a multi-layer convolutional neural network to obtain feature map information;

[0022] Step 3.2, generate PAF vector field and joint point prediction heat map according to feature map information;

[0023] Step 3.3: Based on the joint point prediction results, each joint point is classified into a recognized person from bottom to top through the PAF vector field to generate skeleton information.

[0024] Furthermore, the step 5 also includes determining posture stability; by analyzing the position information and boundingbox ratio between the nodes of the skeleton, the posture stability of the human body is determined.

[0025] Furthermore, detecting a fall event in step 5 includes: when the posture stability is lower than a preset threshold, determining it as a fall event.

[0026] Furthermore, detecting the abnormal waving action in step 5 includes: determining whether there is an abnormal waving action by detecting a specific hand gesture, and determining that the specific hand gesture is an abnormal waving action when it is detected.

[0027] Furthermore, the step 6 also includes:

[0028] Step 6.1, generating alarm information; the alarm information includes the event type, the location where the event occurred, and the time when the event occurred;

[0029] Step 6.2, sending the alarm information to a remote monitoring center via a wireless communication module;

[0030] Step 6.3: After receiving the alarm information, the remote monitoring center processes and responds.

[0031] Furthermore, the privacy protection module can process the depth image and filter designated sensitive information, wherein the designated sensitive information includes a private place.

[0032] Compared with the prior art, the present invention has at least the following beneficial technical effects:

[0033] 1. The present invention combines the PAF vector field algorithm and the depth camera to achieve real-time detection of human posture, especially accurate recognition of fall events and abnormal waving movements, significantly improving the monitoring capability of the inspection robot in complex environments;

[0034] 2. In the process of foreground and background separation and gesture recognition, the present invention applies a privacy protection module to effectively filter and blur sensitive information, thereby ensuring the privacy of personnel in the inspection area and solving the problem of insufficient privacy protection in the prior art.

[0035] 3. The present invention includes a flexible alarm mechanism. When a fall event or abnormal waving action is detected, the system will immediately send the alarm information to the remote monitoring center through the wireless communication module to ensure that emergency measures can be taken in time;

[0036] 4. The present invention introduces advanced data processing algorithms, such as multi-layer convolutional neural networks and partial association fields (PAF), to extract human skeleton key points and posture information from depth images to form a high-level understanding of human activities in the inspection area.

[0037] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a method flow chart of a preferred embodiment of the present invention;

[0039] Figure 2 It is a flow chart of a foreground and background separation algorithm of a preferred embodiment of the present invention;

[0040] Figure 3 is a schematic diagram of depth map skeleton detection according to a preferred embodiment of the present invention;

[0041] Figure 4 It is a privacy protection flow chart of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0043] In the drawings, components with the same structure are indicated by the same numerical reference numerals, and components with similar structures or functions are indicated by similar numerical reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of the components is appropriately exaggerated in some places in the drawings.

[0044] This embodiment provides a service robot behavior recognition method based on PAF vector field algorithm and depth map. Figure 1 As shown, the following steps are included:

[0045] Step 1: Use a depth camera to obtain a depth image, including:

[0046] Step 1.1. Use the DepEye depth camera to obtain the depth image of the inspection area, and compress the depth image value by 1 / 16 for actual display. The DepEye depth camera has high-precision depth perception capabilities and can provide detailed depth information. Before starting the inspection task, the depth image taken by the depth camera is used as the background depth map for the first time. The background depth map is used as the benchmark for subsequent foreground and background separation to help separate the human body information in the foreground. At the same time, the background depth map is also used as the material for the privacy protection module;

[0047] Step 1.2: preprocess the acquired depth image, including extracting the maximum depth value information in the image to limit the detection range and generate a three-channel image (similar to the three channels of a color image, but here it is the three channels of a depth image) for subsequent processing and display.

[0048] Step 2: Use the foreground and background separation algorithm to process the depth image, separate the human body information in the foreground from the background, and use the privacy protection module to perform privacy protection on the human body information, such as Figure 2 As shown, specifically including:

[0049] Step 2.1, compare the depth image currently acquired in real time with the background depth image, calculate their infrared image difference, if the difference is less than a certain threshold (in this embodiment, the threshold is 20), set the corresponding depth image pixel value to 0;

[0050] Step 2.2, use morphological operations to remove noise in the depth image and retain the main foreground objects (such as human bodies); the morphological operations include opening operations to remove small white noise points and closing operations to fill black points;

[0051] Step 2.3: Privacy Protection

[0052] During the foreground and background separation and posture recognition process, the privacy protection module is used to process the image to ensure the privacy of the detected object. The privacy protection module uses the background image of the foreground and background separation algorithm as the display screen, and filters out the part that needs to be protected (such as the human body) according to the algorithm, and uses the background to supplement it. In the subsequent display stage, only the skeleton information of step 3 is used for display.

[0053] Step 3: Use the PAF vector field algorithm to process the depth image and extract the key points of the human skeleton, including:

[0054] Step 3.1, extract features from the depth image through a multi-layer convolutional neural network to obtain a feature map;

[0055] Step 3.2: Generate PAF (Part Affinity Fields) vector field and joint point prediction heat map based on feature map information;

[0056] In this example, there is a human stick figure skeleton link (e.g., shoulder to elbow), defining a PAF vector field P c , where c represents the type of connection (e.g., shoulder to elbow is one type, elbow to wrist is another type). The PAF vector field has a vector P at each pixel location (x, y) c (x, y) represents the normalized distance from point A to point B. If the pixel position (x, y) is not in the area connected to c, the vector field value is 0. The specific formula is as follows:

[0057]

[0058] Among them, v c is the normalized vector from point A to point B:

[0059]

[0060] The normalized vector modulus is always 1, which means that the PAF vector field is a vector field that only describes the direction but not the magnitude. Then, for each pair of connections c in the image, the PAF vector field is defined as the vector from point A to point B in the region. For each pixel position (x, y) in the image, if the position is within the connected region, the value of the PAF vector field is the above vector; otherwise, the value of the PAF vector field is zero, thus constructing a vector field.

[0061] Through the neural network, the probability of all human joints in the depth image is predicted, and the predicted heat map nodes are constructed to capture the human joint information. According to the joint prediction results, each joint is classified from bottom to top to a certain person through the PAF vector field. For example, there are three node predictions representing the left hand in the depth image, which are linked to the corresponding three left arm nodes through PAF, and finally summarized into a person's skeleton to generate complete skeleton information.

[0062] Step 4: Filter out unnecessary sensitive information

[0063] After the PAF vector field skeleton is extracted in step 3, the privacy protection module re-establishes the skeleton information in this step and places the skeleton information in the background image obtained in step 2, such as Figure 3 As shown on the right, Figure 3 The lower left is the original depth map information.

[0064] In addition, the privacy protection module can filter out unnecessary sensitive information when applying foreground and background separation technology, such as private scenes such as bathrooms and bedrooms that may expose privacy. Figure 4 The complete privacy protection process is shown.

[0065] Step 5: Detect whether there is a fall event or abnormal waving action by analyzing the extracted human posture, including:

[0066] Posture stability judgment: By analyzing the position information and bounding box ratio between each node of the skeleton, the posture stability of the human body is comprehensively judged.

[0067] Fall event detection: When the posture stability is lower than the preset threshold, the system determines it as a fall event. Pure skeleton node position information is not real-time and reliable, so the boundingbox is introduced to enhance the overall robustness of the system.

[0068] Abnormal waving detection: By detecting specific gestures, determine whether there is an abnormal waving action. When a specific gesture is detected, the system determines it as an abnormal waving. This event detection is usually used for abnormal situations such as slow falls. If danger or accidents occur, the alarm operation can be triggered by specific waving. Figure 3 The wave state is indicated in the upper left corner.

[0069] Step 6: When a fall event or abnormal waving action is detected, the system triggers an alarm mechanism. The detection criteria can be determined based on the stability of the human body posture, changes in joint angles, etc., including:

[0070] Step 6.1, generate alarm information; the alarm information includes key information such as event type (fall or abnormal waving), location and time of the event, etc., which will be packaged and prepared to be sent to the monitoring center;

[0071] Step 6.2, sending the alarm information to the remote monitoring center through the wireless communication module; the communication method may include WiFi, cellular network or other appropriate wireless communication technology to ensure that the alarm information can be transmitted to the monitoring center in a timely and accurate manner;

[0072] Step 6.2: After receiving the alarm information, the remote monitoring center will quickly process and respond, such as dispatching rescue personnel or initiating other emergency measures.

[0073] The system of the present invention has strong integration and is compatible with other functional modules of the inspection robot to realize an integrated safety monitoring solution. Through an efficient data interaction mechanism, information sharing and data exchange can be realized between the functional modules, thereby improving the inspection robot's ability to handle complex tasks.

[0074] The method of the present invention is integrated with the control module, navigation module, communication module, etc. of the inspection robot to realize data sharing and interaction. Through the efficient data interaction mechanism, each functional module can work together to improve the overall performance of the inspection robot.

[0075] The method of the present invention is applicable to a variety of scenarios, including factories, warehouses, hospitals and other places that require efficient and safe inspections. It has broad application prospects and market value, and can significantly improve the intelligence level of inspection robots. Through real-time monitoring and accurate identification of falls and abnormal waving movements, inspection robots can provide reliable safety guarantees. For example, in a factory environment, inspection robots can conduct inspections on production lines to promptly discover and alarm potential safety hazards; in warehouses, inspection robots can monitor the safety of staff, especially in high-risk areas for key inspections; in hospitals, inspection robots can help monitor the safety of patients, promptly discover emergencies such as falls and alarm, and ensure that patients receive timely assistance.

[0076] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A service robot behavior recognition method based on PAF vector field algorithm and depth map, characterized in that: The method comprises the following steps: Step 1: Use a depth camera to obtain a depth image; Step 2: Use the foreground and background separation algorithm to process the depth image, separate the human body information in the foreground of the depth image from the background, and output the background image; Step 3: Use the PAF vector field algorithm to process the depth image, extract the key points of the human skeleton, generate skeleton information, and obtain the human posture based on the skeleton information; Step 4, using a privacy protection module to call the background image obtained in step 2, and placing the skeleton information obtained in step 3 in the background image; Step 5: Detect whether there is a fall event or abnormal waving action by analyzing the extracted human posture; Step 6: If a fall event or abnormal waving action is detected, the alarm mechanism is triggered.

2. The service robot behavior recognition method based on PAF vector field algorithm and depth map as claimed in claim 1, characterized in that: The step 1 also includes: Step 1.1, use a depth camera to obtain a depth image of the inspection area, and compress the depth image value by 1 / 16; before starting the inspection task, use the depth image taken by the depth camera for the first time as the background depth image; Step 1.2: preprocess the depth image, including extracting the maximum depth value information in the image to limit the detection range and generate a three-channel image.

3. The service robot behavior recognition method based on PAF vector field algorithm and depth map as claimed in claim 2, characterized in that: The step 2 also includes: Step 2.1, calculate the infrared image difference between the current real-time acquired depth image and the background depth image, if the difference is less than the threshold, set the pixel value of the depth image to 0; Step 2.2: Use morphological operations to remove noise from the depth image and retain the human body information in the foreground; Step 2.3: Use the privacy protection module to filter out the human body, supplement it with the background, and output the background image.

4. The service robot behavior recognition method based on PAF vector field algorithm and depth map as claimed in claim 3, characterized in that: The morphological operation includes: performing an opening operation to remove small white noise points, and performing a closing operation to fill in black points.

5. The service robot behavior recognition method based on PAF vector field algorithm and depth map as claimed in claim 1, characterized in that: The step 3 also includes: Step 3.1, extract features from the depth image through a multi-layer convolutional neural network to obtain feature map information; Step 3.2, generate PAF vector field and joint point prediction heat map according to feature map information; Step 3.3: Based on the joint point prediction results, each joint point is classified into a recognized person from bottom to top through the PAF vector field to generate skeleton information.

6. The service robot behavior recognition method based on PAF vector field algorithm and depth map as claimed in claim 1, characterized in that: The step 5 also includes determining posture stability; by analyzing the position information between the nodes of the skeleton and the bounding box ratio, the posture stability of the human body is determined.

7. The service robot behavior recognition method based on PAF vector field algorithm and depth map as claimed in claim 6, characterized in that: The detecting of the fall event in step 5 includes: when the posture stability is lower than a preset threshold, determining it as a fall event.

8. The service robot behavior recognition method based on PAF vector field algorithm and depth map as claimed in claim 1, characterized in that: The detecting of abnormal waving action in step 5 includes: determining whether there is an abnormal waving action by detecting a specific hand gesture, and determining that the specific hand gesture is an abnormal waving action when the specific hand gesture is detected.

9. The service robot behavior recognition method based on PAF vector field algorithm and depth map as claimed in claim 1, characterized in that: The step 6 also includes: Step 6.1, generating alarm information; the alarm information includes the event type, the location where the event occurred, and the time when the event occurred; Step 6.2, sending the alarm information to a remote monitoring center via a wireless communication module; Step 6.3: After receiving the alarm information, the remote monitoring center processes and responds.

10. The service robot behavior recognition method based on PAF vector field algorithm and depth map as claimed in claim 1, characterized in that: The privacy protection module can process the depth image and filter designated sensitive information, where the designated sensitive information includes a private place.

Images