An emergency monitoring robot combined with Raspberry Pi for distortion monitoring

Abstract

The present invention discloses an emergency monitoring robot combined with Raspberry Pi for distortion removal, which relates to the technical field of image processing. It includes a visual recognition module, and the visual recognition module is used to perform distortion removal processing on the images collected by the emergency monitoring robot and identify the target objects in the images. The visual recognition module includes: an image acquisition unit, a distortion removal processing unit, and a target recognition unit. The image acquisition unit is connected to the distortion removal processing unit through a network signal, and the distortion removal processing unit is connected to the target recognition unit through a network signal. By designing the visual recognition module, the present invention realizes the recognition function of the target objects in the images, solves the problem of image distortion caused by camera distortion, improves the usability and accuracy of the images, and improves the recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and specifically to an emergency monitoring robot combined with Raspberry Pi for distortion removal monitoring. Background Art

[0002] With the continuous development of modern industry, intelligent manufacturing and automation have become the main trends of industrial development, and robot technology has developed rapidly, especially in emergency management and single-person single-post operations. In emergency situations, the safety issue of single-person single-post operations is particularly prominent, and effective monitoring means are required. Emergency monitoring robots can perform real-time monitoring and early warning of operators.

[0003] Emergency monitoring robots monitor the status of operators in real time through cameras. However, there are serious distortion problems when cameras monitor the status of operators. Therefore, the collected images will be distorted, which affects the detection accuracy. Therefore, an emergency monitoring robot combined with Raspberry Pi for distortion removal monitoring is extremely important.

[0004] Patent document CN111127514B discloses a method and device for a robot to track a target. The above patent realizes the uninterrupted real-time tracking of the target by the robot, ensures the service efficiency, service quality and activity effect of the robot, and has high tracking efficiency and accuracy. However, the above patent cannot realize the recognition function of the target object in the image.

[0005] Patent document CN114399529B discloses a target selection model for robot interaction and a robot interaction system. The above patent realizes the screening of specific groups of people that meet the robot service scenario. However, the above patent cannot realize the real-time tracking function of operators.

[0006] Patent document CN113052907B discloses a positioning method for a mobile robot in a dynamic environment. The above patent realizes the accurate positioning of the mobile robot in a dynamic environment and improves the accuracy of moving object recognition. However, the above patent cannot realize the fall detection function of operators.

[0007] Patent document CN107992881B discloses a robot dynamic grasping method and system. The above patent realizes high-frame-rate real-time recognition, improves the running speed of the visual motion target tracking algorithm, and improves the accuracy of visual servo. However, the above patent cannot realize the status detection function of operators.

[0008] In summary, the above-mentioned patent cannot achieve the function of identifying target objects in images, cannot achieve the function of real-time tracking of operators, cannot achieve the function of detecting the falls of operators, and cannot achieve the function of detecting the status of operators, resulting in the problems that the emergency monitoring robot cannot identify operators or obstacles in images, cannot conduct real-time monitoring of the operations of operators, cannot detect the falling behavior of operators in a timely manner, and cannot detect the fatigue status of operators;

[0009] Therefore, this application proposes an emergency monitoring robot that combines a Raspberry Pi for distortion monitoring and can achieve the function of identifying target objects in images, the function of real-time tracking of operators, the function of detecting the falls of operators, and the function of detecting the status of operators. Summary of the Invention

[0010] The purpose of the present invention is to provide an emergency monitoring robot that combines a Raspberry Pi for distortion monitoring, so as to solve the technical problems in the above-mentioned background technology that it cannot achieve the function of identifying target objects in images, cannot achieve the function of real-time tracking of operators, cannot achieve the function of detecting the falls of operators, cannot achieve the function of detecting the status of operators, resulting in the emergency monitoring robot being unable to identify operators or obstacles in images, unable to conduct real-time monitoring of the operations of operators, unable to detect the falling behavior of operators in a timely manner, and unable to detect the fatigue status of operators.

[0011] To achieve the above purpose, the present invention provides the following technical solution: An emergency monitoring robot that combines a Raspberry Pi for distortion monitoring, including a visual recognition module, where the visual recognition module is used to perform distortion removal processing on the images collected by the emergency monitoring robot and identify target objects in the images;

[0012] The visual recognition module includes: an image acquisition unit, a distortion removal processing unit, and a target recognition unit. The image acquisition unit is connected to the distortion removal processing unit through a network signal, and the distortion removal processing unit is connected to the target recognition unit through a network signal;

[0013] The image acquisition unit captures image information in real time through a wide-angle camera carried by the emergency monitoring robot, transmits the image data from the wide-angle camera to the Raspberry Pi through a wireless transmission method, and stores the image data in the memory of the Raspberry Pi;

[0014] The distortion removal processing unit uses the OpenCV library in the Raspberry Pi to extract the image data in the memory, runs the distortion removal algorithm, performs distortion removal processing on each frame of the image, and corrects the image by calculating the correspondence between the distorted points and the undistorted points;

[0015] The target recognition unit uses image processing algorithms to extract the feature information in the undistorted image, analyzes and compares the extracted features, recognizes the target objects existing in the image, determines the categories of the target objects, and recognizes the operators and obstacles in the image.

[0016] Preferably, the target recognition unit is connected to a target tracking module through a network signal. The target tracking module is used to continuously track the operator and obtain the actual distance between the emergency monitoring robot and the operator in real time.

[0017] The target tracking module includes: a real-time tracking unit, a distance measurement unit, and a real-time adjustment unit. The real-time tracking unit is connected to the distance measurement unit through a network signal. The distance measurement unit is connected to the real-time adjustment unit through a network signal. The real-time tracking unit is connected to the target recognition unit through a network signal.

[0018] The real-time tracking unit uses a tracking algorithm to continuously track the operator in the image, analyzes the movement trajectory of the operator between consecutive image frames, determines the position of the operator in the image, and uses the image shaping method to calculate the real width of the operator.

[0019] The distance measurement unit uses the camera ranging method to calculate the distance between the emergency monitoring robot and the operator, and sets the expected distance between the emergency monitoring robot and the operator in the safety monitoring mode and the service mode.

[0020] The real-time adjustment unit calculates the offset of the operator from the center of the camera's field of view according to the position of the operator in the image, calculates the deviation distance according to the actual distance and the expected distance between the emergency monitoring robot and the operator, and uses a PID controller to move the emergency monitoring robot in real time.

[0021] Preferably, the target recognition unit is connected to a posture recognition module through a network signal. The fall detection module is used to recognize the posture actions of the operator and judge the user's fall behavior.

[0022] The posture recognition module includes: a key point detection unit, a key point association unit, and a fall detection unit. The key point detection unit is connected to the key point association unit through a network signal. The key point association unit is connected to the fall detection unit through a network signal. The key point detection unit is connected to the target recognition unit through a network signal.

[0023] The key point detection unit uses the OpenPose algorithm to detect key points in the image, extracts the key points of the operator in the image, including the joint points of the human body, and accurately locates the position of each key point in the image.

[0024] The key point association unit matches the detected key points with the human body structure model, compares the position information of the key points and the human body structure features, matches and associates the key points of the same human body, and constructs the posture information of the operator;

[0025] The fall detection unit analyzes the posture information, extracts the characteristic parameters of the posture, calculates the descending speed of the Hip joint in consecutive image frames, compares the Hip joint descending speed with a preset threshold, and determines the fall event of the operator.

[0026] Preferably, the target recognition unit is connected with a face recognition module through a network signal, and the face recognition module is used for face recognition and status detection of the operator;

[0027] The face recognition module includes: a face recognition unit, a status detection unit, and a status reminder unit. The face recognition unit is connected with the status detection unit through a network signal, the status detection unit is connected with the status reminder unit through a network signal, and the face recognition unit is connected with the target recognition unit through a network signal;

[0028] The face recognition unit locates the facial feature points of the operator in the image, extracts the geometric features and texture features of the face, compares the extracted facial features with the face database in the Raspberry Pi, and identifies the identity of the operator;

[0029] The status detection unit calculates the aspect ratio of the eyes, the aspect ratio of the mouth, and the Euler angle offset of the head posture based on the facial feature points, and judges the fatigue status of the operator;

[0030] When the status reminder unit detects that the operator is fatigued, it emits a warning signal through a buzzer to remind the operator to rest.

[0031] Preferably, the real-time adjustment unit is connected with a route planning module through a network signal, and the route planning module is used to detect obstacles in the image and plan the moving route of the emergency monitoring robot;

[0032] The route planning module includes: an obstacle recognition unit, a strategy formulation unit, and an obstacle avoidance evaluation unit. The obstacle recognition unit is connected with an obstacle avoidance execution unit through a network signal, the obstacle avoidance execution unit is connected with the obstacle avoidance evaluation unit through a network signal, the obstacle recognition unit is connected with the target recognition unit through a network signal, and the strategy formulation unit is connected with the real-time adjustment unit through a network signal;

[0033] The obstacle recognition unit uses the image forming method to calculate the real size of the obstacle in the image, sets the size threshold of the obstacle, and marks the obstacle exceeding the threshold as a potential obstacle affecting the movement of the emergency monitoring robot;

[0034] The strategy formulation unit block uses a path planning algorithm to calculate a collision-free path from the current position to the target position, optimizes the calculated path, and sets up an emergency obstacle avoidance mechanism, including emergency stop, quick turn, and reverse;

[0035] The obstacle avoidance evaluation unit records the data during the obstacle avoidance process of the emergency care robot, including the position and size information of the obstacles, as well as the movement trajectory and obstacle avoidance measures of the emergency care robot.

[0036] Preferably, the fall detection unit is connected to a dynamic interaction module through a network signal, and the dynamic interaction module is used for the interaction between the emergency care robot and the operator;

[0037] The dynamic interaction module includes: an emergency response unit, a voice call unit, and an interaction feedback unit. The emergency response unit is connected to the interaction feedback unit through a network signal, the voice call unit is connected to the interaction feedback unit through a network signal, and the emergency response unit is connected to the fall detection unit through a network signal;

[0038] When the emergency care robot is in the safety monitoring mode, the emergency response unit detects that the operator is in a fallen state, triggers an alarm mechanism, sends an alarm message to the safety person in charge using a phone, and emits audible and visual alarms;

[0039] When the emergency care robot is in the service mode, the voice call unit synthesizes and replies with the voice of "delivering tools" after recognizing the word "tool" in the operator's voice command, and controls the emergency care robot to approach the operator through a PID controller;

[0040] The interaction feedback unit records the operator's voice commands, the responses of the emergency care robot, and alarm information, and provides a user interaction interface for switching the safety monitoring mode, service mode of the emergency care robot, and querying historical information.

[0041] Preferably, the real-time adjustment unit is connected to a route planning module through a network signal, and the route planning module is used to detect obstacles in the image and plan the movement route of the emergency care robot;

[0042] The route planning module includes: an obstacle recognition unit, a strategy formulation unit, and an obstacle avoidance evaluation unit. The obstacle recognition unit is connected to an obstacle avoidance execution unit through a network signal, the obstacle avoidance execution unit is connected to the obstacle avoidance evaluation unit through a network signal, the obstacle recognition unit is connected to a target recognition unit through a network signal, and the strategy formulation unit is connected to the real-time adjustment unit through a network signal;

[0043] The obstacle recognition unit uses the image forming method to calculate the real size of the obstacles in the image, sets a size threshold for the obstacles, and marks the obstacles exceeding the threshold as potential obstacles affecting the movement of the emergency care robot;

[0044] The strategy formulation unit block uses a path planning algorithm to calculate a collision-free path from the current position to the target position, optimizes the calculated path, and sets up an emergency obstacle avoidance mechanism, including emergency stop, quick turn, and reverse;

[0045] The obstacle avoidance evaluation unit records the data during the obstacle avoidance process of the emergency monitoring robot, including the position and size information of the obstacles, as well as the motion trajectory and obstacle avoidance measures of the emergency monitoring robot.

[0046] Preferably, the distortion removal algorithm includes the following steps:

[0047] Step 1, camera calibration: Take calibration board images at different angles, process the images using a calibration algorithm, and calculate the internal parameter matrix and distortion coefficients of the camera;

[0048] Step 2, distortion model establishment: Based on the calibration results, establish a distortion model of the camera, including radial distortion and tangential distortion;

[0049] Step 3, distortion correction parameter calculation: Calculate the mapping relationship required for distortion correction based on the calibration results, and generate two mapping tables;

[0050] Step 4, image distortion removal processing: Use the remap function to perform distortion removal processing in combination with the mapping tables calculated in Step 3. The remap function maps each pixel in the distorted image to a new position after distortion removal according to the mapping tables, generating a distortion-free image.

[0051] Preferably, the image shaping method includes the following steps:

[0052] Establish a geometric model: Based on the internal parameter matrix and distortion parameters of the camera, establish a geometric model of camera imaging;

[0053] Calculate the ratio: Select an object with a known true width in the image, measure the pixel width of the object in the image, and calculate the ratio relationship based on the true width and pixel width of the object;

[0054] Calculate the true size: Use the geometric model and the ratio relationship to convert the feature points of the operator in the image coordinates into world coordinates, and calculate the true width of the operator based on the converted world coordinates.

[0055] Preferably, the image processing algorithm includes: edge detection, edge detection, texture feature extraction, and shape feature extraction

[0056] Preferably, the formula for the camera ranging method is:

[0057] D = (W * F) / P;

[0058] Where D is the distance between the camera and the operator, W is the actual width of the operator, and P is the pixel width of the operator in the image.

[0059] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0060] 1. By designing a visual recognition module, the present invention realizes the recognition function of target objects in images, solves the problem of image distortion caused by camera distortion, improves the usability and accuracy of images, and improves the recognition accuracy.

[0061] 2. By designing a target tracking module, the present invention realizes the real-time tracking function of the operator, ensures that the operator is in the center of the camera's field of view and maintains a desired distance from the robot, and improves the accuracy of monitoring.

[0062] 3. By designing a posture recognition module, the present invention realizes the function of detecting the operator's fall, solves the problem that it is difficult to detect the operator's fall, and improves the emergency response speed and the safety of the operator.

[0063] 4. By designing a face recognition module, the present invention realizes the function of detecting the operator's state, accurately recognizes the operator's face, improves the accuracy and efficiency of monitoring, and realizes comprehensive and accurate monitoring of the operator. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 is a schematic diagram of the visual recognition module of the present invention;

[0065] Figure 2 is a schematic diagram of the target tracking module of the present invention;

[0066] Figure 3 is a schematic diagram of the posture recognition module of the present invention;

[0067] Figure 4 is a schematic diagram of the face recognition module of the present invention;

[0068] Figure 5 is a schematic diagram of the route planning module of the present invention;

[0069] Figure 6 is a schematic diagram of the dynamic interaction module of the present invention;

[0070] Figure 7 is a schematic diagram of the flow chart of using the distortion algorithm of the present invention;

[0071] Figure 8 is a schematic diagram of the flow chart of the emergency monitoring robot system of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0072] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0073] Example 1, please refer to Figure 1 、 Figure 2 、 Figure 7 and Figure 8 ,An emergency monitoring robot combined with Raspberry Pi for distortion removal monitoring, including a visual recognition module. The visual recognition module is used to perform distortion removal processing on the images collected by the emergency monitoring robot and identify the target objects in the images. The visual recognition module includes: an image acquisition unit, a distortion removal processing unit, and a target recognition unit. The image acquisition unit is connected to the distortion removal processing unit through a network signal, and the distortion removal processing unit is connected to the target recognition unit through a network signal. The image acquisition unit captures image information in real time through a wide-angle camera carried by the emergency monitoring robot, transmits the image data from the wide-angle camera to the Raspberry Pi through a wireless transmission method, and stores the image data in the memory of the Raspberry Pi. The distortion removal processing unit uses the OpenCV library in the Raspberry Pi to extract the image data in the memory, runs the distortion removal algorithm, performs distortion removal processing on each frame of the image, and corrects the image by calculating the corresponding relationship between the distorted points and the undistorted points. The target recognition unit uses image processing algorithms to extract the feature information in the distorted image, analyzes and compares the extracted features, identifies the target objects existing in the image, determines the categories of the target objects, and identifies the operators and obstacles in the image.

[0074] The real-time tracking unit uses a tracking algorithm to continuously track the operator in the image, analyzes the movement trajectory of the operator between consecutive image frames, determines the position of the operator in the image, and uses the image shaping method to calculate the real width of the operator. The distance measurement unit uses the camera ranging method to calculate the distance between the emergency monitoring robot and the operator, and sets the desired distance between the emergency monitoring robot and the operator in the safety monitoring mode and the service mode. The real-time adjustment unit calculates the offset of the operator from the center of the camera's field of view according to the position of the operator in the image, calculates the deviation distance according to the actual distance and the desired distance between the emergency monitoring robot and the operator, and uses a PID controller to move the emergency monitoring robot in real time.

[0075] Furthermore, the image acquisition unit within the visual recognition module uses a wide-angle camera to capture the image information of the surrounding environment in real time. The wide-angle camera provides a broader field of view, facilitating the all-round monitoring of the robot in complex environments. The wireless transmission technology Wi-Fi is adopted to achieve data transmission between the camera and the Raspberry Pi. The image acquisition unit stores the image data in the memory of the Raspberry Pi. The Raspberry Pi has powerful data processing and storage capabilities, ensuring the integrity and availability of the image data. When the wide-angle camera captures images, distortion will occur, resulting in a large gap between the read image and the actual image, thus affecting the subsequent extraction of feature information and reducing the accuracy of state detection and fall detection. The distortion correction unit uses the Raspberry Pi OpenCV to read the image data in the Raspberry Pi memory and perform distortion correction on the image to improve the accuracy and clarity of the image. The target recognition unit receives the image processed by the distortion correction unit and extracts the feature information in the undistorted image through image processing algorithms within the Raspberry Pi, including edge detection, corner detection, texture feature extraction, and shape feature extraction, etc. The extracted features are compared and analyzed with the feature library within the Raspberry Pi to determine whether there is an object in the image that matches the target object in the feature library, analyze the category of the target object in the image, and judge whether the target object is an operator or an obstacle.

[0076] After the target recognition unit in the visual recognition module detects the presence of an operator in the image, the target recognition unit transmits the undistorted image and the target recognition result of the image to the target tracking module. The real-time tracking unit in the target tracking module uses the tracking algorithm to track the operator recognized in the image based on the target recognition result, continuously locking the position of the operator. When it detects a change in the position of the operator in consecutive image frames, the target recognition unit triggers the distance measurement unit to calculate the distance. At the same time, the target recognition unit further processes the feature information of the operator in the image through the Raspberry Pi. By the contour feature information of the operator in the image, the pixel width of the operator is extracted, and then the image shaping method is used to calculate the real width of the operator in the image, and the pixel width and real width data of the operator are transmitted to the distance measurement unit; the distance measurement unit receives the data transmitted by the target recognition unit, uses the camera ranging method, and calculates the distance between the operator in the image and the camera, that is, the distance between the operator and the robot, according to the principle of similar triangles. At the same time, the distance measurement unit sets the desired distance ranges for the safety guardianship mode and the service mode according to the requirements of the application scenario, so as to ensure the safety distance between the emergency guardianship robot and the operator; the real-time tracking unit transmits the position change and prediction data to the real-time adjustment unit. The real-time adjustment unit calculates the offset of the operator from the center of the camera's field of view according to the position of the operator in the image. The real-time adjustment unit initially determines the moving direction and distance of the robot according to the offset. After the distance measurement unit transmits the actual distance and the desired distance between the operator and the robot to the real-time adjustment unit, the real-time adjustment unit determines the deviation distance according to the difference between the actual distance and the desired distance, so as to further determine the moving direction and distance of the robot. The real-time adjustment unit adjusts the position of the robot in real time through the PID controller, so as to ensure that the operator is in the center of the camera's field of view and ensure that the distance between the robot and the operator is the desired distance.

[0077] Embodiment 2, please refer to Figure 1 , Figure 3 and Figure 8, an emergency monitoring robot combined with Raspberry Pi for distortion removal and monitoring. The target recognition unit uses image processing algorithms to extract feature information from the distortion-removed image, analyzes and compares the extracted features, identifies the target objects existing in the image, determines the categories of the target objects, and identifies the operators and obstacles in the image. The target recognition unit is connected with a posture recognition module through a network signal. The fall detection module is used to recognize the posture actions of the operator and judge the user's falling behavior. The posture recognition module includes: a key point detection unit, a key point association unit, and a fall detection unit. The key point detection unit is connected with the key point association unit through a network signal. The key point association unit is connected with the fall detection unit through a network signal. The key point detection unit is connected with the target recognition unit through a network signal. The key point detection unit uses the OpenPose algorithm to detect key points in the image, extracts the key points of the operator in the image, including the joint points of the human body, and accurately locates the position of each key point in the image. The key point association unit matches the detected key points with the human body structure model, compares the position information of the key points and the human body structure features, matches and associates the key points of the same human body, and constructs the posture information of the operator. The fall detection unit analyzes the posture information, extracts the characteristic parameters of the posture, calculates the descending speed of the Hip joint in consecutive image frames, compares the Hip joint descending speed with a preset threshold, and judges the fall event of the operator.

[0078] Furthermore, when the target recognition unit detects an operator in the image, the target recognition unit transmits the distortion-removed image and the target recognition result of the image to the posture recognition module. According to the target recognition result, the key point detection unit in the posture recognition module uses the OpenPose algorithm to detect key points in the image. The OpenPose algorithm extracts features of the operator in the image through a deep convolutional neural network, extracts key points such as the joint points of the human body, such as shoulders, elbows, wrists, knees, ankles, etc., and accurately locates the position of each key point in the image, thereby providing basic data for subsequent analysis and processing. After receiving the key point data transmitted by the key point detection unit, the key point association unit associates the detected key points with the human body structure model, and matches and associates the key points of the same human body by comparing the position information of the key points and the human body structure features, thereby constructing complete posture information and avoiding false detection and missed detection. The fall detection unit analyzes the posture information, analyzes the positions, angles and mutual relationships of various parts of the operator's body, extracts the characteristic parameters of the posture, such as the descending speed of the Hip joint, etc. The fall detection unit calculates the descending speed of the Hip joint in consecutive image frames. By comparing the Hip joint descending speed with a preset threshold, if the descending speed exceeds the threshold, it is judged that the operator has a fall event, thereby timely discovering the abnormal situation of the operator and providing an important basis for emergency response.

[0079] Example 3, please refer to Figure 1 、 Figure 4 and Figure 8 , an emergency monitoring robot combined with Raspberry Pi for distortion removal monitoring. The target recognition unit uses an image processing algorithm to extract feature information in the undistorted image, analyzes and compares the extracted features, identifies the target objects existing in the image, determines the categories of the target objects, and identifies the operators and obstacles in the image; the key point association unit matches the detected key points with the human body structure model, compares the position information of the key points and the human body structure features, matches and associates the key points of the same human body, and constructs the pose information of the operator; the target recognition unit is connected with a face recognition module through a network signal, and the face recognition module is used for face recognition and status detection of the operator; the face recognition module includes: a face recognition unit, a status detection unit, and a status reminder unit. The face recognition unit is connected with the status detection unit through a network signal, the status detection unit is connected with the status reminder unit through a network signal, and the face recognition unit is connected with the target recognition unit through a network signal; the face recognition unit locates the facial feature points of the operator in the image, extracts the geometric features and texture features of the face, compares the extracted facial features with the face database in the Raspberry Pi, and identifies the identity of the operator; the status detection unit calculates the aspect ratio of the eyes, the aspect ratio of the mouth, and the Euler angle offset of the head pose according to the facial feature points, and judges the fatigue state of the operator; the status reminder unit issues a warning signal through a buzzer when it detects that the operator is fatigued, reminding the operator to rest.

[0080] Further, after the target recognition unit detects the presence of an operator in the image, the target recognition unit transmits the undistorted image and the target recognition result of the image to the face recognition module. According to the operator's pose information transmitted by the key point association unit, the face recognition unit in the pose recognition module quickly locates the head of the operator in the image, thereby facilitating the face recognition unit to locate the facial feature points of the operator in the image, and extract the geometric features and texture features of the face. Analyze the positional relationship of the operator's eyes, nose, and mouth according to the geometric features, and analyze the operator's skin texture, spots, etc. according to the texture features. Then, the face recognition unit compares the extracted features with the face features in the Raspberry Pi database to determine the identity of the operator; after the face recognition unit determines the identity of the operator, it notifies the status detection unit to perform the status detection of the operator. The status detection unit receives the facial features extracted by the face recognition unit and calculates parameters such as the aspect ratio of the eyes, the aspect ratio of the mouth, and the Euler angle offset of the head pose according to the facial feature points. When the operator is fatigued, such as when the operator's eyes are squinted, closed, or the head is drooping or tilted, it will cause changes in parameters such as the aspect ratio of the eyes or the Euler angle offset of the head pose. The status detection unit compares the facial feature parameters of the image with the threshold. When the facial feature parameters of multiple consecutive frames of images are higher or lower than the threshold, the status detection unit determines that the operator is in a fatigued state; the status detection unit transmits a fatigue signal to the status reminder unit, and after receiving the signal, the status reminder unit controls the buzzer on the robot to emit a warning signal to remind the operator to rest.

[0081] Embodiment 4, please refer to Figure 1 , Figure 2 , Figure 5 and Figure 8, An emergency monitoring robot combined with a Raspberry Pi for distortion removal monitoring. The target recognition unit uses an image processing algorithm to extract feature information from the undistorted image, analyzes and compares the extracted features, identifies the target objects existing in the image, determines the categories of the target objects, and identifies the operators and obstacles in the image. The real-time adjustment unit calculates the offset between the operator and the center of the camera's field of view according to the position of the operator in the image, calculates the deviation distance according to the actual distance and the desired distance between the emergency monitoring robot and the operator, and uses a PID controller to move the emergency monitoring robot in real time. The real-time adjustment unit is connected to a route planning module through a network signal. The route planning module is used to detect obstacles in the image and plan the movement route of the emergency monitoring robot. The route planning module includes: an obstacle recognition unit, a strategy formulation unit, and an obstacle avoidance evaluation unit. The obstacle recognition unit is connected to an obstacle avoidance execution unit through a network signal. The obstacle avoidance execution unit is connected to an obstacle avoidance evaluation unit through a network signal. The obstacle recognition unit is connected to the target recognition unit through a network signal. The strategy formulation unit is connected to the real-time adjustment unit through a network signal. The obstacle recognition unit uses the image formation method to calculate the real size of the obstacles in the image, sets the size threshold of the obstacles, and marks the obstacles exceeding the threshold as potential obstacles affecting the movement of the emergency monitoring robot. The strategy formulation unit uses a path planning algorithm to calculate a collision-free path from the current position to the target position, optimizes the calculated path, and sets an emergency obstacle avoidance mechanism, including emergency stop, quick turn, and reverse. The obstacle avoidance evaluation unit records the data during the obstacle avoidance process of the emergency monitoring robot, including the position and size information of the obstacles, as well as the movement trajectory and obstacle avoidance measures of the emergency monitoring robot.

[0082] Furthermore, to ensure that the operator is in the center of the camera's field of view and at a desired distance from the robot, the real-time adjustment unit moves the emergency monitoring robot in real time. During the movement, when the target recognition unit recognizes an obstacle in the image, the target recognition unit transmits the undistorted image and the target recognition result of the image to the route planning module. After receiving the image, the obstacle recognition unit in the route planning module uses the image shaping method to calculate the real size of the obstacle in the image and compares the real size of the obstacle with the threshold data. When the real size of the obstacle exceeds the threshold, it indicates that this obstacle will affect the movement of the emergency monitoring robot and the emergency monitoring robot needs to avoid it. At this time, the obstacle recognition unit will mark the obstacle that exceeds the threshold as a potential obstacle and mark it in the image. The strategy formulation unit receives the marked image. The strategy formulation unit calculates a collision-free path from the current position to the target position based on the image, optimizes it according to the actual situation, and sets up an emergency obstacle avoidance mechanism, such as emergency stop, quick turn, and reverse, etc., so that the robot can quickly respond when encountering unexpected situations and avoid collisions. The strategy formulation unit transmits the calculated and optimized route to the real-time adjustment unit, and the real-time adjustment unit moves the emergency monitoring robot according to the route. During the obstacle avoidance process, the obstacle avoidance evaluation unit records in real time data such as the position and size information of the obstacle, the movement trajectory of the robot, and the obstacle avoidance measures, analyzes and evaluates the recorded data, including judging whether the robot has successfully avoided all obstacles and whether it moves along the planned path, etc., so as to judge the rationality of the path planning and the effectiveness of the obstacle avoidance measures, etc. The obstacle avoidance evaluation unit optimizes the path planning algorithm of the obstacle module according to the evaluation data, thereby improving the accuracy of the subsequent route planning of the strategy formulation unit.

[0083] Example 5, please refer to Figure 3 , Figure 6 and Figure 8, an emergency monitoring robot combined with Raspberry Pi distortion removal monitoring, a fall detection unit analyzes posture information, extracts characteristic parameters of the posture, calculates the descent speed of the Hip joint in continuous image frames, compares the descent speed of the Hip joint with a preset threshold, and determines the fall event of the operator; the fall detection unit is connected to a dynamic interaction module through a network signal, and the dynamic interaction module is used for the interaction between the emergency monitoring robot and the operator; the dynamic interaction module includes: an emergency response unit, a voice call unit and an interactive feedback unit, the emergency response unit is connected to the interactive feedback unit through a network signal, the voice call unit is connected to the interactive feedback unit through a network signal, and the emergency response unit is connected to the interactive feedback unit through a network signal It is connected with a fall detection unit; when the emergency monitoring robot is in the safety monitoring mode, the emergency response unit detects that the operator is in a fall state, triggers the alarm mechanism, uses the phone to send an alarm message to the safety person in charge, and issues an audible and visual alarm; when the emergency monitoring robot is in the service mode, the voice call unit recognizes that "tools" appear in the operator's voice command, synthesizes and replies with the voice of "tools being transported", and controls the emergency monitoring robot to approach the operator through the PID controller; the interactive feedback unit records the operator's voice commands, the emergency monitoring robot's response and alarm information, and provides a user interactive interface for switching the emergency monitoring robot's safety monitoring mode, service mode, and querying historical information.

[0084] Furthermore, after the operator starts the emergency monitoring robot, the operating mode of the emergency monitoring robot is switched through the user interaction interface in the interactive feedback unit. In the safety monitoring mode, after the fall detection unit detects that the operator is in a fall state, the fall detection unit transmits the information to the emergency response unit. After receiving the information, the emergency response unit immediately triggers the alarm mechanism, sends an alarm message to the safety person in charge by phone, and starts the sound and light alarm on the robot to ensure that the alarm information can be quickly conveyed to the relevant personnel and help the relevant personnel to quickly locate, thereby improving the emergency response speed; in the service mode, the voice call unit uses a voice recognition algorithm to convert the operator's voice command into text information, and searches for the specific keyword "tool" in the converted text information. When "tool" is searched, the voice call unit synthesizes and replies with the voice of "transporting tools", and controls the moving trajectory and speed of the emergency monitoring robot through the PID controller to make the emergency monitoring robot close to the operator; during the operation of the emergency monitoring robot, the interactive feedback unit records the operator's voice command, the robot's response and alarm information and other data in real time, and classifies and manages them so that the operator can query through the user interaction interface.

[0085] Example 6, please refer to Figure 2 , Figure 6 and Figure 8, An emergency monitoring robot combined with a Raspberry Pi for distortion removal and monitoring. The distance measurement unit uses the camera ranging method to calculate the distance between the emergency monitoring robot and the operator, and sets the expected distance between the emergency monitoring robot and the operator in the safety monitoring mode and the service mode; The real-time adjustment unit calculates the offset of the operator from the center of the camera's field of view according to the operator's location, calculates the deviation distance according to the actual distance and the expected distance between the emergency monitoring robot and the operator, and uses a PID controller to move the emergency monitoring robot in real time; The interaction feedback unit records the voice commands of the operator, the responses of the emergency monitoring robot, and the alarm information, and provides a user interface for switching the safety monitoring mode, service mode of the emergency monitoring robot, and querying historical information.

[0086] Furthermore, after the operator activates the emergency monitoring robot, the working mode of the emergency monitoring robot is switched through the user interface in the interaction feedback unit. After the operator determines the working mode, the interaction feedback unit transmits the working mode of the emergency monitoring robot to the distance measurement unit. The distance measurement unit calculates the actual distance between the emergency monitoring robot and the operator in real time. The distance measurement unit transmits the expected distance in the working mode of the emergency monitoring robot to the real-time adjustment unit. At the same time, the distance measurement unit transmits the actual distance between the emergency monitoring robot and the operator to the real-time adjustment unit. The real-time adjustment unit calculates the difference between the actual distance and the set expected distance, and the real-time adjustment unit moves the emergency monitoring robot in real time through a PID controller, so as to ensure that the actual distance between the emergency monitoring robot and the operator is the expected distance.

[0087] Working principle: The operator activates the emergency monitoring robot and switches the working mode of the emergency monitoring robot through the user interface in the dynamic interaction module. The visual recognition module captures the surrounding environment images in real time. The visual recognition module performs distortion removal processing on the images and identifies the operator and obstacles in the images. The face recognition module identifies the face of the operator in the images and confirms the identity of the operator;

[0088] After confirming the identity of the operator, the target tracking module continuously tracks the operator in the images. The target tracking module calculates the deviation distance in the corresponding working mode according to the working mode information of the emergency monitoring robot transmitted by the dynamic interaction module. The target tracking module moves the emergency monitoring robot in real time according to the deviation distance and the route planned by the route planning module;

[0089] The posture recognition module extracts the key points of the operator in the image and constructs the posture information of the operator. The posture recognition module compares the feature parameters of the posture with the threshold. At the same time, the face recognition module compares the facial feature parameters with the threshold. When the feature parameters of the posture exceed the threshold, the posture recognition module sends an alarm to the safety supervisor through the dynamic interaction module. When the facial feature parameters exceed the threshold, the face recognition module reminds the operator.

[0090] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above-described exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, in any sense, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be embraced within the present invention. Any reference signs in the claims should not be construed as limiting the claims involved.

Claims

1. An emergency monitoring robot combined with Raspberry Pi distortion removal monitoring, characterized by: It includes a visual recognition module, which is used to perform dedistortion processing on the image collected by the emergency monitoring robot and identify the target object in the image; The visual recognition module includes: an image acquisition unit, a dedistortion processing unit and a target recognition unit, wherein the image acquisition unit is connected to the dedistortion processing unit via a network signal, and the dedistortion processing unit is connected to the target recognition unit via a network signal; The image acquisition unit captures image information in real time through a wide-angle camera carried by the emergency monitoring robot, transmits the image data from the wide-angle camera to the Raspberry Pi through wireless transmission, and stores the image data in the memory of the Raspberry Pi; The dedistortion processing unit uses the OpenCV library in the Raspberry Pi to extract the image data in the memory, runs the dedistortion algorithm, performs dedistortion processing on each frame of the image, and corrects the image by calculating the correspondence between the distorted points and the undistorted points; The target recognition unit uses an image processing algorithm to extract feature information from the dedistorted image, analyzes and compares the extracted features, recognizes the target object in the image, determines the category of the target object, and recognizes the operator and obstacles in the image; The target recognition unit is connected to a target tracking module via a network signal, and the target tracking module is used to continuously track the operator and obtain the actual distance between the emergency monitoring robot and the operator in real time; The target tracking module includes: a real-time tracking unit, a distance measurement unit and a real-time adjustment unit, wherein the real-time tracking unit is connected to the distance measurement unit via a network signal, the distance measurement unit is connected to the real-time adjustment unit via a network signal, and the real-time tracking unit is connected to the target identification unit via a network signal; The real-time tracking unit uses a tracking algorithm to continuously track the operator in the image, analyzes the movement trajectory of the operator between consecutive image frames, determines the position of the operator in the image, and uses an image forming method to calculate the actual width of the operator; The distance measurement unit uses a camera distance measurement method to calculate the distance between the emergency monitoring robot and the operator, and sets the expected distance between the emergency monitoring robot and the operator in the safety monitoring mode and the service mode; The real-time adjustment unit calculates the offset between the operator and the center of the camera field of view according to the position of the operator in the image, calculates the deviation distance according to the actual distance and expected distance between the emergency monitoring robot and the operator, and uses a PID controller to move the emergency monitoring robot in real time.

2. The emergency monitoring robot combined with Raspberry Pi de-distortion monitoring according to claim 1, characterized in that: The target recognition unit is connected to a posture recognition module via a network signal, and the posture recognition module is used to recognize the posture movements of the operator and determine the user's falling behavior; The posture recognition module includes: a key point detection unit, a key point association unit, and a fall detection unit. The key point detection unit is connected to the key point association unit via a network signal. The key point association unit is connected to the fall detection unit via a network signal. The key point detection unit is connected to the target recognition unit via a network signal. The key point detection unit uses the OpenPose algorithm to detect key points of the image, extract the key points of the operator in the image, including the joints of the human body, and accurately locate the position of each key point in the image; The key point association unit matches the detected key points with the human body structure model, compares the position information of the key points and the human body structure characteristics, matches and associates the key points of the same human body, and constructs the posture information of the operator; The fall detection unit analyzes the posture information, extracts the characteristic parameters of the posture, calculates the descent speed of the Hip joint in continuous image frames, compares the descent speed of the Hip joint with a preset threshold, and determines the fall event of the operator.

3. The emergency monitoring robot combined with Raspberry Pi de-distortion monitoring according to claim 1, characterized in that: The target recognition unit is connected to a facial recognition module via a network signal, and the facial recognition module is used for face recognition and status detection of operators; The facial recognition module includes: a face recognition unit, a state detection unit, and a state reminder unit. The face recognition unit is connected to the state detection unit via a network signal, the state detection unit is connected to the state reminder unit via a network signal, and the face recognition unit is connected to the target recognition unit via a network signal. The face recognition unit locates the facial feature points of the operator in the image, extracts the geometric features and texture features of the face, compares the extracted facial features with the face database in the Raspberry Pi, and identifies the identity of the operator; The state detection unit calculates the eye aspect ratio, mouth aspect ratio and head posture Euler angle offset based on facial feature points to determine the fatigue state of the operator; When the status reminder unit detects that the operator is fatigued, it will send out a warning signal through a buzzer to remind the operator to rest.

4. The emergency monitoring robot combined with Raspberry Pi de-distortion monitoring according to claim 1, characterized in that: The real-time adjustment unit is connected to a route planning module via a network signal, and the route planning module is used to detect obstacles in the image and plan a moving route for the emergency monitoring robot; The route planning module includes: an obstacle recognition unit, a strategy formulation unit and an obstacle avoidance evaluation unit, wherein the obstacle recognition unit is connected to the obstacle avoidance execution unit via a network signal, the obstacle avoidance execution unit is connected to the obstacle avoidance evaluation unit via a network signal, the obstacle recognition unit is connected to the target recognition unit via a network signal, and the strategy formulation unit is connected to the real-time adjustment unit via a network signal; The obstacle recognition unit uses the image forming method to calculate the real size of the obstacle in the image, sets the size threshold of the obstacle, and marks the obstacles exceeding the threshold as potential obstacles that affect the movement of the emergency monitoring robot; The strategy formulation unit uses a path planning algorithm to calculate a collision-free path from the current position to the target position, optimizes the calculated path, and sets up emergency obstacle avoidance mechanisms, including emergency stop, quick turn, and back off; The obstacle avoidance evaluation unit records the data of the emergency monitoring robot during the obstacle avoidance process, including the location and size information of the obstacle, as well as the motion trajectory and obstacle avoidance measures of the emergency monitoring robot.

5. The emergency monitoring robot combined with Raspberry Pi de-distortion monitoring according to claim 2, characterized in that: The fall detection unit is connected to a dynamic interaction module via a network signal, and the dynamic interaction module is used for interaction between the emergency monitoring robot and the operator; The dynamic interaction module includes: an emergency response unit, a voice call unit and an interactive feedback unit, the emergency response unit is connected to the interactive feedback unit via a network signal, the voice call unit is connected to the interactive feedback unit via a network signal, and the emergency response unit is connected to the fall detection unit via a network signal; When the emergency monitoring robot is in the safety monitoring mode, the emergency response unit detects that the operator is in a falling state, triggers the alarm mechanism, sends an alarm message to the safety person in charge by phone, and issues an audible and visual alarm; When the emergency monitoring robot is in service mode, the voice call unit recognizes the word "tool" in the operator's voice command, synthesizes and replies with the voice of "transporting tools", and controls the emergency monitoring robot to approach the operator through the PID controller; The interactive feedback unit records the operator's voice commands, the emergency monitoring robot's responses and alarm information, and provides a user interactive interface for switching the emergency monitoring robot's safety monitoring mode, service mode, and querying historical information.

6. The emergency monitoring robot combined with Raspberry Pi de-distortion monitoring according to claim 1, characterized in that: The de-distortion algorithm comprises the following steps: Step 1: Camera calibration: Take images of the calibration plate at different angles, process the images using the calibration algorithm, and calculate the camera's intrinsic parameter matrix and distortion coefficient; Step 2: Distortion model establishment: According to the calibration results, the distortion model of the camera is established, including radial distortion and tangential distortion; Step 3, distortion correction parameter calculation: Calculate the mapping relationship required for distortion correction according to the calibration results and generate two mapping tables; Step 4: Image dedistortion: Use the remap function in combination with the mapping table calculated in step 3 to perform dedistortion. The remap function maps each pixel in the distorted image to a new position after dedistortion according to the mapping table to generate an undistorted image.

7. The emergency monitoring robot combined with Raspberry Pi de-distortion monitoring according to claim 1, characterized in that: The image forming method comprises the following steps: Establishing a geometric model: Establishing a geometric model of camera imaging based on the camera’s intrinsic parameter matrix and distortion parameters; Calculate the ratio: select an object with a known real width in the image, measure the pixel width of the object in the image, and calculate the ratio based on the real width and pixel width of the object; Calculate the real size: Use the geometric model and proportional relationship to convert the operator's feature points in the image coordinates into world coordinates, and calculate the operator's real width based on the converted world coordinates.

8. The emergency monitoring robot combined with Raspberry Pi de-distortion monitoring according to claim 1, characterized in that: The image processing algorithm includes: edge detection, corner point detection, texture feature extraction and shape feature extraction.

9. The emergency monitoring robot combined with Raspberry Pi de-distortion monitoring according to claim 2, characterized in that: The formula of the camera ranging method is: D = (W*F) / P; Where D is the distance between the camera and the operator, W is the actual width of the operator, and P is the pixel width of the operator in the image.

Images