Emergency monitoring robot combined with Raspberry Pi distortion removal monitoring

Abstract

The invention discloses an emergency monitoring robot combined with Raspberry Pi distortion removal monitoring, and relates to the technical field of image processing, the emergency monitoring robot comprises a visual identification module, and the visual identification module is used for carrying out distortion removal processing on an image collected by the emergency monitoring robot and identifying a target object in the image; the visual identification module comprises an image acquisition unit, a distortion removal processing unit and a target identification unit, the image acquisition unit is connected with the distortion removal processing unit through a network signal, and the distortion removal processing unit is connected with the target identification unit through a network signal. According to the invention, the visual identification module is designed, so that the function of identifying the target object in the image is realized, the problem of image distortion caused by camera distortion is solved, the availability and accuracy of the image are improved, and the identification precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an emergency monitoring robot combined with Raspberry Pi distortion removal monitoring. Background Art

[0002] With the continuous development of modern industry, intelligent manufacturing and automation have become the main trends of industrial development, and robotics technology has developed rapidly, especially in emergency management and single-person single-post operations. In emergency situations, the safety issues of single-person single-post operations are particularly prominent, and effective monitoring methods are required. Emergency monitoring robots can monitor and warn operators in real time.

[0003] The emergency monitoring robot monitors the status of the operator in real time through the camera. However, the camera has serious distortion problems when monitoring the status of the operator, so the collected image will be distorted, which affects the detection accuracy. Therefore, an emergency monitoring robot combined with Raspberry Pi for distortion removal and monitoring is extremely important.

[0004] Patent document CN111127514B discloses a method and device for tracking a target by a robot. The above patent realizes the robot's uninterrupted real-time tracking of the target, ensures the robot's service efficiency, service quality and activity effect, and has high tracking efficiency and accuracy. However, the above patent cannot realize the recognition function of the target object in the image.

[0005] Patent document CN114399529B discloses a target selection model and a robot interaction system for robot interaction. The above patent realizes the screening of specific groups of people that meet the robot service scenarios, but the above patent cannot realize the real-time tracking function of the operators.

[0006] Patent document CN113052907B discloses a method for positioning a mobile robot in a dynamic environment. The above patent realizes accurate positioning of the mobile robot in a dynamic environment and improves the accuracy of identifying moving objects, but the above patent cannot realize the function of detecting the fall of operators.

[0007] Patent document CN107992881B discloses a robot dynamic grasping method and system. The above patent realizes high frame rate real-time recognition, improves the running speed of the visual motion target tracking algorithm, and improves the accuracy of visual servoing, but the above patent cannot realize the status detection function of the operator.

[0008] In summary, the above patents cannot realize the recognition function of the target object in the image, cannot realize the real-time tracking function of the operator, cannot realize the fall detection function of the operator, and cannot realize the state detection function of the operator, resulting in the problem that the emergency monitoring robot cannot recognize the operator or obstacles in the image, cannot monitor the operator's work in real time, cannot detect the operator's fall behavior in time, and cannot detect the fatigue state of the operator; To this end, the present application proposes an emergency monitoring robot that realizes the recognition function of target objects in images, the real-time tracking function of operators, the fall detection function of operators, and the status detection function of operators combined with Raspberry Pi distortion removal monitoring. Summary of the invention

[0009] The purpose of the present invention is to provide an emergency monitoring robot combined with Raspberry Pi distortion removal monitoring to solve the technical problems proposed in the above background technology that the target object recognition function in the image cannot be realized, the real-time tracking function of the operator cannot be realized, the fall detection function of the operator cannot be realized, and the status detection function of the operator cannot be realized, resulting in the emergency monitoring robot being unable to recognize the operator or obstacles in the image, unable to monitor the operator's work in real time, unable to detect the operator's fall behavior in time, and unable to detect the operator's fatigue state.

[0010] To achieve the above-mentioned object, the present invention provides the following technical solutions: an emergency monitoring robot combined with Raspberry Pi dedistortion monitoring, comprising a visual recognition module, wherein the visual recognition module image is used to dedistort the image collected by the emergency monitoring robot and identify the target object in the image; The visual recognition module includes: an image acquisition unit, a dedistortion processing unit and a target recognition unit, wherein the image acquisition unit is connected to the dedistortion processing unit via a network signal, and the dedistortion processing unit is connected to the target recognition unit via a network signal; The image acquisition unit captures image information in real time through a wide-angle camera carried by the emergency monitoring robot, transmits the image data from the wide-angle camera to the Raspberry Pi through wireless transmission, and stores the image data in the memory of the Raspberry Pi; The dedistortion processing unit uses the OpenCV library in the Raspberry Pi to extract the image data in the memory, runs the dedistortion algorithm, performs dedistortion processing on each frame of the image, and corrects the image by calculating the correspondence between the distorted points and the undistorted points; The target recognition unit uses an image processing algorithm to extract feature information from the dedistorted image, analyzes and compares the extracted features, recognizes the target object in the image, determines the category of the target object, and recognizes the operator and obstacles in the image.

[0011] Preferably, the target recognition unit is connected to a target tracking module via a network signal, and the target tracking module is used to continuously track the operator and obtain the actual distance between the emergency monitoring robot and the operator in real time; The target tracking module includes: a real-time tracking unit, a distance measurement unit and a real-time adjustment unit, the real-time tracking unit is connected to the distance measurement unit through a network signal, the distance measurement unit is connected to the real-time adjustment unit through a network signal, and the real-time tracking unit is connected to the target recognition unit through a network signal; The real-time tracking unit uses a tracking algorithm to continuously track the operator in the image, analyzes the movement trajectory of the operator between consecutive image frames, determines the position of the operator in the image, and uses the image forming method to calculate the actual width of the operator; The distance measurement unit uses the camera distance measurement method to calculate the distance between the emergency monitoring robot and the operator, and sets the expected distance between the emergency monitoring robot and the operator in the safety monitoring mode and the service mode; The real-time adjustment unit calculates the offset between the operator and the center of the camera's field of view according to the operator's position in the image, calculates the deviation distance according to the actual distance and expected distance between the emergency monitoring robot and the operator, and uses a PID controller to move the emergency monitoring robot in real time.

[0012] Preferably, the target recognition unit is connected to a posture recognition module via a network signal, and the fall detection module is used to recognize the posture movement of the operator and determine the user's fall behavior; The posture recognition module includes: a key point detection unit, a key point association unit, and a fall detection unit. The key point detection unit is connected to the key point association unit via a network signal. The key point association unit is connected to the fall detection unit via a network signal. The key point detection unit is connected to the target recognition unit via a network signal. The key point detection unit uses the OpenPose algorithm to detect key points of the image, extract the key points of the operator in the image, including the joints of the human body, and accurately locate the position of each key point in the image; The key point association unit matches the detected key points with the human body structure model, compares the position information of the key points and the human body structure characteristics, matches and associates the key points of the same human body, and constructs the posture information of the operator; The fall detection unit analyzes the posture information, extracts the characteristic parameters of the posture, calculates the descent speed of the Hip joint in continuous image frames, compares the descent speed of the Hip joint with a preset threshold, and determines the fall event of the operator.

[0013] Preferably, the target recognition unit is connected to a facial recognition module via a network signal, and the facial recognition module is used for face recognition and status detection of the operator; The facial recognition module includes: a face recognition unit, a state detection unit, and a state reminder unit. The face recognition unit is connected to the state detection unit via a network signal, the state detection unit is connected to the state reminder unit via a network signal, and the face recognition unit is connected to the target recognition unit via a network signal. The face recognition unit locates the facial feature points of the operator in the image, extracts the geometric features and texture features of the face, compares the extracted facial features with the face database in the Raspberry Pi, and identifies the identity of the operator; The state detection unit calculates the eye aspect ratio, mouth aspect ratio and head posture Euler angle offset based on facial feature points to determine the fatigue state of the operator; When the status reminder unit detects that the operator is fatigued, it will send out a warning signal through a buzzer to remind the operator to rest.

[0014] Preferably, the real-time adjustment unit is connected to a route planning module via a network signal, and the route planning module is used to detect obstacles in the image and plan the moving route of the emergency monitoring robot; The route planning module includes: an obstacle recognition unit, a strategy formulation unit and an obstacle avoidance evaluation unit, wherein the obstacle recognition unit is connected to the obstacle avoidance execution unit via a network signal, the obstacle avoidance execution unit is connected to the obstacle avoidance evaluation unit via a network signal, the obstacle recognition unit is connected to the target recognition unit via a network signal, and the strategy formulation unit is connected to the real-time adjustment unit via a network signal; The obstacle recognition unit uses the image forming method to calculate the real size of the obstacle in the image, sets the size threshold of the obstacle, and marks the obstacles exceeding the threshold as potential obstacles that affect the movement of the emergency monitoring robot; The strategy formulation unit uses a path planning algorithm to calculate a collision-free path from the current position to the target position, optimizes the calculated path, and sets up emergency obstacle avoidance mechanisms, including emergency stop, quick turn, and back off; The obstacle avoidance evaluation unit records the data of the emergency monitoring robot during the obstacle avoidance process, including the location and size information of the obstacle, as well as the motion trajectory and obstacle avoidance measures of the emergency monitoring robot.

[0015] Preferably, the fall detection unit is connected to a dynamic interaction module via a network signal, and the dynamic interaction module is used for interaction between the emergency monitoring robot and the operator; The dynamic interaction module includes: an emergency response unit, a voice call unit and an interactive feedback unit, the emergency response unit is connected to the interactive feedback unit via a network signal, the voice call unit is connected to the interactive feedback unit via a network signal, and the emergency response unit is connected to the fall detection unit via a network signal; When the emergency monitoring robot is in the safety monitoring mode, the emergency response unit detects that the operator is in a falling state, triggers the alarm mechanism, sends an alarm message to the safety person in charge by phone, and issues an audible and visual alarm; When the emergency monitoring robot is in service mode, the voice call unit recognizes the word "tool" in the operator's voice command, synthesizes and replies with the voice of "tools are being transported", and controls the emergency monitoring robot to approach the operator through the PID controller; The interactive feedback unit records the operator's voice commands, the emergency monitoring robot's responses and alarm information, and provides a user interactive interface for switching the emergency monitoring robot's safety monitoring mode, service mode, and querying historical information.

[0016] Preferably, the real-time adjustment unit is connected to a route planning module via a network signal, and the route planning module is used to detect obstacles in the image and plan the moving route of the emergency monitoring robot; The route planning module includes: an obstacle recognition unit, a strategy formulation unit and an obstacle avoidance evaluation unit, wherein the obstacle recognition unit is connected to the obstacle avoidance execution unit via a network signal, the obstacle avoidance execution unit is connected to the obstacle avoidance evaluation unit via a network signal, the obstacle recognition unit is connected to the target recognition unit via a network signal, and the strategy formulation unit is connected to the real-time adjustment unit via a network signal; The obstacle recognition unit uses the image forming method to calculate the real size of the obstacle in the image, sets the size threshold of the obstacle, and marks the obstacles exceeding the threshold as potential obstacles that affect the movement of the emergency monitoring robot; The strategy formulation unit uses a path planning algorithm to calculate a collision-free path from the current position to the target position, optimizes the calculated path, and sets up emergency obstacle avoidance mechanisms, including emergency stop, quick turn, and back off; The obstacle avoidance evaluation unit records the data of the emergency monitoring robot during the obstacle avoidance process, including the location and size information of the obstacle, as well as the motion trajectory and obstacle avoidance measures of the emergency monitoring robot.

[0017] Preferably, the de-distortion algorithm comprises the following steps: Step 1: Camera calibration: Take images of the calibration plate at different angles, process the images using the calibration algorithm, and calculate the camera's intrinsic parameter matrix and distortion coefficient; Step 2: Distortion model establishment: According to the calibration results, the distortion model of the camera is established, including radial distortion and tangential distortion; Step 3, distortion correction parameter calculation: Calculate the mapping relationship required for distortion correction according to the calibration results and generate two mapping tables; Step 4: Image dedistortion: Use the remap function in combination with the mapping table calculated in step 3 to perform dedistortion. The remap function maps each pixel in the distorted image to a new position after dedistortion according to the mapping table to generate an undistorted image.

[0018] Preferably, the image forming method comprises the following steps: Establishing a geometric model: Establishing a geometric model of camera imaging based on the camera’s intrinsic parameter matrix and distortion parameters; Calculate the ratio: select an object with a known real width in the image, measure the pixel width of the object in the image, and calculate the ratio based on the real width and pixel width of the object; Calculate the real size: Use the geometric model and proportional relationship to convert the operator's feature points in the image coordinates into world coordinates, and calculate the operator's real width based on the converted world coordinates.

[0019] Preferably, the image processing algorithm includes: edge detection, edge detection, texture feature extraction and shape feature extraction Preferably, the formula of the camera ranging method is: D = (W*F) / P; Where D is the distance between the camera and the operator, W is the actual width of the operator, and P is the pixel width of the operator in the image.

[0020] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention realizes the recognition function of the target object in the image by designing a visual recognition module, solves the image distortion problem caused by camera distortion, improves the usability and accuracy of the image, and improves the recognition accuracy; 2. The present invention realizes the real-time tracking function of the operator by designing a target tracking module, ensuring that the operator is in the center of the camera's field of view and maintains a desired distance from the robot, thereby improving the accuracy of monitoring; 3. The present invention realizes the fall detection function of the operator by designing a posture recognition module, solves the problem that it is difficult to detect the operator after falling, and improves the emergency response speed and the safety of the operator; 4. The present invention realizes the function of detecting the status of the operating personnel by designing a facial recognition module, realizes accurate recognition of the operating personnel's face, improves the accuracy and efficiency of monitoring, and realizes comprehensive and accurate monitoring of the operating personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1It is a schematic diagram of a visual recognition module of the present invention; Figure 2 It is a schematic diagram of a target tracking module of the present invention; Figure 3 is a schematic diagram of a gesture recognition module of the present invention; Figure 4 is a schematic diagram of a facial recognition module of the present invention; Figure 5 This is a schematic diagram of a route planning module of the present invention; Figure 6 is a schematic diagram of a dynamic interaction module of the present invention; Figure 7 A schematic diagram of the use process of the distortion algorithm of the present invention; Figure 8 It is a schematic diagram of the flow of the emergency monitoring robot system of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Example 1, please refer to Figure 1 , Figure 2 , Figure 7 and Figure 8 , an emergency monitoring robot combined with Raspberry Pi dedistortion monitoring, including a visual recognition module, the visual recognition module image is used to perform dedistortion processing on the image collected by the emergency monitoring robot and identify the target object in the image; the visual recognition module includes: an image acquisition unit, a dedistortion processing unit and a target recognition unit, the image acquisition unit is connected to the dedistortion processing unit through a network signal, and the dedistortion processing unit is connected to the target recognition unit through a network signal; the image acquisition unit captures image information in real time through a wide-angle camera carried by the emergency monitoring robot, transmits the image data from the wide-angle camera to the Raspberry Pi through wireless transmission, and stores the image data in the memory of the Raspberry Pi; the dedistortion processing unit uses the OpenCV library in the Raspberry Pi to extract the image data in the memory, runs the dedistortion algorithm, performs dedistortion processing on each frame of the image, and corrects the image by calculating the corresponding relationship between the distortion point and the non-distortion point; the target recognition unit uses the image processing algorithm to extract feature information in the dedistorted image, analyzes and compares the extracted features, identifies the target object in the image, determines the category of the target object, and identifies the operator and obstacles in the image; The real-time tracking unit uses a tracking algorithm to continuously track the workers in the image, analyzes the movement trajectory of the workers between consecutive image frames, determines the position of the workers in the image, and uses the image forming method to calculate the actual width of the workers; the distance measurement unit uses the camera ranging method to calculate the distance between the emergency monitoring robot and the workers, and sets the expected distance between the emergency monitoring robot and the workers in the safety monitoring mode and service mode; the real-time adjustment unit calculates the offset between the workers and the center of the camera's field of view according to the position of the workers in the image, calculates the deviation distance according to the actual distance and expected distance between the emergency monitoring robot and the workers, and uses a PID controller to move the emergency monitoring robot in real time.

[0024] Furthermore, the image acquisition unit in the visual recognition module uses a wide-angle camera to capture image information of the surrounding environment in real time. The wide-angle camera provides a wider field of view, which is convenient for the robot to perform all-round monitoring in complex environments. Wireless transmission technology Wi-Fi is used to realize data transmission between the camera and the Raspberry Pi. The image acquisition unit stores the image data in the memory of the Raspberry Pi, where the Raspberry Pi has powerful data processing and storage capabilities, ensuring the integrity and availability of the image data; the wide-angle camera will produce distortion when capturing images, resulting in a large gap between the read image and the actual image, thereby affecting the subsequent feature information extraction and reducing the accuracy of state detection and fall detection. Accuracy: The de-distortion processing unit uses Raspberry Pi OpenCV to read the image data in the Raspberry Pi memory and perform de-distortion processing to correct the image, thereby improving the accuracy and clarity of the image; the target recognition unit receives the image processed by the de-distortion processing unit, and extracts the feature information in the de-distorted image through the image processing algorithm in the Raspberry Pi, including edge detection, corner point detection, texture feature extraction, and shape feature extraction, and compares and analyzes the extracted features with the feature library in the Raspberry Pi to determine whether there is an object in the image that matches the target object in the feature library, analyze the category of the target object in the image, and determine whether the target object is an operator or an obstacle; When the target recognition unit in the visual recognition module detects the presence of a worker in the image, the target recognition unit transmits the dedistorted image and the target recognition result of the image to the target tracking module. The real-time tracking unit in the target tracking module tracks the worker identified in the image using a tracking algorithm based on the target recognition result, and continuously locks the position of the worker. When a change in the position of the worker in the continuous image frames is detected, the target recognition unit triggers the distance measurement unit to perform distance calculation. At the same time, the target recognition unit further processes the feature information of the worker in the image through the Raspberry Pi, extracts the pixel width of the worker through the contour feature information of the worker in the image, and then uses the image forming method to calculate the true width of the worker in the image, and transmits the pixel width and true width data of the worker to the distance measurement unit; the distance measurement unit receives the data transmitted by the target recognition unit, uses the camera ranging method, and calculates the distance between the worker and the camera in the image according to the principle of similar triangles. The distance between the cameras is the distance between the operator and the robot. At the same time, the distance measurement unit sets the expected distance range of the safety monitoring mode and the service mode according to the needs of the application scenario, so as to ensure the safe distance between the emergency monitoring robot and the operator; the real-time tracking unit transmits the position change and prediction data to the real-time adjustment unit. The real-time adjustment unit calculates the offset between the operator and the center of the camera's field of view according to the position of the operator in the image. The real-time adjustment unit preliminarily determines the movement direction and distance of the robot according to the offset. After the distance measurement unit transmits the actual distance and the expected distance between the operator and the robot to the real-time adjustment unit, the real-time adjustment unit determines the deviation distance according to the difference between the actual distance and the expected distance, so as to further determine the movement direction and distance of the robot. The real-time adjustment unit adjusts the position of the robot in real time through the PID controller, so as to ensure that the operator is in the center of the camera's field of view and the distance between the robot and the operator is the expected distance.

[0025] Example 2, please refer to Figure 1 , Figure 3 and Figure 8, an emergency monitoring robot combined with Raspberry Pi distortion removal monitoring, the target recognition unit uses an image processing algorithm to extract feature information from the dedistorted image, analyzes and compares the extracted features, recognizes the target object in the image, determines the category of the target object, and recognizes the operator and obstacles in the image; the target recognition unit is connected to a posture recognition module via a network signal, and the fall detection module is used to recognize the posture movements of the operator and judge the user's fall behavior; the posture recognition module includes: a key point detection unit, a key point association unit, and a fall detection unit, the key point detection unit is connected to the key point association unit via a network signal, the key point association unit is connected to the fall detection unit via a network signal, and the key point The detection unit is connected to the target recognition unit through a network signal; the key point detection unit uses the OpenPose algorithm to detect key points of the image, extracts the key points of the operator in the image, including the joints of the human body, and accurately locates the position of each key point in the image; the key point association unit matches the detected key points with the human body structure model, compares the position information of the key points and the human body structure characteristics, matches and associates the key points of the same human body, and constructs the posture information of the operator; the fall detection unit analyzes the posture information, extracts the characteristic parameters of the posture, calculates the descent speed of the Hip joint in continuous image frames, compares the descent speed of the Hip joint with the preset threshold, and determines the fall event of the operator.

[0026] Furthermore, when the target recognition unit detects that a worker appears in the image, the target recognition unit transmits the dedistorted image and the target recognition result of the image to the posture recognition module. The key point detection unit in the posture recognition module uses the OpenPose algorithm to detect key points of the image based on the target recognition result. The OpenPose algorithm extracts features of the worker in the image through a deep convolutional neural network, extracts key points such as joints of the human body, such as shoulders, elbows, wrists, knees, ankles, etc., and accurately locates the position of each key point in the image, thereby providing basic data for subsequent analysis and processing; after the key point association unit receives the key point data transmitted by the key point detection unit, the key point association unit The detected key points are associated with the human body structure model, and by comparing the position information of the key points and the human body structure characteristics, the key points of the same human body are matched and associated, thereby constructing complete posture information to avoid false detection and missed detection; the fall detection unit analyzes the posture information, analyzes the position, angle and relationship of various parts of the operator's body, and extracts characteristic parameters of the posture, such as the descent speed of the Hip joint. The fall detection unit will calculate the descent speed of the Hip joint in continuous image frames, and compare the descent speed of the Hip joint with a preset threshold. If the descent speed exceeds the threshold, it is judged that the operator has a fall event, thereby promptly discovering the abnormal situation of the operator and providing an important basis for emergency response.

[0027] Example 3, please refer to Figure 1 , Figure 4 and Figure 8 , an emergency monitoring robot combined with Raspberry Pi distortion removal monitoring, the target recognition unit uses an image processing algorithm to extract feature information from the dedistorted image, analyzes and compares the extracted features, recognizes the target object in the image, determines the category of the target object, and recognizes the operator and obstacles in the image; the key point association unit matches the detected key points with the human body structure model, compares the position information of the key points and the human body structure features, matches and associates the key points of the same human body, and constructs the posture information of the operator; the target recognition unit is connected to a facial recognition module through a network signal, and the facial recognition module is used for face recognition and state detection of the operator; the facial recognition module includes: a facial recognition unit , status detection unit, status reminder unit, the face recognition unit is connected to the status detection unit through network signals, the status detection unit is connected to the status reminder unit through network signals, and the face recognition unit is connected to the target recognition unit through network signals; the face recognition unit locates the facial feature points of the operator in the image, extracts the geometric features and texture features of the face, compares the extracted facial features with the face database in the Raspberry Pi, and identifies the identity of the operator; the status detection unit calculates the eye aspect ratio, mouth aspect ratio and head posture Euler angle offset according to the facial feature points to determine the fatigue state of the operator; when the status reminder unit detects that the operator is tired, it sends a warning signal through a buzzer to remind the operator to take a rest.

[0028] Furthermore, when the target recognition unit detects the presence of a worker in the image, the target recognition unit transmits the dedistorted image and the target recognition result of the image to the facial recognition module. The face recognition unit in the posture recognition module quickly locates the head of the worker in the image based on the worker posture information transmitted by the key point association unit, thereby facilitating the face recognition unit to locate the facial feature points of the worker in the image, and extract the geometric features and texture features of the face, analyze the positional relationship of the worker's eyes, nose, and mouth based on the geometric features, and analyze the worker's skin texture, spots, etc. based on the texture features. Then, the face recognition unit compares the extracted features with the facial features in the Raspberry Pi database to determine the identity of the worker. The face recognition unit determines the worker After the identity of the operator is determined, the state detection unit is notified to perform the state detection of the operator. The state detection unit receives the facial features extracted by the face recognition unit, and calculates parameters such as the eye aspect ratio, mouth aspect ratio, and head posture Euler angle offset according to the facial feature points. When the operator is tired, such as the operator's eyes are squinting, closed, or the head is drooping or tilted, the parameters such as the eye aspect ratio or the head posture Euler angle offset will change. The state detection unit compares the facial feature parameters of the image with the threshold. When the facial feature parameters of multiple consecutive frames of images are higher or lower than the threshold, the state detection unit determines that the operator is in a fatigued state; the state detection unit transmits the fatigue signal to the state reminder unit. After receiving the signal, the state reminder unit controls the buzzer on the robot to send out a warning signal to remind the operator to rest.

[0029] Example 4, please refer to Figure 1 , Figure 2 , Figure 5 and Figure 8, an emergency monitoring robot combined with Raspberry Pi dedistortion monitoring, wherein the target recognition unit uses an image processing algorithm to extract feature information from a dedistorted image, analyzes and compares the extracted features, recognizes the target object in the image, determines the category of the target object, and recognizes the operator and obstacles in the image; the real-time adjustment unit calculates the offset between the operator and the center of the camera field of view according to the position of the operator in the image, calculates the deviation distance according to the actual distance and expected distance between the emergency monitoring robot and the operator, and uses a PID controller to move the emergency monitoring robot in real time; the real-time adjustment unit is connected to a route planning module through a network signal, and the route planning module is used to detect obstacles in the image and plan the moving route of the emergency monitoring robot; the route planning module includes: an obstacle recognition unit, a strategy formulation unit, and an obstacle avoidance evaluation unit The obstacle recognition unit is connected to the obstacle avoidance execution unit through network signals, the obstacle avoidance execution unit is connected to the obstacle avoidance evaluation unit through network signals, the obstacle recognition unit is connected to the target recognition unit through network signals, and the strategy formulation unit is connected to the real-time adjustment unit through network signals; the obstacle recognition unit uses an image forming method to calculate the true size of the obstacle in the image, sets the size threshold of the obstacle, and marks the obstacles exceeding the threshold as potential obstacles that affect the movement of the emergency monitoring robot; the strategy formulation unit uses a path planning algorithm to calculate a collision-free path from the current position to the target position, optimizes the calculated path, and sets an emergency obstacle avoidance mechanism, including emergency stop, quick turn and retreat; the obstacle avoidance evaluation unit records the data of the emergency monitoring robot during the obstacle avoidance process, including the position and size information of the obstacle, as well as the motion trajectory and obstacle avoidance measures of the emergency monitoring robot.

[0030] Furthermore, in order to ensure that the operator is in the center of the camera's field of view and maintains a desired distance from the robot, the real-time adjustment unit moves the emergency monitoring robot in real time. During the movement, when the target recognition unit recognizes that an obstacle appears in the image, the target recognition unit transmits the dedistorted image and the target recognition result of the image to the route planning module. After receiving the image, the obstacle recognition unit in the route planning module uses the image forming method to calculate the true size of the obstacle in the image, and compares the true size of the obstacle with the threshold data. When the true size of the obstacle exceeds the threshold, it means that the obstacle will affect the movement of the emergency monitoring robot and the emergency monitoring robot needs to avoid it. At this time, the obstacle recognition unit will mark the obstacle exceeding the threshold as a potential obstacle and mark it in the image; the strategy formulation unit receives the marked image, and the strategy formulation unit calculates the current obstacle based on the image. A collision-free path from the previous position to the target position is calculated and optimized according to the actual situation, and emergency obstacle avoidance mechanisms are set, such as emergency stop, quick turn and retreat, so that the robot can respond quickly to emergencies and avoid collisions. The strategy formulation unit transmits the calculated and optimized route to the real-time adjustment unit, and the real-time adjustment unit moves the emergency monitoring robot according to the route; in the obstacle avoidance process, the obstacle avoidance evaluation unit records the position and size information of the obstacle, as well as the robot's motion trajectory and obstacle avoidance measures in real time, and analyzes and evaluates the recorded data, including judging whether the robot has successfully avoided all obstacles and whether it has moved according to the planned path, so as to judge the rationality of the path planning, the effectiveness of the obstacle avoidance measures, etc. The obstacle avoidance evaluation unit optimizes the path planning algorithm of the obstacle module according to the evaluation data, thereby improving the accuracy of the subsequent route planning of the strategy formulation unit.

[0031] Example 5, please refer to Figure 3 , Figure 6 and Figure 8, an emergency monitoring robot combined with Raspberry Pi distortion removal monitoring, a fall detection unit analyzes posture information, extracts characteristic parameters of the posture, calculates the descent speed of the Hip joint in continuous image frames, compares the descent speed of the Hip joint with a preset threshold, and determines the fall event of the operator; the fall detection unit is connected to a dynamic interaction module through a network signal, and the dynamic interaction module is used for the interaction between the emergency monitoring robot and the operator; the dynamic interaction module includes: an emergency response unit, a voice call unit and an interactive feedback unit, the emergency response unit is connected to the interactive feedback unit through a network signal, the voice call unit is connected to the interactive feedback unit through a network signal, and the emergency response unit is connected to the interactive feedback unit through a network signal It is connected with a fall detection unit; when the emergency monitoring robot is in the safety monitoring mode, the emergency response unit detects that the operator is in a fall state, triggers the alarm mechanism, uses the phone to send an alarm message to the safety person in charge, and issues an audible and visual alarm; when the emergency monitoring robot is in the service mode, the voice call unit recognizes that "tools" appear in the operator's voice command, synthesizes and replies with the voice of "tools being transported", and controls the emergency monitoring robot to approach the operator through the PID controller; the interactive feedback unit records the operator's voice commands, the emergency monitoring robot's response and alarm information, and provides a user interactive interface for switching the emergency monitoring robot's safety monitoring mode, service mode, and querying historical information.

[0032] Furthermore, after the operator starts the emergency monitoring robot, the operating mode of the emergency monitoring robot is switched through the user interaction interface in the interactive feedback unit. In the safety monitoring mode, after the fall detection unit detects that the operator is in a fall state, the fall detection unit transmits the information to the emergency response unit. After receiving the information, the emergency response unit immediately triggers the alarm mechanism, sends an alarm message to the safety person in charge by phone, and starts the sound and light alarm on the robot to ensure that the alarm information can be quickly conveyed to the relevant personnel and help the relevant personnel to quickly locate, thereby improving the emergency response speed; in the service mode, the voice call unit uses a voice recognition algorithm to convert the operator's voice command into text information, and searches for the specific keyword "tool" in the converted text information. When "tool" is searched, the voice call unit synthesizes and replies with the voice of "transporting tools", and controls the moving trajectory and speed of the emergency monitoring robot through the PID controller to make the emergency monitoring robot close to the operator; during the operation of the emergency monitoring robot, the interactive feedback unit records the operator's voice command, the robot's response and alarm information and other data in real time, and classifies and manages them so that the operator can query through the user interaction interface.

[0033] Example 6, please refer to Figure 2 , Figure 6 and Figure 8An emergency monitoring robot combined with Raspberry Pi distortion removal monitoring, the distance measurement unit uses the camera ranging method to calculate the distance between the emergency monitoring robot and the operator, and sets the expected distance between the emergency monitoring robot and the operator in the safety monitoring mode and the service mode; the real-time adjustment unit calculates the offset between the operator and the center of the camera field of view according to the position of the operator, calculates the deviation distance according to the actual distance and the expected distance between the emergency monitoring robot and the operator, and uses a PID controller to move the emergency monitoring robot in real time; the interactive feedback unit records the voice commands of the operator, the response and alarm information of the emergency monitoring robot, and provides a user interactive interface for switching the safety monitoring mode, service mode and querying historical information of the emergency monitoring robot.

[0034] Furthermore, after the operator starts the emergency monitoring robot, the operating mode of the emergency monitoring robot is switched through the user interaction interface in the interactive feedback unit. After the operator determines the working mode, the interactive feedback unit transmits the working mode of the emergency monitoring robot to the distance measurement unit. The distance measurement unit calculates the actual distance between the emergency monitoring robot and the operator in real time. The distance measurement unit transmits the expected distance under the working mode of the emergency monitoring robot to the real-time adjustment unit. At the same time, the distance measurement unit transmits the actual distance between the emergency monitoring robot and the operator to the real-time adjustment unit. The real-time adjustment unit moves the emergency monitoring robot in real time through the PID controller according to the difference between the actual distance and the set expected distance, thereby ensuring that the actual distance between the emergency monitoring robot and the operator is the expected distance.

[0035] Working principle: the operator starts the emergency monitoring robot and switches the working mode of the emergency monitoring robot through the user interaction interface in the dynamic interaction module. The visual recognition module captures the surrounding environment image in real time. The visual recognition module dedistorts the image and identifies the operator and obstacles in the image. The facial recognition module identifies the face of the operator in the image and confirms the identity of the operator. After confirming the identity of the operator, the target tracking module continuously tracks the operator in the image. According to the working mode information of the emergency monitoring robot transmitted by the dynamic interaction module, the target tracking module calculates the deviation distance under the corresponding working mode. The target tracking module moves the emergency monitoring robot in real time according to the deviation distance and the route planned by the route planning module. The posture recognition module extracts the key points of the operator in the image and constructs the posture information of the operator. The posture recognition module compares the characteristic parameters of the posture with the threshold. At the same time, the facial recognition module compares the facial characteristic parameters with the threshold. When the characteristic parameters of the posture exceed the threshold, the posture recognition module sends an alarm to the safety officer through the dynamic interaction module. When the facial characteristic parameters exceed the threshold, the facial recognition module reminds the operator.

[0036] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. An emergency monitoring robot combined with Raspberry Pi distortion removal monitoring, characterized by: It includes a visual recognition module, wherein the visual recognition module image is used to perform dedistortion processing on the image collected by the emergency monitoring robot and identify the target object in the image; The visual recognition module includes: an image acquisition unit, a dedistortion processing unit and a target recognition unit, wherein the image acquisition unit is connected to the dedistortion processing unit via a network signal, and the dedistortion processing unit is connected to the target recognition unit via a network signal; The image acquisition unit captures image information in real time through a wide-angle camera carried by the emergency monitoring robot, transmits the image data from the wide-angle camera to the Raspberry Pi through wireless transmission, and stores the image data in the memory of the Raspberry Pi; The dedistortion processing unit uses the OpenCV library in the Raspberry Pi to extract the image data in the memory, runs the dedistortion algorithm, performs dedistortion processing on each frame of the image, and corrects the image by calculating the correspondence between the distorted points and the undistorted points; The target recognition unit uses an image processing algorithm to extract feature information from the dedistorted image, analyzes and compares the extracted features, recognizes the target object in the image, determines the category of the target object, and recognizes the operator and obstacles in the image.

2. The emergency monitoring robot combined with Raspberry Pi de-distortion monitoring according to claim 1, characterized in that: The target recognition unit is connected to a target tracking module via a network signal, and the target tracking module is used to continuously track the operator and obtain the actual distance between the emergency monitoring robot and the operator in real time; The target tracking module includes: a real-time tracking unit, a distance measurement unit and a real-time adjustment unit, the real-time tracking unit is connected to the distance measurement unit through a network signal, the distance measurement unit is connected to the real-time adjustment unit through a network signal, and the real-time tracking unit is connected to the target recognition unit through a network signal; The real-time tracking unit uses a tracking algorithm to continuously track the operator in the image, analyzes the movement trajectory of the operator between consecutive image frames, determines the position of the operator in the image, and uses the image forming method to calculate the actual width of the operator; The distance measurement unit uses the camera distance measurement method to calculate the distance between the emergency monitoring robot and the operator, and sets the expected distance between the emergency monitoring robot and the operator in the safety monitoring mode and the service mode; The real-time adjustment unit calculates the offset between the operator and the center of the camera's field of view according to the operator's position in the image, calculates the deviation distance according to the actual distance and expected distance between the emergency monitoring robot and the operator, and uses a PID controller to move the emergency monitoring robot in real time.

3. The emergency monitoring robot combined with Raspberry Pi de-distortion monitoring according to claim 1, characterized in that: The target recognition unit is connected to a posture recognition module via a network signal, and the fall detection module is used to recognize the posture movements of the operator and determine the user's fall behavior; The posture recognition module includes: a key point detection unit, a key point association unit, and a fall detection unit. The key point detection unit is connected to the key point association unit via a network signal. The key point association unit is connected to the fall detection unit via a network signal. The key point detection unit is connected to the target recognition unit via a network signal. The key point detection unit uses the OpenPose algorithm to detect key points of the image, extract the key points of the operator in the image, including the joints of the human body, and accurately locate the position of each key point in the image; The key point association unit matches the detected key points with the human body structure model, compares the position information of the key points and the human body structure characteristics, matches and associates the key points of the same human body, and constructs the posture information of the operator; The fall detection unit analyzes the posture information, extracts the characteristic parameters of the posture, calculates the descent speed of the Hip joint in continuous image frames, compares the descent speed of the Hip joint with a preset threshold, and determines the fall event of the operator.

4. The emergency monitoring robot combined with Raspberry Pi de-distortion monitoring according to claim 1, characterized in that: The target recognition unit is connected to a facial recognition module via a network signal, and the facial recognition module is used for face recognition and status detection of operators; The facial recognition module includes: a face recognition unit, a state detection unit, and a state reminder unit. The face recognition unit is connected to the state detection unit via a network signal, the state detection unit is connected to the state reminder unit via a network signal, and the face recognition unit is connected to the target recognition unit via a network signal. The face recognition unit locates the facial feature points of the operator in the image, extracts the geometric features and texture features of the face, compares the extracted facial features with the face database in the Raspberry Pi, and identifies the identity of the operator; The state detection unit calculates the eye aspect ratio, mouth aspect ratio and head posture Euler angle offset based on facial feature points to determine the fatigue state of the operator; When the status reminder unit detects that the operator is fatigued, it will send out a warning signal through a buzzer to remind the operator to rest.

5. The emergency monitoring robot combined with Raspberry Pi de-distortion monitoring according to claim 2, characterized in that: The real-time adjustment unit is connected to a route planning module via a network signal, and the route planning module is used to detect obstacles in the image and plan a moving route for the emergency monitoring robot; The route planning module includes: an obstacle recognition unit, a strategy formulation unit and an obstacle avoidance evaluation unit, wherein the obstacle recognition unit is connected to the obstacle avoidance execution unit via a network signal, the obstacle avoidance execution unit is connected to the obstacle avoidance evaluation unit via a network signal, the obstacle recognition unit is connected to the target recognition unit via a network signal, and the strategy formulation unit is connected to the real-time adjustment unit via a network signal; The obstacle recognition unit uses the image forming method to calculate the real size of the obstacle in the image, sets the size threshold of the obstacle, and marks the obstacles exceeding the threshold as potential obstacles that affect the movement of the emergency monitoring robot; The strategy formulation unit uses a path planning algorithm to calculate a collision-free path from the current position to the target position, optimizes the calculated path, and sets up emergency obstacle avoidance mechanisms, including emergency stop, quick turn, and back off; The obstacle avoidance evaluation unit records the data of the emergency monitoring robot during the obstacle avoidance process, including the location and size information of the obstacle, as well as the motion trajectory and obstacle avoidance measures of the emergency monitoring robot.

6. The emergency monitoring robot combined with Raspberry Pi de-distortion monitoring according to claim 3, characterized in that: The fall detection unit is connected to a dynamic interaction module via a network signal, and the dynamic interaction module is used for interaction between the emergency monitoring robot and the operator; The dynamic interaction module includes: an emergency response unit, a voice call unit and an interactive feedback unit, the emergency response unit is connected to the interactive feedback unit via a network signal, the voice call unit is connected to the interactive feedback unit via a network signal, and the emergency response unit is connected to the fall detection unit via a network signal; When the emergency monitoring robot is in the safety monitoring mode, the emergency response unit detects that the operator is in a falling state, triggers the alarm mechanism, sends an alarm message to the safety person in charge by phone, and issues an audible and visual alarm; When the emergency monitoring robot is in service mode, the voice call unit recognizes the word "tool" in the operator's voice command, synthesizes and replies with the voice of "transporting tools", and controls the emergency monitoring robot to approach the operator through the PID controller; The interactive feedback unit records the operator's voice commands, the emergency monitoring robot's responses and alarm information, and provides a user interactive interface for switching the emergency monitoring robot's safety monitoring mode, service mode, and querying historical information.

7. The emergency monitoring robot combined with Raspberry Pi de-distortion monitoring according to claim 1, characterized in that: The de-distortion algorithm comprises the following steps: Step 1: Camera calibration: Take images of the calibration plate at different angles, process the images using the calibration algorithm, and calculate the camera's intrinsic parameter matrix and distortion coefficient; Step 2: Distortion model establishment: According to the calibration results, the distortion model of the camera is established, including radial distortion and tangential distortion; Step 3, distortion correction parameter calculation: Calculate the mapping relationship required for distortion correction according to the calibration results and generate two mapping tables; Step 4: Image dedistortion: Use the remap function in combination with the mapping table calculated in step 3 to perform dedistortion. The remap function maps each pixel in the distorted image to a new position after dedistortion according to the mapping table to generate an undistorted image.

8. The emergency monitoring robot combined with Raspberry Pi de-distortion monitoring according to claim 2, characterized in that: The image forming method comprises the following steps: Establishing a geometric model: Establishing a geometric model of camera imaging based on the camera’s intrinsic parameter matrix and distortion parameters; Calculate the ratio: select an object with a known real width in the image, measure the pixel width of the object in the image, and calculate the ratio based on the real width and pixel width of the object; Calculate the real size: Use the geometric model and proportional relationship to convert the operator's feature points in the image coordinates into world coordinates, and calculate the operator's real width based on the converted world coordinates.

9. The emergency monitoring robot combined with Raspberry Pi de-distortion monitoring according to claim 1, characterized in that: The image processing algorithm includes: edge detection, edge detection, texture feature extraction and shape feature extraction.

10. The emergency monitoring robot combined with Raspberry Pi de-distortion monitoring according to claim 2, characterized in that: The formula of the camera ranging method is: D = (W*F) / P; Where D is the distance between the camera and the operator, W is the actual width of the operator, and P is the pixel width of the operator in the image.

Images