A mobile service robot based on computer vision

Abstract

The present application discloses a mobile service robot based on computer vision. The mobile service robot based on computer vision includes a gesture detection module, a fall detection module, a radar following module, and a news summary module. The gesture detection module is used to obtain hand images through a camera for gesture detection processing to obtain gesture detection results; the fall detection module is used to obtain human body images based on the gesture detection results for fall detection processing to obtain fall detection results; the radar following module is used to obtain laser radar data based on the gesture detection results to follow the movement of the mobile service robot based on computer vision; the news summary module is used to obtain news data based on the gesture detection results for summary processing to obtain news summary text. The embodiments of the present application can organically combine mobile robots with artificial intelligence technology and apply them to service robots for the elderly, and can be widely used in the field of robotics technology.

Description

Technical Field

[0001] The present application relates to the field of robotics, and in particular to a mobile service robot based on computer vision. Background Art

[0002] With the rapid development of science and technology, robots are increasingly being used in all aspects of daily life. Among related technologies, there are home security robots that can monitor the home environment in real time and promptly report dangerous situations. They feature low cost, high reliability, and a high level of intelligence. Mid-range animal-shaped companion robots and high-end integrated companion robots have also been designed to address the psychological and practical needs of the elderly. These designs emphasize interaction between the robot and the elderly and provide companionship. However, in actual applications, service robots specifically designed for the elderly are relatively rare. As a result, most service robots are immobile butler robots. Mobile service robots are generally large, lack interaction logic tailored to the elderly, are expensive, and rarely utilize artificial intelligence technology, which affects their applicability and functionality.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to propose a mobile service robot based on computer vision, which can improve the applicability and functionality of the robot in providing services to the elderly.

[0005] To achieve the above objectives, one aspect of an embodiment of the present application provides a mobile service robot based on computer vision, wherein the mobile service robot based on computer vision includes a gesture detection module, a fall detection module, a radar following module, and a news summary module;

[0006] The gesture detection module is used to obtain a hand image through a camera to perform gesture detection processing and obtain a gesture detection result;

[0007] The fall detection module is configured to obtain a human body image based on the gesture detection result, perform fall detection processing, and obtain a fall detection result;

[0008] The radar following module is used to obtain laser radar data according to the gesture detection result to follow the computer vision-based movable service robot;

[0009] The news summary module is used to obtain news data according to the gesture detection result, perform summary processing, and obtain a news summary text.

[0010] In some embodiments, the gesture detection module includes a hand image acquisition unit, a hand key point extraction unit, and a gesture recognition unit;

[0011] The hand image acquisition unit is used to acquire a hand image through a camera;

[0012] The hand key point extraction unit is used to perform key point extraction processing on the hand image to obtain hand key points;

[0013] The gesture recognition unit is used to perform gesture recognition processing on the hand image according to the hand key points to obtain a gesture recognition result.

[0014] In some embodiments, the gesture recognition unit includes a coordinate calculation subunit, a distance calculation subunit, an extension state determination subunit, and a gesture matching subunit;

[0015] The coordinate calculation subunit is used to determine the key point coordinate data of the fingers according to the key points of the hand;

[0016] The distance calculation subunit is used to perform distance calculation processing on the key point coordinate data to obtain the coordinate distance;

[0017] The extension state determining subunit is configured to determine the extension state of the finger according to the coordinate distance;

[0018] The gesture matching subunit is used to perform gesture matching on the extended state of the finger to obtain a gesture recognition result.

[0019] In some embodiments, the fall detection module includes a human image acquisition unit, a human key point extraction unit, a fall judgment unit, and an alarm unit;

[0020] The human body image acquisition unit is used to acquire a human body image through a camera;

[0021] The human body key point extraction unit is used to perform key point extraction processing on the human body image to obtain human body key points;

[0022] The fall judgment unit is used to perform fall judgment processing on the key points of the human body according to the preset fall conditions to obtain a fall judgment result;

[0023] The alarm unit is used to issue a buzzer alarm according to the fall judgment result.

[0024] In some embodiments, the human body key point extraction unit includes an image conversion subunit, a posture estimation subunit, and a post-processing subunit;

[0025] The image conversion subunit is used to perform color space conversion processing on the human body image to obtain a converted image;

[0026] The posture estimation subunit is used to perform key point position prediction processing on the human body image through a key point prediction model to obtain posture key points;

[0027] The post-processing subunit is used to perform outlier removal and smoothing processing on the posture key points to obtain human body key points.

[0028] In some embodiments, the radar following module includes a lidar data subscription unit, a node activation unit, and a motion control unit;

[0029] The laser radar data subscription unit is used to subscribe to obtain the laser radar data;

[0030] The node starting unit is used to start the follower node according to the laser radar data;

[0031] The motion control unit is used to control the computer vision-based movable service robot to follow the movement by issuing motion instructions through the follower node.

[0032] In some embodiments, the news summarization module includes a news data acquisition unit and a text processing unit;

[0033] The news data acquisition unit is used to acquire news data;

[0034] The text processing unit is used to perform summary generation processing on the news data through a generative language model to obtain a news summary text.

[0035] In some embodiments, the news data acquisition unit includes a data crawling subunit and an image recognition subunit;

[0036] The data crawling subunit is used to obtain web page data, extract and process the web page data using regular expressions to obtain news data;

[0037] The image recognition subunit is used to obtain a text image, and perform text recognition processing on the text image through an optical character recognition engine to obtain news data.

[0038] In some embodiments, the computer vision-based mobile service robot further includes a robot operating system;

[0039] The robot operating system is used to communicate and control the computer vision-based mobile service robot.

[0040] The embodiments of the present application include at least the following beneficial effects: the present application provides a mobile service robot based on computer vision, which performs gesture detection through a gesture detection module, can perform human-computer interaction through gesture recognition, simplifies the difficulty of operation, and provides a reliable operation method for elderly groups with accents that cannot be accurately recognized or other elderly groups who have difficulty in voice control; in addition, the present solution follows and detects falls of the elderly through a fall detection module and a radar following module, can flexibly adapt to the home environment where the elderly live, and its working environment can be closer to the actual life of the elderly; thereby improving the applicability and functionality of the service robot for the elderly group, and can better meet the needs of the elderly and improve their quality of life. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 1 is a schematic diagram of the system structure of a computer vision-based mobile service robot provided in an embodiment of the present application;

[0042] Figure 2 This is a hardware configuration diagram of a computer vision-based mobile service robot provided in an embodiment of the present application;

[0043] Figure 3 This is a data processing flow chart of a mobile service robot based on computer vision provided in an embodiment of the present application;

[0044] Figure 4 This is a data processing flow chart of a gesture recognition module provided in an embodiment of the present application;

[0045] Figure 5 This is a data processing flow chart of a fall detection module provided in an embodiment of the present application;

[0046] Figure 6 This is a data processing flow chart of a radar following module provided in an embodiment of the present application;

[0047] Figure 7 This is a data processing flow chart of a news summary module provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are merely examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0049] It will be understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0050] The terms "at least one", "plurality", "each", "any", etc. used in this application include "at least one", "two" or more, "plurality" or "each", "any" or "any one", "each" or "any one" as used herein.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0052] Before explaining the embodiments of the present application in detail, some of the nouns and terms involved in the embodiments of the present application are first explained. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0053] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0054] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Pre-trained models, also known as large models or basic models, can be fine-tuned and widely applied to downstream tasks across various AI disciplines. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0055] A graphics processing unit (GPU), also known as a display core, visual processor, or display chip, is a microprocessor specifically designed to perform image and graphics-related calculations on personal computers, workstations, game consoles, and some mobile devices (such as tablets and smartphones).

[0056] In related technologies, research on human-robot interaction has mostly focused on how different technologies can improve service robots for the elderly. For example, the MART robot can perform daily care for the elderly, such as eating and shaving. The HealthBot elderly service robot can provide services such as vital sign measurement, reminders, entertainment, communication, and safety assurance. However, in actual applications, it has been found that there are relatively few service robots specifically designed for the elderly, resulting in the majority of related service robots being immobile butler robots. Mobile service robots are generally large in size, lack interaction logic tailored to the elderly, are expensive, and have limited use of artificial intelligence technology, which affects the applicability and functionality of service robots. In order to bring the fruits of scientific and technological development to the elderly, facilitate their lives, and reduce the risk of accidental damage to their lives and health, this proposal proposes a service robot specifically for the elderly that combines mobile robots with artificial intelligence technology.

[0057] In view of this, an embodiment of the present application provides a mobile service robot based on computer vision. The solution includes a gesture detection module, a fall detection module, a radar following module, and a news summary module. The gesture detection module implements an interactive mode based on gesture recognition, which can solve the operation problems of most elderly people and provide a reliable operation method for elderly people with accents that cannot be accurately recognized or other elderly people who have difficulty with voice control. In addition, the fall detection module in this solution integrates a fall detection function based on human posture estimation, which can determine whether the elderly person has fallen by identifying the elderly person's physical state. Combined with the mobile robot design with radar following, it can flexibly adapt to the home environment where the elderly live, and its working environment can be closer to the actual life of the elderly, thereby improving the applicability and functionality of the service robot for the elderly population.

[0058] The present invention provides a computer vision-based mobile service robot, which relates to the field of robotics. The present invention provides a computer vision-based mobile service robot that can be used in home settings, public settings, and elderly service centers and other elderly service locations.

[0059] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0060] Reference Figure 1 , an embodiment of the present application provides a mobile service robot based on computer vision, including a gesture detection module 101, a fall detection module 102, a radar following module 103 and a news summary module 104;

[0061] The gesture detection module 101 is used to obtain a hand image through a camera to perform gesture detection processing and obtain a gesture detection result;

[0062] The fall detection module 102 is configured to obtain a human body image according to the gesture detection result, perform fall detection processing, and obtain a fall detection result;

[0063] The radar following module 103 is configured to obtain laser radar data according to the gesture detection result to follow the computer vision-based mobile service robot;

[0064] The news summary module 104 is used to obtain news data according to the gesture detection result, perform summary processing, and obtain a news summary text.

[0065] In the computer vision-based mobile service robot of the present application, a gesture detection module acquires hand images for gesture detection and obtains corresponding gesture detection results. Hand images can be acquired using a camera, a video camera, or other image acquisition device located within the mobile service robot. It is contemplated that the present application can also acquire a global image using a camera, and then extract the hand image from the global image using image processing techniques such as screenshots and magnification. Human-computer interaction can then be performed based on the corresponding gesture detection results. The fall detection module can be activated based on the gesture detection results to acquire a human image for fall detection and obtain a fall detection result. The human image can be extracted from a global image acquired using a camera or other device, or directly captured using a camera or other device. The human image is analyzed using a neural network model within the fall detection module to obtain a fall detection result. Appropriate alerts and other processing can be performed based on the fall detection result. The radar following module can be activated based on the gesture detection results, and by acquiring lidar data, the mobile service robot can initiate follow-up movement, allowing it to follow a designated target. The news summary module can also be started based on the gesture detection results. By obtaining news data and summarizing the news data based on the generative language model, a news summary text can be obtained, which can be broadcast by voice or displayed on the display.

[0066] Reference Figure 2In the embodiment of the present application, the development board adopts the ROSMASTER X3 JETSON NANO-B01 model, on which components such as radar, depth camera, GPU computing platform, voice interaction expansion board, power motor, power battery pack, high-definition touch screen, and Robot Operating System (ROS) expansion board are integrated. The radar adopts the A1M8 model, and the parameters are set as follows: angular resolution ≤ 1°; power supply current 100mA; power consumption 0.5W; measurement radius 0.15m-12m; sampling frequency 8k. The depth camera uses 3D technology ORBBECR monocular structured light; other parameters are set as follows: accuracy 1m+-3mm; resolution@frame rate (depth) is set to 640×480@30fps, 320×240@30fps, 160x120@30fps; resolution@frame rate (RGB) is set to 1920*1080@30fps, 1280*720@30fps, 640*480@30fps; supports USB2.0; the depth processing chip uses MX6000; the field of view (FOV) is: H58.4°*V45.8°; the working range is 0.6-8m; power consumption is <2.5W. The GPU computing platform uses a 64-bit ARMA57 @ 1.43GHz integrated graphics processor and a 128-core NVIDIA Maxwell @ 921MHz GPU. Other parameters include: 0.5 TFLOPS (FP16) AI computing power; 5W / 10W power consumption; 64GB USB expansion drive; 4GB 64-bit LPDDR4 memory at 25.6GB / s. The voice expansion module includes the YB-4001 module controller; two electret microphones; USB and serial communication; a signal-to-noise ratio of 65dB; a pickup range of 6m; 5V operating voltage; 300mA operating current; and dimensions of 65mm*56mm*1.6mm. The motor uses a 12V constant voltage permanent magnet brushless motor with a rated power of ≤4W and a PH2.0 interface. The parameters of the power battery pack are set as: capacity is 6000mAh; rated discharge current is 8.8A; size is 65*55*37mm; maximum discharge current is 10A; charger is 12.6V800mA.

[0067] In an embodiment of the present application, a radar is used to follow a target person during fall detection, so that the robot maintains a fixed distance from the target person to ensure that the depth camera captures a full-body image of the target person; an Astro Pro depth camera can be used to acquire real-time images, extracting each frame of the image for hand key point extraction for gesture recognition, human key point extraction for fall detection, and text extraction for text news summaries; a JETSON NANO 4GB GPU computing platform is used for real-time image processing, deep learning posture estimation, and sensor data fusion for gesture recognition, fall detection, and text recognition; the power motor is the driving force source of the robot, providing mechanical power to enable the robot car to move, and controlling the current of the motor to achieve control of the robot's speed and direction; the power battery pack stores and provides electrical energy, providing the required power for the motor and other electronic equipment of the robot; the high-definition touch screen enables user interaction with the robot, displaying real-time images and key point recognition results of the gesture recognition and fall detection functions, as well as relevant content of the news summary.

[0068] Reference Figure 3 In the embodiment of the present application, data is collected through an RGB camera and a lidar. The RGB camera is used to obtain images for gesture recognition, fall detection, and news summary tasks, and the lidar is used to obtain the distance and angle data of the object in front for following tasks. The JETSON main control then processes the data based on the ROS system, such as processing image information, radar information, text information, etc., and can also obtain real-time news information from the network for corresponding processing. Finally, the system outputs the relevant processing results to the corresponding components, such as using a four-wheel motor to drive the car to follow the target, playing music, broadcasting dates, fall reminders and news summaries through a Bluetooth speaker, and displaying detection content and program outputs on an LCD display.

[0069] In some embodiments, the gesture detection module includes a hand image acquisition unit, a hand key point extraction unit, and a gesture recognition unit;

[0070] The hand image acquisition unit is used to acquire a hand image through a camera;

[0071] The hand key point extraction unit is used to perform key point extraction processing on the hand image to obtain hand key points;

[0072] The gesture recognition unit is used to perform gesture recognition processing on the hand image according to the hand key points to obtain a gesture recognition result.

[0073] In this embodiment of the present application, a hand image acquisition unit is used to open a camera to acquire a hand image. The hand key point extraction unit then performs key point extraction processing on the hand image. Specifically, a multimedia machine learning model application framework, such as meidapipe, is used to extract hand key points. The mp.solutions.pose module, which is specifically used for posture detection, and the mp.solutions.drawing_utils module, a tool for drawing posture markers and connecting lines on images, are imported from the MediaPipe library. After configuring the relevant parameters, the hand key points are extracted. The gesture recognition unit then performs gesture recognition processing on the hand image based on the hand key points to obtain a gesture recognition result.

[0074] In some embodiments, the gesture recognition unit includes a coordinate calculation subunit, a distance calculation subunit, an extension state determination subunit, and a gesture matching subunit;

[0075] The coordinate calculation subunit is used to determine the key point coordinate data of the fingers according to the key points of the hand;

[0076] The distance calculation subunit is used to perform distance calculation processing on the key point coordinate data to obtain the coordinate distance;

[0077] The extension state determining subunit is configured to determine the extension state of the finger according to the coordinate distance;

[0078] The gesture matching subunit is used to perform gesture matching on the extended state of the finger to obtain a gesture recognition result.

[0079] In an embodiment of the present application, the gesture recognition unit includes a coordinate calculation subunit, a distance calculation subunit, a stretch state determination subunit, and a gesture matching subunit. The coordinate calculation subunit determines the key point coordinate data of the finger based on the key points of the hand, and then the distance calculation subunit performs distance calculation processing on the key point coordinate data to obtain the coordinate distance. The stretch state determination subunit determines the stretch state of the finger based on the coordinate distance. Specifically, the finger stretch state can be detected using the finger_stretch_detect function. The function accepts three parameters point1, point2, and point3, which represent the coordinates of the three key points of the finger respectively. The function determines the stretch state of the finger by calculating the Euclidean distance between two vectors. If the distance from the third key point to the first key point is greater than the distance from the second key point to the first key point, then the finger is stretched and the result 1 is returned; otherwise, the finger is not stretched and the result 0 is returned. The detect_hands_gesture function is then used to check the state of each finger one by one and match different gestures. The "one" gesture means the second finger is extended, and the other fingers are not extended; the "two" gesture means the second and third fingers are extended, and the other fingers are not extended; the "three" gesture means the second, third, and fourth fingers are extended, and the other fingers are not extended; the "four" gesture means the first finger is not extended, and the other fingers are extended; the "five" gesture means all five fingers are extended; the "six" gesture means the first and last fingers are extended, and the other fingers are not extended; the "OK" gesture means the third, fourth, and fifth fingers are extended, and the first and second fingers are not extended; the "stone" gesture means all five fingers are not extended. Figure 4 , and finally, the gesture matching subunit performs gesture matching on the extended state of the finger to obtain the gesture recognition result, such as recognizing the "one" gesture: obtaining the current time, displaying it on the image, and reading the current time through the voice engine; recognizing the "two" gesture: creating a Rosmaster instance, obtaining the battery voltage percentage, and displaying it on the image; recognizing the "three" gesture: calling the play_music() function to play music; recognizing the "four" gesture: starting fall detection and radar following; recognizing the "five" gesture: starting the online news summary function; recognizing the "six" gesture: starting the text news summary function; recognizing the "stone" gesture: stopping the running function. It can be imagined that the embodiment of the present application can also independently set different gestures to turn on or off the corresponding functions. Therefore, the embodiment of the present application obtains key points for gesture matching through the gesture recognition unit, and can turn on or off the corresponding function according to the recognition result, providing a simpler operation method and reducing the operation difficulty for the elderly.

[0080] In some embodiments, the fall detection module includes a human image acquisition unit, a human key point extraction unit, a fall judgment unit, and an alarm unit;

[0081] The human body image acquisition unit is used to acquire a human body image through a camera;

[0082] The human body key point extraction unit is used to perform key point extraction processing on the human body image to obtain human body key points;

[0083] The fall judgment unit is used to perform fall judgment processing on the key points of the human body according to the preset fall conditions to obtain a fall judgment result;

[0084] The alarm unit is used to issue a buzzer alarm according to the fall judgment result.

[0085] In the embodiment of the present application, the fall detection function is enabled according to the corresponding gesture detection result. Figure 5 The data flow in the fall detection module is as follows: The camera is turned on in follow mode to capture a human image. The image frame is input into the multimedia machine learning model application framework to extract key points of human posture. Then, based on the extracted key points and preset fall conditions, fall detection is performed to obtain a fall detection result. Finally, a buzzer alarm is issued based on the fall detection result. This embodiment of the present application pre-sets certain conditions to determine whether a fall has occurred, such as requiring the shoulder to be lower than the ankle, indicating a body tilt. This is done by calculating two ratios of the shoulder key points and the ankle key points: rate1 represents the ratio of the right leg height to the distance from the shoulder to the ankle, and rate2 represents the ratio of the left leg height to the distance from the shoulder to the ankle. When these two ratios are greater than a threshold of 2.5, determined through multiple experimental tests, one of the fall conditions is considered met. When the angle between the left or right leg and the horizontal is less than 40 degrees, it indicates that the legs are not in a normal standing position. When all of the above conditions are met, a fall is determined. If the fall conditions are met, a fall message is output and a buzzer sound is triggered for one second. Add text to the image, display the results of posture detection, and display the currently processed image in the window. This embodiment of the application acquires human body images for fall detection, and can identify the physical state of the elderly based on human posture estimation to determine whether a fall has occurred, thereby improving the safety of the service robot and enabling the service robot to better serve the elderly.

[0086] In some embodiments, the human body key point extraction unit includes an image conversion subunit, a posture estimation subunit, and a post-processing subunit;

[0087] The image conversion subunit is used to perform color space conversion processing on the human body image to obtain a converted image;

[0088] The posture estimation subunit is used to perform key point position prediction processing on the human body image through a key point prediction model to obtain posture key points;

[0089] The post-processing subunit is used to perform outlier removal and smoothing processing on the posture key points to obtain human body key points.

[0090] In this embodiment of the present application, the human key point extraction unit calls the process_frame function to process the current frame image. The image conversion subunit converts the image from BGR color space to RGB color space. This is because the MediaPipe library typically processes images in RGB format. The pose estimation subunit then uses the pose module of the MediaPipe library to process the RGB image and obtain pose detection results. The MediaPipe.pose module of MediaPipe utilizes a convolutional neural network from deep learning for pose estimation. MediaPipe uses a lightweight, real-time deep learning model, prioritizing real-time performance compared to some large deep learning models that require significant computational resources. This model undergoes extensive training and learns to represent human key points in images. The trained neural network predicts the locations of human key points from the image. These key points include body parts such as the head, shoulders, elbows, wrists, hips, knees, and ankles. The detected key points undergo a series of post-processing steps in the post-processing subunit to improve accuracy. This includes removing outliers and smoothing key point trajectories. Finally, the model outputs a data structure containing the locations of human key points. This key point information is subsequently used for fall detection. The embodiment of the present application initializes a series of lists to store the coordinates of key points and specific information of some key points. Check whether the key points of the human posture are successfully detected. Iterate 32 key points of the human body in a loop, obtain the coordinates of the key points, and save them in different lists as needed. The embodiment of the present application performs fall detection by calculating the coordinates of some specific body parts, such as the head, shoulders, ankles, etc., and calculating the angles between the left and right legs and the horizontal plane. It can combine artificial intelligence technology to improve the accuracy of fall detection of service robots.

[0091] In some embodiments, the radar following module includes a lidar data subscription unit, a node activation unit, and a motion control unit;

[0092] The laser radar data subscription unit is used to subscribe to obtain the laser radar data;

[0093] The node starting unit is used to start the follower node according to the laser radar data;

[0094] The motion control unit is used to control the computer vision-based movable service robot to follow the movement by issuing motion instructions through the follower node.

[0095] In the embodiment of the present application, the radar following module includes a laser radar data subscription unit, a node starting unit and a motion control unit. It should be noted that the robot's following function is nested in the fall detection function. When the fall detection is started, the radar following function will be started at the same time, and will stop when the fall detection stops. Figure 6 In this embodiment, while initiating fall detection, the radar following function is activated. Parameters are then initialized to subscribe to lidar data. The nearest object in front is then located, and PID control is used to direct the robot toward the target while maintaining a certain distance. Specifically, this embodiment activates the lidar node, the vehicle chassis drive node, and the handle control node through a node startup unit. The lidar following node is then activated, initializing the vehicle motion control, linear velocity PID controller, angular velocity PID controller, and so on. The lidar data is then subscribed to through a lidar data subscription unit. A dynamic reconfiguration server is then created using the Server class within the motion control unit. Once the corresponding configuration file is modified, it modifies system parameters. The lidar scan data is obtained and filtered based on the set valid angle range, priority follow angle range, and distance threshold. The valid distance and angle data are stored in a list. The minimum distance and its ID in the list are found, indicating that the point is the target to be followed. If the absolute value of the difference between the minimum distance and the set distance is less than 0.1, the minimum distance is set to the set distance. This is to prevent small fluctuations from affecting the robot's motion. The minimum distance and the set distance are input into the PID controller to calculate the robot's desired linear velocity. Similarly, to orient the robot toward its target, the target angle and 0 degrees are input into the PID controller to calculate the robot's desired angular velocity. Velocity information is then published through the ROS system, enabling the robot to orient toward the nearest target while maintaining a constant distance. During this time, the aforementioned configuration file parameters can be modified using ROS tools. The previously created ROS dynamic reconfiguration server will then invoke a callback function to modify system parameters, enabling runtime adjustments of system parameters such as the LiDAR angle, priority angle, target response distance, and PID controller parameters, thereby improving system flexibility and debugging efficiency.

[0096] In some embodiments, the news summarization module includes a news data acquisition unit and a text processing unit;

[0097] The news data acquisition unit is used to acquire news data;

[0098] The text processing unit is used to perform summary generation processing on the news data through a generative language model to obtain a news summary text.

[0099] In this embodiment of the present application, news data is acquired through a news data acquisition unit. There are two ways to acquire news data: by crawling web news or by capturing news text with a camera. The text processing unit then summarizes and generates the news data using a generative language model to produce a news summary text. The generative language model used in this embodiment of the present application utilizes the T5 PEGASUS model, an open-source Chinese generative language model that integrates the T5 and PEGASUS models. The core concept of the T5 model is to convert various natural language processing tasks into natural language generation tasks for processing. The PEGASUS model, meanwhile, is a pre-training method that can be used as a general-purpose generative pre-training task. This model is further pre-trained by constructing tasks using Chinese corpus. Furthermore, the Chinese word segmenter has been improved to make it more suitable for the Chinese context, and the training corpus has been carefully screened and subjected to denoising and other processing. This embodiment of the present application imports the weight file of a trained summary text generation model. This file includes the model's weight parameters and the language knowledge learned during training. Simultaneously, the corresponding word segmenter is loaded to segment the text into units that the model can understand, such as words or subwords. Encode the news text and convert it into a numerical representation that the model can process. This usually includes the use of an embedding layer to map vocabulary to a continuous vector space. Next, use the T5 PEGASUS model for text generation. The model uses a pre-training method to learn language patterns and information from a large-scale corpus. During the generation process, the model can automatically adjust the degree of generalization of the generated text and generate summaries of different lengths and levels of detail according to the task requirements. Figure 7 , the embodiment of the present application can set the parameters of the network request to pass into the basic website, obtain the HTML of the web page and parse it, and extract the news title and text; it can also obtain the news image by opening the device camera, and then after processing through grayscale, Gaussian blur and corrosion, use the optical character recognition (OCR) engine (Tesseract) to obtain Chinese text; finally, by importing the weight file and loading it into the T5 PEGASUS model and word segmenter, the news text is encoded to generate a news summary. Therefore, by setting the maximum length of the generated summary, the embodiment of the present application can infer summary statements of different degrees of generalization from the original news text, so that the service robot can flexibly respond to different needs and generate a summary that is both concise and specific.

[0100] In some embodiments, the news data acquisition unit includes a data crawling subunit and an image recognition subunit;

[0101] The data crawling subunit is used to obtain web page data, extract and process the web page data using regular expressions to obtain news data;

[0102] The image recognition subunit is used to obtain a text image, and perform text recognition processing on the text image through an optical character recognition engine to obtain news data.

[0103] In an embodiment of the present application, the data crawling subunit sets the parameters of the network request, including the HTTP version and User-Agent. Among them, User-Agent is part of the HTTP protocol header and is used to identify the client information. By setting the maximum amount of news data to be saved and passing in the base URL (baseurl), preparations are made for the subsequent acquisition of news data. urllib.request is a module in the urllib library specifically used to open and read resources on the Internet. The base URL is passed to it to obtain the HTML of the web page. HTML is a markup language used to describe the structure and content of a web page. The BeautifulSoup library is then used to parse the HTML and extract data, and regular expressions are used to extract news headlines and text. All the acquired news data is compiled into a list and saved. The image recognition subunit opens the device's camera through OpenCV. OpenCV is a classic computer vision library that provides a variety of computer vision and image processing tools for processing images, videos, camera input, etc. Next, the program waits for the user to press any key, acquires an image from the camera, and displays it in the window. The acquired image is then converted to a grayscale image through OpenCV, and Gaussian blur and erosion operations are performed on it to reduce the difficulty of text recognition. Tesseract is used to recognize Chinese text. Tesseract is an open-source optical character recognition (OCR) engine designed to convert text in images into computer-readable text. In the embodiments of this application, news data can be obtained by crawling online news data or by performing text recognition on images captured by a camera. This enriches the way service robots obtain data and summarizes text through a generative language model, improving the intelligence of service robots.

[0104] In some embodiments, the computer vision-based mobile service robot further includes a robot operating system;

[0105] The robot operating system is used to communicate and control the computer vision-based mobile service robot.

[0106] In an embodiment of the present application, the robot operating system can model the software functions of the robot into nodes, and the nodes communicate by sending messages to each other, thereby controlling the software functions of the robot.

[0107] In the embodiment of the present application, the gesture detection module in the robot is specially designed for the needs of the elderly. Taking into account the special needs and usage habits of the elderly group, the interactive mode of gesture recognition makes the operation simpler and easier to understand. The embodiment of the present application forms an integrated intelligent service system by organically integrating a variety of artificial intelligence technologies into a mobile robot, which includes intelligent gesture recognition, fall detection based on human posture estimation, generative language models, etc., to provide comprehensive and diverse services for the elderly group, breaking through the limitations of the single function of traditional robots. By combining simple hardware devices (such as lidar, cameras, four-wheel robot platforms, etc.) with intelligent algorithms, efficient gesture recognition, posture estimation and other functions are deeply integrated to provide the robot with more powerful perception and decision-making capabilities, making it more intelligent and reliable when serving the elderly.

[0108] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0109] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0110] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0111] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0112] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A mobile service robot based on computer vision, characterized in that: The computer vision-based mobile service robot includes a gesture detection module, a fall detection module, a radar following module and a news summary module; The gesture detection module is used to obtain a hand image through a camera to perform gesture detection processing and obtain a gesture detection result; The fall detection module is configured to obtain a human body image based on the gesture detection result, perform fall detection processing, and obtain a fall detection result; The radar following module is used to obtain laser radar data according to the gesture detection result to control the computer vision-based mobile service robot to follow the movement; The news summary module is used to obtain news data according to the gesture detection result, perform summary processing, and obtain a news summary text; The news summary module includes a news data acquisition unit and a text processing unit; The news data acquisition unit is used to acquire news data; The text processing unit is used to summarize and generate the news data using a generative language model to obtain a news summary text; The news data acquisition unit includes a data crawling subunit and an image recognition subunit; The data crawling subunit is used to obtain web page data, extract and process the web page data using regular expressions to obtain news data; The image recognition subunit is used to obtain a text image, and perform text recognition processing on the text image through an optical character recognition engine to obtain news data.

2. The computer vision-based mobile service robot according to claim 1, characterized in that: The gesture detection module includes a hand image acquisition unit, a hand key point extraction unit and a gesture recognition unit; The hand image acquisition unit is used to acquire a hand image through a camera; The hand key point extraction unit is used to perform key point extraction processing on the hand image to obtain hand key points; The gesture recognition unit is used to perform gesture recognition processing on the hand image according to the hand key points to obtain a gesture recognition result.

3. The computer vision-based mobile service robot according to claim 2, characterized in that: The gesture recognition unit includes a coordinate calculation subunit, a distance calculation subunit, an extension state determination subunit and a gesture matching subunit; The coordinate calculation subunit is used to determine the key point coordinate data of the fingers according to the key points of the hand; The distance calculation subunit is used to perform distance calculation on the key point coordinate data to obtain the coordinate distance; The extension state determining subunit is configured to determine the extension state of the finger according to the coordinate distance; The gesture matching subunit is used to perform gesture matching on the extended state of the finger to obtain a gesture recognition result.

4. The computer vision-based mobile service robot according to claim 1, characterized in that: The fall detection module includes a human body image acquisition unit, a human body key point extraction unit, a fall judgment unit and an alarm unit; The human body image acquisition unit is used to acquire a human body image through a camera; The human body key point extraction unit is used to perform key point extraction processing on the human body image to obtain human body key points; The fall judgment unit is used to perform fall judgment processing on the key points of the human body according to the preset fall conditions to obtain a fall judgment result; The alarm unit is used to issue a buzzer alarm according to the fall judgment result.

5. The computer vision-based mobile service robot according to claim 4, characterized in that: The human body key point extraction unit includes an image conversion subunit, a posture estimation subunit and a post-processing subunit; The image conversion subunit is used to perform color space conversion processing on the human body image to obtain a converted image; The posture estimation subunit is used to perform key point position prediction processing on the human body image through a key point prediction model to obtain posture key points; The post-processing subunit is used to perform outlier removal and smoothing processing on the posture key points to obtain human body key points.

6. The computer vision-based mobile service robot according to claim 1, characterized in that: The radar following module includes a lidar data subscription unit, a node startup unit and a motion control unit; The laser radar data subscription unit is used to subscribe to obtain the laser radar data; The node starting unit is used to start the follower node according to the laser radar data; The motion control unit is used to control the computer vision-based movable service robot to follow the movement by issuing motion instructions through the follower node.

7. The computer vision-based mobile service robot according to any one of claims 1 to 6, characterized in that: The computer vision-based mobile service robot further includes a robot operating system; The robot operating system is used to communicate and control the computer vision-based mobile service robot.

Images