Service robot control method and system based on millimeter wave and vision dual control

By combining millimeter wave and vision technology to identify user gestures, the problem of service robots' difficulty in identifying in lighting environments is solved, and its practicality in different environments is improved.

CN118915723APending Publication Date: 2024-11-08NEUSOFT INST GUANGDONG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410945791.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing service robots are difficult to recognize user gestures in lighting environments, and are greatly affected by lighting and the environment, which limits their operations in restaurants and public places.

Method used

Using a dual-control method based on millimeter wave and vision, we use a visual camera or millimeter wave radar to identify user gestures by obtaining task instructions, building mobile routes, and judging lighting conditions. We use a visual camera or millimeter wave radar to identify user gestures, and combine convolutional neural networks for gesture recognition.

Benefits of technology

It improves the practicality of service robots in different environments, ensures that user gestures can be accurately recognized when lighting conditions are poor, and expands its scope of application.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118915723A_ABST
    Figure CN118915723A_ABST
Patent Text Reader

Abstract

The service robot control method based on millimeter wave and vision dual control comprises the following steps that a task instruction is obtained, and the task instruction is analyzed to determine a target position where a service robot needs to move; constructing a moving route according to the target position and the position of the robot, and driving the service robot to the target position; after the service robot arrives at the target position, judging whether the current illumination meets an application condition or not, if so, starting the visual camera to obtain a user gesture, and if not, starting the millimeter-wave radar to obtain radar data in the current environment, and analyzing the user gesture according to the radar data; and the service robot makes corresponding services according to the gestures of the user. And through combination of millimeter wave and vision dual control, the application range of the service robot is wider, and the practicability of the service robot in different environments is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a service robot control method and system based on millimeter wave and vision dual control. Background Art

[0002] With the continuous development of robot technology, many catering or service industries have begun to use robots to provide corresponding services. For example, after receiving instructions, they move to the corresponding location to provide services to users, such as purchasing beverages, transporting food, or providing users with corresponding map information.

[0003] However, some services require subsequent operations based on image recognition, such as determining the robot's actions through user gestures and built-in gesture commands. When recognizing user gestures, it is easily affected by lighting and the environment, which limits the robot's operation in restaurants and public places. Therefore, it is urgent to find a way to reduce the impact of the environment and ensure that the robot can recognize user gestures in a lighting environment. Summary of the invention

[0004] In view of the above-mentioned defects, the purpose of the present invention is to propose a service robot control method and system based on millimeter wave and vision dual control, which improves the practicability of the service robot in different environments.

[0005] To achieve this purpose, the present invention adopts the following technical solution: A service robot control method based on millimeter wave and visual dual control comprises the following steps:

[0006] Step S1: Obtaining a task instruction, parsing the task instruction to determine the target position to which the service robot is to move;

[0007] Step S2: constructing a moving route according to the target position and the position of the robot, and driving the service robot to the target position;

[0008] Step S3: When the service robot reaches the target position, it determines whether the current lighting meets the application conditions. If so, the visual camera is started to obtain the user gesture. If not, the millimeter wave radar is started to obtain radar data in the current environment, and the user gesture is analyzed according to the radar data.

[0009] Step S4: The service robot provides corresponding services according to the user's gestures.

[0010] Preferably, the task instruction is Bluetooth data.

[0011] Preferably, the steps of establishing the moving route in step S2 are specifically as follows:

[0012] Build corresponding nodes according to the location of the service and determine the adjacency relationship between the nodes;

[0013] Create a queue to be searched and a queue that has been searched;

[0014] The current position of the service robot is taken as the starting node, and the target position is taken as the final node;

[0015] Search for the adjacent nodes of the starting node, put the starting node and the adjacent nodes into the searched queue, and put the other nodes into the waiting-to-search queue;

[0016] Search the adjacent nodes of the nodes in the searched queue, put the adjacent nodes of the queue to be searched into the searched queue, and put other nodes into the queue to be searched, repeat the steps until the end node appears in the searched queue, and construct the movement route according to the parent nodes between the nodes in the searched queue.

[0017] Preferably, before executing step S3, the following steps need to be performed:

[0018] Different gesture radar data are processed in the distance dimension and Doppler dimension to obtain the distance-Doppler characteristic graph;

[0019] Remove interference from the range-Doppler feature map and filter moving target points to obtain a feature map for gesture recognition;

[0020] The feature map of gesture recognition is input into the convolutional neural network for learning and classification.

[0021] Preferably, the convolutional neural network comprises two convolutional layers, two maximum pooling layers, a flattening layer and two fully connected layers, and the output of the convolutional neural network is connected to the input of the LSTM network;

[0022] The first convolutional layer is the CoordConv layer, which adds two channels containing coordinate information to the input.

[0023] A service robot control system based on millimeter wave and visual dual control, using the service robot control method based on millimeter wave and visual dual control, including an instruction acquisition module, a driving module, a gesture judgment module and a service module;

[0024] The instruction acquisition module is used to: acquire task instructions, analyze the task instructions to determine the target position to which the service robot is to move;

[0025] The driving module is used to construct a moving route according to the target position and the position of the robot, and drive the service robot to the target position;

[0026] The gesture judgment module is used to judge whether the current lighting meets the application conditions when the service robot reaches the target position. If so, the visual camera is started to obtain the user gesture. If not, the millimeter wave radar is started to obtain the radar data in the current environment and the user gesture is analyzed according to the radar data.

[0027] The service module is used to drive the service robot to provide corresponding services according to user gestures.

[0028] Preferably, the driving module includes a construction submodule, a search submodule and a route submodule;

[0029] The construction submodule is used to construct corresponding nodes according to the location of the service and cut the adjacency relationship between the nodes;

[0030] Create a queue to be searched and a queue that has been searched;

[0031] The current position of the service robot is taken as the starting node, and the target position is taken as the final node;

[0032] The search submodule is used to search for adjacent nodes of the starting node, put the adjacent nodes into the searched queue, and put other nodes into the to-be-searched queue;

[0033] Search for the adjacent nodes of the nodes in the searched queue, put the adjacent nodes into the searched queue, and put the other nodes into the queue to be searched;

[0034] When the final node appears in the searched queue, the route submodule constructs the moving route according to the parent nodes between the nodes.

[0035] Preferably, the gesture judgment module includes a radar data judgment submodule;

[0036] The radar data judgment submodule is used to process different gesture radar data in the distance dimension and the Doppler dimension to obtain a distance-Doppler characteristic graph;

[0037] Remove interference from the range-Doppler feature map and filter moving target points to obtain a feature map for gesture recognition;

[0038] The feature map of gesture recognition is input into the convolutional neural network for learning and classification.

[0039] One of the above technical solutions has the following advantages or beneficial effects: through the combination of millimeter wave and visual dual control, the application scope of the service robot is wider, and the practicality of the service robot in different environments is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of an embodiment of the method of the present invention.

[0041] Figure 2 It is a schematic diagram of the structure of an embodiment of the system of the present invention. DETAILED DESCRIPTION

[0042] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0043] In the description of the embodiments of the present invention, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0044] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, the meaning of "plurality" is two or more. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0045] like Figures 1-2 As shown, the service robot control method based on millimeter wave and visual dual control includes the following steps:

[0046] Step S1: Obtaining a task instruction, parsing the task instruction to determine the target position to which the service robot is to move;

[0047] Step S2: constructing a moving route according to the target position and the position of the robot, and driving the service robot to the target position;

[0048] Step S3: When the service robot reaches the target position, it determines whether the current lighting meets the application conditions. If so, the visual camera is started to obtain the user gesture. If not, the millimeter wave radar is started to obtain radar data in the current environment, and the user gesture is analyzed according to the radar data.

[0049] Step S4: The service robot provides corresponding services according to the user's gestures.

[0050] When the lighting conditions do not meet the application conditions, the image recognition technology collects pictures with lighting effects, resulting in the inability to recognize the user's gesture. For this reason, the present invention is provided with image recognition technology and millimeter wave radar technology to realize the user's timing recognition. When the lighting conditions meet the application conditions, the visual camera is started. At this time, due to the movement to the user's position, the direct visual camera can obtain the user's image containing gestures, and then the user's gesture can be judged by inputting the image into the existing recognition model. When the lighting conditions do not meet the application gesture, the millimeter wave radar is started, and the user's gesture is parsed through the data returned by the radar. Since the radar data is acquired through the reflection of the radar, the radar gesture recognition is not sensitive to the change of lighting conditions and can work normally in complex environments such as darkness, uneven lighting or strong light. However, it may be affected by other objects, so there will be misjudgment when using radar data to recognize simple gestures, but in the case of poor lighting conditions, the accuracy of gestures recognized by radar data is higher than that of image recognition. In the service robot, the corresponding gestures are associated with the corresponding services. For example, when gesture 1 is made, the robot can output the first beverage, when gesture 2 is made, the robot can output the second beverage, when gesture 3 is made, the robot can hand out tableware, etc. The combination of millimeter wave and visual dual control makes the service robot more widely applicable and improves its practicality in different environments.

[0051] Preferably, the task instruction is Bluetooth data.

[0052] The service robot of the present invention can be used in crowded places such as restaurants and train stations. When sending task instructions, if wireless signals are used, they are easily affected by nearby signals, resulting in the task instructions not being able to reach the service robot in time. Therefore, the present invention uses Bluetooth data as the data type of the task instruction to ensure that the service robot can respond immediately in crowded places.

[0053] Preferably, the steps of establishing the moving route in step S2 are specifically as follows:

[0054] Build corresponding nodes according to the location of the service and determine the adjacency relationship between the nodes;

[0055] Create a queue to be searched and a queue that has been searched;

[0056] The current position of the service robot is taken as the starting node, and the target position is taken as the final node;

[0057] Search for the adjacent nodes of the starting node, put the starting node and the adjacent nodes into the searched queue, and put the other nodes into the waiting-to-search queue;

[0058] Search the adjacent nodes of the nodes in the searched queue, put the adjacent nodes of the queue to be searched into the searched queue, and put other nodes into the queue to be searched, repeat the steps until the end node appears in the searched queue, and construct the movement route according to the parent nodes between the nodes in the searched queue.

[0059] The following example is used to explain the above. For example, there are multiple seats in a restaurant, and each seat is a service position. Therefore, the seats can be used as nodes, and then the adjacency relationship between the nodes is determined according to the layout between the seats. The specific adjacency relationship can be as follows:

[0060] A:['B,'C'],

[0061] B:['A','D','E'],

[0062] C:['A','F'],

[0063] D:['B'],

[0064] E:['B','F'],

[0065] F:['C','E'],

[0066] Node A is adjacent to nodes B and C, and node B is adjacent to nodes A, D, and E. If the server is currently at node A, node A is the starting node, and the node of the task instruction is issued by node F, then node F is the final node;

[0067] At this time, node A, B and C will be placed in the searched queue, and nodes D, E and F are still in the queue to be searched. Since F is not in the searched queue, it is necessary to continue searching the adjacent nodes of the searched queue. At this time, the adjacent nodes of the searched queue are nodes B and C.

[0068] For node B, the adjacent nodes in the queue to be searched are nodes D and E. At this time, nodes D and E are put into the queue to be searched;

[0069] For node C, the adjacent node in the queue to be searched is node F, so node F is put into the queue to be searched;

[0070] Now the F node has fallen into the queue to be searched. At this time in the searched queue, the negative node of the F node is the C node, and the parent node of the C node is the A node. At this time, the movement route A→C→F is constructed.

[0071] The mobile route construction method of the present invention does not need to use the route planning algorithm integrated in ROS, which reduces the cost of system development, so that the present technical solution can perform route planning on low-density maps such as restaurants and train stations under low-cost conditions.

[0072] Preferably, before executing step S3, the following steps need to be performed:

[0073] Different gesture radar data are processed in the distance dimension and Doppler dimension to obtain the distance-Doppler characteristic graph;

[0074] Specifically, the distance dimension processing can use the radar pulse repetition frequency (PRF) and the delay time of the echo signal to calculate the distance of the target. The Doppler dimension processing can analyze the change of the echo signal frequency (i.e., Doppler shift) through FFT to infer the radial velocity of the target.

[0075] By processing the distance and Doppler dimensions of radar data, the dynamic change information of gestures in three-dimensional space (including distance, speed and angle) can be accurately extracted. This method can directly capture the subtle movement characteristics of gestures and provide a high-quality data foundation for subsequent recognition.

[0076] Remove interference from the range-Doppler feature map and filter moving target points to obtain a feature map for gesture recognition;

[0077] The interference removal can be achieved by setting appropriate amplitude and speed thresholds to remove interference from background noise and static objects, or by using morphological filters (such as corrosion, expansion, opening operation, closing operation) to further clean up the image and highlight the moving target. Then, when the target points are screened, features that can represent the gesture, such as the position, speed, trajectory, etc. of the target, can be extracted from the processed RDI, and finally the gesture features can be converted into a two-dimensional matrix, etc., in the form of CNN input.

[0078] After obtaining the range-Doppler feature map, by removing the interference signals (such as clutter generated by static objects, background noise, etc.), the signal-to-noise ratio of the data can be significantly improved, thereby enhancing the accuracy and robustness of gesture recognition.

[0079] The feature map of gesture recognition is input into the convolutional neural network for learning and classification.

[0080] Preferably, the convolutional neural network comprises two convolutional layers, two maximum pooling layers, a flattening layer and two fully connected layers, and the output of the convolutional neural network is connected to the input of the LSTM network;

[0081] The first convolutional layer is the CoordConv layer, which adds two channels containing coordinate information to the input.

[0082] Since there are no specific image features in the radar signal for identification, only information such as posture, the CoordConv layer in the present invention adds two coordinate channels (representing x and y coordinates respectively) to the input feature map. The CoordConv layer enables the convolution process to perceive the spatial position information of each element in the feature map, which helps to improve the recognition accuracy in tasks that require position information, such as target detection and posture estimation, and improves the recognition of radar information feature maps.

[0083] By stacking two convolutional layers and two maximum pooling layers, the network can extract abstract features of the input data layer by layer. The convolutional layer is responsible for extracting local features, while the pooling layer reduces the amount of calculation and the number of parameters by downsampling, while improving the robustness of the features. The flattening layer flattens the output of the convolutional layer and the pooling layer into a one-dimensional vector for subsequent processing by the fully connected layer. The fully connected layer maps the feature vector to the output space through the weight matrix and the bias term to achieve classification.

[0084] A service robot control system based on millimeter wave and visual dual control, using the service robot control method based on millimeter wave and visual dual control, including an instruction acquisition module, a driving module, a gesture judgment module and a service module;

[0085] The instruction acquisition module is used to: acquire task instructions, analyze the task instructions to determine the target position to which the service robot is to move;

[0086] The driving module is used to construct a moving route according to the target position and the position of the robot, and drive the service robot to the target position;

[0087] The gesture judgment module is used to judge whether the current lighting meets the application conditions when the service robot reaches the target position. If so, the visual camera is started to obtain the user gesture. If not, the millimeter wave radar is started to obtain the radar data in the current environment and the user gesture is analyzed according to the radar data.

[0088] The service module is used to drive the service robot to provide corresponding services according to user gestures.

[0089] Preferably, the driving module includes a construction submodule, a search submodule and a route submodule;

[0090] The construction submodule is used to construct corresponding nodes according to the location of the service and cut the adjacency relationship between the nodes;

[0091] Create a queue to be searched and a queue that has been searched;

[0092] The current position of the service robot is taken as the starting node, and the target position is taken as the final node;

[0093] The search submodule is used to search for adjacent nodes of the starting node, put the adjacent nodes into the searched queue, and put other nodes into the to-be-searched queue;

[0094] Search for the adjacent nodes of the nodes in the searched queue, put the adjacent nodes into the searched queue, and put the other nodes into the queue to be searched;

[0095] When the final node appears in the searched queue, the route submodule constructs the moving route according to the parent nodes between the nodes.

[0096] Preferably, the gesture judgment module includes a radar data judgment submodule;

[0097] The radar data judgment submodule is used to process different gesture radar data in the distance dimension and the Doppler dimension to obtain a distance-Doppler characteristic graph;

[0098] Remove interference from the range-Doppler feature map and filter moving target points to obtain a feature map for gesture recognition;

[0099] The feature map of gesture recognition is input into the convolutional neural network for learning and classification.

[0100] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0101] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A service robot control method based on millimeter wave and visual dual control, characterized in that: The steps include: Step S1: Obtaining a task instruction, parsing the task instruction to determine the target position to which the service robot is to move; Step S2: constructing a moving route according to the target position and the position of the robot, and driving the service robot to the target position; Step S3: When the service robot reaches the target position, it determines whether the current lighting meets the application conditions. If so, the visual camera is started to obtain the user gesture. If not, the millimeter wave radar is started to obtain radar data in the current environment, and the user gesture is analyzed according to the radar data. Step S4: The service robot provides corresponding services according to the user's gestures.

2. The service robot control method based on millimeter wave and visual dual control according to claim 1 is characterized in that: The task instruction is Bluetooth data.

3. The service robot control method based on millimeter wave and visual dual control according to claim 1 is characterized in that: The steps of establishing the moving route in step S2 are specifically as follows: Build corresponding nodes according to the location of the service and determine the adjacency relationship between the nodes; Create a queue to be searched and a queue that has been searched; The current position of the service robot is taken as the starting node, and the target position is taken as the final node; Search for the adjacent nodes of the starting node, put the starting node and the adjacent nodes into the searched queue, and put the other nodes into the waiting-to-search queue; Search the adjacent nodes of the nodes in the searched queue, put the adjacent nodes of the queue to be searched into the searched queue, and put other nodes into the queue to be searched, repeat the steps until the end node appears in the searched queue, and construct the movement route according to the parent nodes between the nodes in the searched queue.

4. The service robot control method based on millimeter wave and visual dual control according to claim 1 is characterized in that: Before executing step S3, the following steps need to be executed: Different gesture radar data are processed in the distance dimension and Doppler dimension to obtain the distance-Doppler characteristic graph; Remove interference from the range-Doppler feature map and filter moving target points to obtain a feature map for gesture recognition; The feature map of gesture recognition is input into the convolutional neural network for learning and classification.

5. The service robot control method based on millimeter wave and vision dual control according to claim 4 is characterized in that: The convolutional neural network comprises two convolutional layers, two maximum pooling layers, a flat layer and two fully connected layers, and the output of the convolutional neural network is connected to the input of the LSTM network; The first convolutional layer is the CoordConv layer, which adds two channels containing coordinate information to the input.

6. A service robot control system based on millimeter wave and visual dual control, characterized in that: A control method for a service robot using millimeter wave and visual dual control as claimed in any one of claims 1 to 5, comprising an instruction acquisition module, a driving module, a gesture judgment module and a service module; The instruction acquisition module is used to: acquire task instructions, analyze the task instructions to determine the target position to which the service robot is to move; The driving module is used to construct a moving route according to the target position and the position of the robot, and drive the service robot to the target position; The gesture judgment module is used to judge whether the current lighting meets the application conditions when the service robot reaches the target position. If so, the visual camera is started to obtain the user gesture. If not, the millimeter wave radar is started to obtain the radar data in the current environment and the user gesture is analyzed according to the radar data. The service module is used to drive the service robot to provide corresponding services according to user gestures.

7. The service robot control system based on millimeter wave and vision dual control according to claim 6 is characterized in that: The driving module includes a construction submodule, a search submodule and a route submodule; The construction submodule is used to construct corresponding nodes according to the location of the service and cut the adjacency relationship between the nodes; Create a queue to be searched and a queue that has been searched; The current position of the service robot is taken as the starting node, and the target position is taken as the final node; The search submodule is used to search for adjacent nodes of the starting node, put the adjacent nodes into the searched queue, and put other nodes into the to-be-searched queue; Search for the adjacent nodes of the nodes in the searched queue, put the adjacent nodes into the searched queue, and put the other nodes into the queue to be searched; When the final node appears in the searched queue, the route submodule constructs the moving route according to the parent nodes between the nodes.

8. According to the service robot control system based on millimeter wave and visual dual control as claimed in claim 6, the gesture judgment module includes a radar data judgment submodule; The radar data judgment submodule is used to process different gesture radar data in the distance dimension and the Doppler dimension to obtain a distance-Doppler characteristic graph; Remove interference from the range-Doppler feature map and filter moving target points to obtain a feature map for gesture recognition; The feature map of gesture recognition is input into the convolutional neural network for learning and classification.