Robot patrol method based on user voice and safety patrol robot

Abstract

The invention discloses a robot patrol method based on user voice and a safety patrol robot, and the method comprises the steps: generating a three-dimensional map according to the position information of an object in a predetermined region; according to the RGB image and the coordinates of the articles in the three-dimensional map, labels are set for the articles in the predetermined area, and a semantic map is generated; and processing the voice information to obtain a target point label, calculating an action route according to the semantic map and the target point label, obtaining image information of the target point label, and replying to the user terminal. The semantic map with the article labels is formed according to the position information of the predetermined area and the article information, the labels and the positions of the corresponding articles are obtained by processing the obtained voice information, and therefore the robot moves to the article positions according to the semantic map to achieve patrol work based on user voice, the patrol efficiency of the patrol robot is improved, and the patrol robot experience is improved. And personnel can conveniently obtain related information of the safety problem at the first time.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent patrol robots, and in particular to a robot patrol method based on user voice and a safety patrol robot. Background Art

[0002] With the continuous advancement of science and technology, intelligent robots are gradually applied to various fields of people's lives, and multiple types with different functions have been derived. Security robots, also known as security robots, are semi-autonomous, autonomous or fully controlled robots that assist humans in completing security protection work. As a sub-field of the robot industry, security robots are based on actual production and life needs and are used to solve safety hazards, patrol monitoring and disaster warnings.

[0003] In the prior art, security robots can only patrol predetermined routes, but cannot patrol predetermined locations according to user instructions. This results in that when a suspicious security issue occurs in a specific area, the patrol robot cannot obtain relevant information about the security issue in the first time, and humans are still required to take the initiative to investigate the actual situation of the problem. In the environment of larger factories or warehouses, there is a problem of untimely investigation of problems.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] In order to solve the problem that security robots in the prior art can only implement patrol work on predetermined routes, when suspicious situations occur, the patrol robots cannot explore the predetermined locations according to user instructions, and manual investigation by staff is still required, and there is a problem of untimely investigation in a larger area environment, the present invention proposes a security patrol robot and a robot patrol method based on user voice.

[0006] The present invention is achieved through the following technical solutions:

[0007] A robot inspection method based on user voice, wherein the robot inspection method based on user voice includes:

[0008] Acquire location information of objects in a predetermined area, and generate a three-dimensional map according to the location information of objects in the predetermined area;

[0009] At the same time, taking RGB images of the objects in the predetermined area, and setting labels for the objects in the predetermined area according to the RGB images and the coordinates of the objects in the three-dimensional map;

[0010] generating a semantic map according to the three-dimensional map and the label of the predetermined area;

[0011] Acquire voice information, process the voice information to acquire a target point label, calculate an action route according to the semantic map and the target point label, and move to an area where the target point label is located;

[0012] The image information of the target point label is obtained, the image information is understood through the visual language model, and the user terminal is responded to through the large language model.

[0013] In the robot inspection method based on user voice, taking an RGB image of the objects in the predetermined area, and setting labels for the objects in the predetermined area according to the RGB image and the coordinates of the objects in the three-dimensional map include:

[0014] Given a frame of point cloud L P, according to the external parameter matrix of radar and camera, the point cloud L P converts from the radar coordinate system to the camera coordinate system to obtain the point cloud C P;

[0015] According to the camera's intrinsic parameter matrix, the point cloud C P is projected from the camera coordinate system to the image coordinate system to obtain the point cloud in the image coordinate system I P;

[0016] The point cloud is segmented by a semantic segmentation model L The images acquired at the same time are semantically segmented to generate corresponding segmentation masks, and the segmentation masks and the point clouds are compared. I P is matched, and when any point falls within the segmentation mask, the point cloud L Label the points in P.

[0017] The robot inspection method based on user voice, wherein the acquiring of voice information, processing of the voice information to acquire target point labels, and calculating of the action route according to the semantic map and the target point labels include:

[0018] Obtain several action routes based on the relative position of the current position and the target point label;

[0019] Calculating cost map value gradients for a number of the action routes, and selecting the best action route according to the cost map value gradients;

[0020] The cost map value gradient is defined as:

[0021] in,

[0022] All objects in the predetermined area are modeled as rectangles, and the size of the modeling is defined. The center point of the rectangle is located at (x obj ,y obj );

[0023] Cost(x θ ,y θ ) is the cost map value of the object edge along the direction θ;

[0024] Cost(x obj ,y obj ) is the cost map value of the object center;

[0025] θ is the Euclidean distance between the center and edge points of the object;

[0026] d safe The safety distance is preset according to the actual size of the robot;

[0027] r is the resolution of the cost map.

[0028] The robot inspection method based on user voice, wherein the Euclidean distance is the distance between the current position and the target point label along the θ direction;

[0029] The calculation formula of the Euclidean distance is:

[0030] D(θ)=[(x θ -x robot ) 2 +(y θ -y robot ) 2 ]·r 2 .

[0031] The robot inspection method based on user voice, wherein the step of calculating the cost map value gradient for the plurality of action routes and selecting the best action route according to the cost map value gradient comprises:

[0032] Setting a scoring function to calculate scores of the plurality of action routes;

[0033] Comparing the scores of the several action routes, and taking the action route with the highest score as the best action route;

[0034] The scoring function is:

[0035] in,

[0036] w g is the weight parameter of the cost map value gradient;

[0037] w d is the weight parameter of the Euclidean distance.

[0038] The robot inspection method based on user voice, wherein the step of obtaining image information of target point labels, understanding the image information through a visual language model, and responding to the user terminal through a large language model comprises:

[0039] Extracting target point labels from the speech information through a large language model and generating a structured detection task;

[0040] Calling a visual language model to perform image understanding on the image information of the target point label;

[0041] The user terminal is responded to through a large language model based on the content understood by the image.

[0042] The large language model includes environmental information semantic text, instance semantic text for completing the detection task, and parameter constraint text for structured output of the large language model.

[0043] A safety inspection robot, wherein the safety inspection robot comprises:

[0044] Vehicle body;

[0045] A radar, the radar being arranged at a predetermined height of the vehicle body, the radar being used to generate a three-dimensional map;

[0046] A camera assembly, the camera assembly is fixedly arranged on the vehicle body and connected to the radar signal, and the camera assembly is used to capture images;

[0047] A processor, the processor is connected to the radar and the camera assembly signals, and a processing program is pre-stored in the processor;

[0048] a memory connected to the processor circuit;

[0049] A communication component is connected to the processor signal, and is used to obtain user instructions and reply to the user terminal.

[0050] The safety inspection robot, wherein the camera assembly comprises:

[0051] An RGB-D camera, wherein the RGB-D camera is used to obtain an RGB image;

[0052] A thermal imaging camera, which is used to obtain status information of an object;

[0053] The RGB-D camera and the thermal imaging camera are fixedly arranged at the front end of the vehicle body.

[0054] The safety inspection robot, wherein the radar is a three-dimensional laser radar;

[0055] A mounting frame is fixedly arranged on the vehicle body, and the radar is fixedly arranged on the top of the mounting frame.

[0056] The security inspection robot, wherein the communication component includes: a voice collector and a wireless signal transceiver.

[0057] The beneficial effects of the present invention are as follows: the present invention forms a semantic map with object labels based on predetermined area location information and object information, obtains the label and position of the corresponding object by processing the acquired voice information, and thus moves to the object position according to the semantic map to realize patrol work based on user voice, thereby improving the patrol efficiency of patrol robots and facilitating personnel to obtain relevant information on safety issues at the first time. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a flow chart of a robot inspection method based on user voice of the present invention;

[0059] Figure 2 It is a schematic diagram of the process of setting labels on items in a predetermined area in the robot inspection method based on user voice of the present invention;

[0060] Figure 3 is a schematic diagram of a semantic map at different viewing angles in the robot inspection method based on user voice of the present invention;

[0061] Figure 4 It is an internal workflow diagram of the robot of the present invention completing the detection task under the guidance of the large language model;

[0062] Figure 5 It is a schematic diagram of the three-dimensional structure of the safety inspection robot of the present invention;

[0063] Figure 6 It is a structural explosion diagram of the safety inspection robot of the present invention.

[0064] exist Figures 1 to 6 Middle: 10, vehicle body; 20, radar; 21, mounting bracket; 30, camera assembly; 31, RGB-D camera; 32, thermal imaging camera. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0066] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications will also change accordingly.

[0067] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0068] In the prior art, security robots can only patrol predetermined routes, but cannot patrol predetermined locations according to user instructions. This results in that when a suspicious security issue occurs in a specific area, the patrol robot cannot obtain relevant information about the security issue in the first time, and humans are still required to take the initiative to investigate the actual situation of the problem. In the environment of larger factories or warehouses, there is a problem of untimely investigation of problems.

[0069] Based on the above problems in the prior art, Figure 1 As shown, the present invention provides a robot inspection method based on user voice, and the robot inspection method based on user voice includes:

[0070] S100, obtaining location information of objects in a predetermined area, and generating a three-dimensional map according to the location information of objects in the predetermined area;

[0071] S200, at the same time, taking an RGB image of the objects in the predetermined area, and setting labels for the objects in the predetermined area according to the RGB image and the coordinates of the objects in the three-dimensional map;

[0072] S300, generating a semantic map according to the three-dimensional map and the label of the predetermined area;

[0073] S400, acquiring voice information, and processing the voice information to acquire a target point label, calculating an action route according to the semantic map and the target point label, and moving to an area where the target point label is located;

[0074] S500: Obtain image information of a target point label, understand the image information through a visual language model, and respond to the user terminal through a large language model.

[0075] In the present application, the predetermined area is the area where the security patrol robot is to conduct security patrols, which may specifically be offices, factories and other areas. There are various items in these areas. Therefore, in order to determine the location of the items so that the security patrol robot can determine the location of the items to be patrolled through user instructions, in this embodiment, the predetermined area and the items in the predetermined area are scanned in advance by the radar of the security patrol robot to form a three-dimensional map with modeling. At the same time, the items in the predetermined area are photographed to form an RGB image for identifying the type of the item. Combined with the location information of the photographed items, labels are set for the items so that several items have unique identification characteristics. After setting the labels of all items, combined with the three-dimensional map formed as above, a labeled semantic map is formed, so that when the user provides information, the label corresponding to the item can be determined through the information, and then the position of the label is determined in the semantic map to achieve accurate navigation.

[0076] The patrol robots in the existing technology cannot meet the needs of practical applications in terms of navigation efficiency and success rate. The reasons include three factors: 1. The existing technology ignores the high spatial accuracy required for the inspection task, resulting in the robot being unable to accurately reach the target point to be inspected; 2. Navigation methods that rely solely on deep neural networks or reinforcement learning require a large amount of training resources and large data sets, but these conditions are greatly limited in practical applications; 3. Existing patrol robots do not incorporate downstream task planning into the navigation and manual interaction in the detection scenario, and therefore cannot achieve precise point-to-point inspection while completing the human-computer interaction function.

[0077] In the present application, based on the above-mentioned semantic map with object labels, when the patrol robot obtains voice information, it can process the voice information to extract the label content of the relevant target points in the user's voice information, and then calculate the action path according to the semantic map and the target point labels to achieve accurate navigation effect. In the present application, the three-dimensional map generated by the radar and the RGB image obtained by the camera are combined to form rich semantic information while providing stable depth estimation. This process can not only provide spatial details for the map, but also link objects with meaningful labels. While providing a basis for the navigation process of specific target points, it also provides a comparison basis for subsequent patrol tasks.

[0078] In actual use, the action route can be calculated through the above-mentioned semantic map and target point label, and the robot moves to the area where the target point label is located according to the action route. After the patrol robot obtains the image information of the target point label, it understands the image information through the visual language model, thereby judging the status of the target point label, and responds to the user terminal through the large language model, thereby achieving the effect of active exploration by auxiliary personnel, effectively improving the patrol efficiency, and ensuring the timeliness of problem detection.

[0079] Based on the above embodiment, in a specific implementation of the present invention, while generating the three-dimensional map, taking an RGB image of the objects in the predetermined area, and setting labels for the objects in the predetermined area according to the RGB image and the coordinates of the objects in the three-dimensional map include:

[0080] S210, given a frame of point cloud L P, according to the external parameter matrix of radar and camera, the point cloud L P converts from the radar coordinate system to the camera coordinate system to obtain the point cloud C P;

[0081] S220, transform the point cloud according to the camera's intrinsic parameter matrix C P is projected from the camera coordinate system to the image coordinate system to obtain the point cloud in the image coordinate system I P;

[0082] S230, using a semantic segmentation model to segment the point cloud L The images acquired at the same time are semantically segmented to generate corresponding segmentation masks, and the segmentation masks and the point clouds are compared. I P is matched, and when any point falls within the segmentation mask, the point cloud L Label the points in P.

[0083] For easier understanding, please refer to the attached drawings. Figure 2 , given a frame of point cloud L P, the present invention uses the external parameter matrix of laser radar and camera to transform the point cloud L P is converted from the laser radar coordinate system to the camera coordinate system to obtain the point cloud C P; then, the point cloud is transformed into C P is projected from the camera coordinate system to the image coordinate system to obtain the point cloud in the image coordinate system I P; At the same time, the pre-trained semantic segmentation model is used in L P performs semantic segmentation on the images captured at the same time and generates the corresponding segmentation mask. Then the point cloud I P is matched with the segmentation mask. If any point falls within the segmentation mask, the point cloud is givenL The points in P are assigned corresponding labels, and then combined with the laser mapping algorithm to generate a global map containing semantic information.

[0084] Combining the above embodiment and step S300, the generated semantic map is as follows: Figure 3 As shown, Figure 3 The left and right parts are semantic maps observed from different perspectives, where the purple, light green, dark green, orange and brown point clouds represent TVs, whiteboards, chairs, keyboards and monitors, respectively. By labeling the specific spatial positions and objects in the three-dimensional map, the inspection robot can accurately understand the objects to be inspected and their locations, thereby accurately obtaining the current status information of the object to complete the inspection.

[0085] In the above embodiments, since the current three-dimensional lidar positioning and navigation framework is not optimized for detection tasks. In the top-down semantic map, the robot often cannot accurately locate the detection point of the target. For example, when the target is against a wall, the robot sometimes navigates to the other side. Although the robot does reach the vicinity of the target object from the top-down map, this is not a correct detection point. To solve this problem, the present application also proposes a method of utilizing semantic information and target cost gradient to ensure the accuracy of inspection positioning.

[0086] Specifically, the above-mentioned acquiring voice information, processing the voice information to acquire a target point label, and calculating an action route according to the semantic map and the target point label include:

[0087] S410, obtaining several action routes according to the relative position of the current position and the target point tag;

[0088] S420, calculating the cost map value gradient for the plurality of action routes, and selecting the best action route according to the cost map value gradient;

[0089] In this embodiment, since the area within the predetermined region may be large, depending on the placement of the items, there may be multiple navigation routes for the patrol robot to reach the target point tag from the current position. In order to enable the patrol robot to select the best action route, in this embodiment, a cost map value gradient is introduced according to the route conditions to ensure that the patrol robot can reach the target point tag in the safest and fastest way to complete the patrol work.

[0090] Specifically, the cost map value gradient is defined as:

[0091]

[0092] When establishing the above three-dimensional map, all objects in the above predetermined area are modeled as rectangles, and the size of the modeling is defined. The center point of the rectangle is (x obj ,y obj );

[0093] Cost(x θ ,y θ ) is the cost map value of the object edge along the direction θ; Cost(x obj ,y obj ) is the cost map value of the object center; θ is the Euclidean distance between the object center and the edge point; d safe is the safety distance preset according to the actual size of the robot; r is the resolution of the cost map (meters per grid unit); the use of the above cost map value gradient solves the problem of accurately locating the detection point when the target object is blocked or the space is limited. The higher the cost map value gradient, the safer the current route. Therefore, through the numerical comparison of the above cost map value gradient, the safety inspection robot can autonomously select the safest route to move, ensuring that the inspection personnel can complete the task.

[0094] Furthermore, the above-mentioned Euclidean distance is the distance between the current position and the target point along the θ direction. The Euclidean distance represents the physical distance between the robot and the target point. The smaller the Euclidean distance, the shorter the action distance. By calculating the Euclidean distance, the safety inspection robot can use the Euclidean distance as the basis for determining the fastest action route, thereby ensuring the efficiency of the safety inspection robot in performing tasks.

[0095] Specifically, the calculation formula of Euclidean distance is:

[0096] D(θ)=[(x θ -x robot ) 2 +(y θ -y robot ) 2 ]·r 2

[0097] Based on the above embodiments, this embodiment can calculate the cost map value gradient for the above S420 and several of the action routes, and select the best action route according to the cost map value gradient for further optimization, so that the security patrol robot can refer to both the cost map value gradient and the Euclidean distance when selecting the action route, thereby selecting the best route with safety and speed.

[0098] Specifically, the steps are as follows:

[0099] S421, setting a scoring function to calculate the scores of the plurality of action routes;

[0100] S422, comparing the scores of the plurality of action routes, and taking the action route with the highest score as the best action route;

[0101] The specific scoring function is:

[0102] Among them, w g and w d is a weight parameter used to balance the importance of gradient and distance. g and w d , the system can give priority to safety (high gradient value) or efficiency (short distance). The candidate point S(θ) with the highest score will be selected as the best target point for the robot to approach, and the normal direction of the candidate point toward the center point of the target object will be used as the orientation of the robot's target point.

[0103] In another possible implementation of the present invention, the above-mentioned acquiring the image information of the target point label, understanding the image information through the visual language model, and replying to the user terminal through the large language model includes:

[0104] S510, extracting target point labels in the speech information through a large language model, and generating a structured detection task;

[0105] S520, calling a visual language model to perform image understanding on the image information of the target point label;

[0106] S530: Reply to the user terminal through the large language model according to the content understood from the image.

[0107] The large language model includes environmental information semantic text, instance semantic text for completing the detection task, and parameter constraint text for structured output of the large language model.

[0108] In this embodiment, the present application adopts a large language model as a high-level task planner for zero samples, and combines it with a traditional low-level task planning framework to form a more general solution; specifically, the large language model is first used to interpret the voice information and extract the navigation target correspondence in the natural language command, and at the same time determine whether to perform the detection task, and generate a structured detection task that the robot can understand through the large language model to realize the navigation process. In the subsequent detection process, the visual language model is also called to enable the robot to understand the image, so that the robot can give a more accurate response to the user through the large language model.

[0109] Among them, the above-mentioned large language model includes semantic text of environmental information. By encoding the semantic graph into text and providing it to the large language model through a parameter configuration file, the robot's understanding of the environment can be promoted; at the same time, the large language model is also provided with instance semantic text for completing the detection task. The instance semantic text covers the entire process from receiving user commands to generating responses, so as to train the robot to perform more accurate patrol work; in addition, structured parameter constraints are imposed on the output of the large language model to ensure the correct generation of detection primitives that the robot can execute in real time.

[0110] In a specific embodiment, the system initialization prompt is composed of several parts, and the inspection robot is responsible for approaching and inspecting an object;

[0111] The following is the semantic information after the map is converted into text:

[0112] The function str(get_semantic_info(map.yaml)) provides relevant information about objects in the working environment, including:

[0113] -id: the specific number of the object;

[0114] - Name: the name of the object;

[0115] - Position: the location of the object in the map;

[0116] -Size: The two-dimensional size of an object.

[0117] The robot’s ability is to output structured raw detection data based on user language instructions and provided environmental information;

[0118] For example, if the user's command is "go to the largest table and see what's on the table", then you need to find the largest table from the available data and output it in the following fixed format:

[0119]

[0120] In this embodiment, if Figure 4 As shown, Figure 4 The internal workflow of the robot completing the detection task under the guidance of the big language model: First, the big language model extracts the target object information from the user instruction and matches it with the semantic map data to generate the detection primitive. Then, the robot navigates to the best point and captures the image, which is then processed into a description by the big language model, simplified by the big language model and returned as a response.

[0121] Based on the above embodiments, the present invention provides a robot inspection method based on user voice. Through the robot inspection method based on user voice, the three-dimensional map can be combined with the label of the object to generate a semantic map with location information and object labels, and by processing the user's voice instructions, the robot can obtain the location and content of the task according to the semantics, automatically select the best route to inspect the task location, and give a reply to the user, which can assist personnel in completing the storage and inspection work in a predetermined scenario, effectively improve the inspection efficiency, and facilitate personnel to obtain relevant information on safety issues as soon as possible.

[0122] Based on the above method, the present invention also provides a safety inspection robot, such as Figure 5 As shown, the security patrol robot includes a body 10, a radar 20, a camera assembly 30, a processor, a memory and a communication assembly, wherein the body 10 is used to carry other functional components and provide mobility for the security patrol robot; the radar 20 is set at a predetermined height of the body 10, and is used to form a three-dimensional map in the above method; the camera assembly 30 is used to shoot the RGB image in the above method to generate labels for objects in a predetermined area, or to form contrast information in subsequent patrol work; the processor is connected to the radar 20 and the camera assembly 30 signals, and a processing program is pre-stored in the processor for processing voice information and image information through the above large language model, and completing other control operations; the processor is used to store relevant data, and the memory is connected to the processor circuit so that the processor can call relevant stored data; the communication assembly is used to obtain user instructions and reply to the user's terminal.

[0123] Specifically, in order to achieve the effect of safety inspection, in this embodiment, if Figure 6 As shown, the camera assembly 30 specifically includes an RGB-D camera 31 and a thermal imaging camera 32, wherein the RGB-D camera 31 is used to obtain RGB images, specifically for setting labels for objects in a predetermined scene, and providing current status image information during security inspections, and the thermal imaging camera 32 is set to obtain the temperature of objects in the scene, so that when it is specifically applied to actual scenes such as warehouses and factories, the security inspection robot has the ability to identify fires and other situations, thereby providing rapid feedback to users.

[0124] In this embodiment, the RGB-D camera 31 and the thermal imaging camera 32 are both fixedly arranged at the front end of the vehicle body 10 to capture image information of the target point.

[0125] Furthermore, if Figure 6As shown, the above-mentioned radar 20 is a three-dimensional laser radar. The three-dimensional laser radar is an optical remote sensing technology that uses laser to measure the distance and speed of the target. It can generate an accurate three-dimensional image of the target object, which can facilitate the accurate construction of a three-dimensional map, thereby providing a basis for the navigation function of the security patrol robot.

[0126] In this embodiment, a mounting frame 21 is fixedly provided on the vehicle body 10, and the radar 20 is fixedly provided on the top of the mounting frame 21. The mounting position of the radar 20 can be raised by the mounting frame 21, so as to facilitate scanning of the panoramic view of the predetermined area.

[0127] In another embodiment of the present invention, the above-mentioned communication component specifically includes a voice collector and a wireless signal transceiver, wherein the voice collector is used to obtain the user's voice information, and cooperate with the large language model in the processor to process the voice information obtained by the voice collector to generate task instructions that can be recognized by the security patrol robot; the wireless signal transceiver is used to send feedback information to the user terminal. Because the patrol position of the security patrol robot may be far away from the user in different scenarios, the patrol information of the security patrol robot after checking the target point position can be fed back to the user in real time through the wireless signal transceiver, so as to achieve the effect of timely troubleshooting.

[0128] In summary, the present invention provides a robot inspection method based on user voice and a security inspection robot, which includes: obtaining the location information of items in a predetermined area, generating a three-dimensional map according to the location information of items in the predetermined area; at the same time, taking RGB images of the items in the predetermined area, setting labels for the items in the predetermined area according to the RGB images and the coordinates of the items in the three-dimensional map; generating a semantic map according to the three-dimensional map and the labels of the predetermined area; obtaining voice information, and processing the voice information to obtain target point labels, calculating the action route according to the semantic map and the target point labels, and moving to the area where the target point labels are located; obtaining the image information of the target point labels, and comparing it with the target point label information in the semantic map, and replying to the user terminal. The present invention forms a semantic map with item labels according to the predetermined area location information and item information, obtains the label and position of the corresponding item by processing the acquired voice information, and thus moves to the item position according to the semantic map to realize the inspection work based on the user voice, improves the inspection efficiency of the inspection robot, and facilitates personnel to obtain relevant information about safety issues in the first time.

[0129] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A robot inspection method based on user voice, characterized in that: The robot inspection method based on user voice includes: Acquire location information of objects in a predetermined area, and generate a three-dimensional map according to the location information of objects in the predetermined area; At the same time, taking RGB images of the objects in the predetermined area, and setting labels for the objects in the predetermined area according to the RGB images and the coordinates of the objects in the three-dimensional map; generating a semantic map according to the three-dimensional map and the label of the predetermined area; Acquire voice information, process the voice information to acquire a target point label, calculate an action route according to the semantic map and the target point label, and move to an area where the target point label is located; The image information of the target point label is obtained, the image information is understood through the visual language model, and the user terminal is responded to through the large language model.

2. The robot inspection method based on user voice according to claim 1 is characterized in that: Taking an RGB image of the objects in the predetermined area, and setting labels for the objects in the predetermined area according to the RGB image and the coordinates of the objects in the three-dimensional map includes: Given a frame of point cloud L P, according to the external parameter matrix of radar and camera, the point cloud L P converts from the radar coordinate system to the camera coordinate system to obtain the point cloud C P; According to the camera's intrinsic parameter matrix, the point cloud C P is projected from the camera coordinate system to the image coordinate system to obtain the point cloud in the image coordinate system I P; The point cloud is segmented by a semantic segmentation model L The images acquired at the same time are semantically segmented to generate corresponding segmentation masks, and the segmentation masks and the point clouds are compared. I P is matched, and when any point falls within the segmentation mask, the point cloud L Label the points in P.

3. The robot inspection method based on user voice according to claim 1 is characterized in that: The acquiring of voice information, processing of the voice information to acquire a target point label, and calculating an action route according to the semantic map and the target point label include: Obtain several action routes based on the relative position of the current position and the target point label; Calculating cost map value gradients for a number of the action routes, and selecting the best action route according to the cost map value gradients; The cost map value gradient is defined as: in, All objects in the predetermined area are modeled as rectangles, and the size of the modeling is defined. The center point of the rectangle is located at (x obj ,y obj ); Cost(x θ ,y θ ) is the cost map value of the object edge along the direction θ; Cost(x obj ,y obj ) is the cost map value of the object center; θ is the Euclidean distance between the center and edge points of the object; d safe The safety distance is preset according to the actual size of the robot; r is the resolution of the cost map.

4. The robot inspection method based on user voice according to claim 3 is characterized in that: The Euclidean distance is the distance between the current position and the target point label along the θ direction; The calculation formula of the Euclidean distance is: D(θ)=[(x θ -x robot ) 2 +(y θ -y robot ) 2 ]·r 2 .

5. The robot inspection method based on user voice according to claim 4 is characterized in that: Calculating the cost map value gradient for the plurality of action routes and selecting the best action route according to the cost map value gradient comprises: Setting a scoring function to calculate scores of the plurality of action routes; Comparing the scores of the several action routes, and taking the action route with the highest score as the best action route; The scoring function is: in, w g is the weight parameter of the cost map value gradient; w d is the weight parameter of the Euclidean distance.

6. The robot inspection method based on user voice according to claim 1 is characterized in that: The acquiring of the image information of the target point label, understanding the image information through the visual language model, and responding to the user terminal through the large language model includes: Extracting target point labels from the speech information through a large language model and generating a structured detection task; Calling a visual language model to perform image understanding on the image information of the target point label; The user terminal is responded to through a large language model based on the content understood by the image. The large language model includes environmental information semantic text, instance semantic text for completing the detection task, and parameter constraint text for structured output of the large language model.

7. A safety inspection robot, characterized in that: The safety inspection robot comprises: Vehicle body; A radar, the radar being arranged at a predetermined height of the vehicle body, the radar being used to generate a three-dimensional map; A camera assembly, the camera assembly is fixedly arranged on the vehicle body and connected to the radar signal, and the camera assembly is used to capture images; A processor, the processor is connected to the radar and the camera assembly signals, and a processing program is pre-stored in the processor; a memory connected to the processor circuit; A communication component is connected to the processor signal, and is used to obtain user instructions and reply to the user terminal.

8. The safety inspection robot according to claim 7, characterized in that: The camera assembly comprises: An RGB-D camera, wherein the RGB-D camera is used to obtain an RGB image; A thermal imaging camera, which is used to obtain status information of an object; The RGB-D camera and the thermal imaging camera are fixedly arranged at the front end of the vehicle body.

9. The security inspection robot according to claim 7, characterized in that: The radar is a three-dimensional laser radar; A mounting frame is fixedly arranged on the vehicle body, and the radar is fixedly arranged on the top of the mounting frame.

10. The security inspection robot according to claim 7, characterized in that: The communication component includes: a voice collector and a wireless signal transceiver.

Images