On-site investigation and material supply method, system and equipment based on rescue robot

Abstract

The present invention discloses a method, system and equipment for on-site investigation and material supply based on a rescue robot, the method comprising: obtaining a planned path to enter a rescue site based on a priori map, and sending an operation instruction to a chassis control system based on the planned path, so that the rescue robot enters the accident site according to the planned path according to the operation instruction; obtaining environmental data and real-time image data of the accident site, using a multi-classifier fusion algorithm to identify the disaster site environmental data in real time, and transmitting the environmental data and real-time image data to a command center; receiving a rescue strategy formulated by the command center based on the real-time image data, and completing on-site investigation and material supply based on the rescue strategy. Thus, the technical problem of low efficiency of on-site investigation and material supply of rescue robots in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of emergency rescue equipment, and in particular to a rescue robot-based on-site investigation and material supply method, system and equipment. Background Art

[0002] Natural disasters and man-made accidents are generally characterized by low predictability and great destructive power when they occur. There are also many uncertainties at the accident site, and secondary disasters such as explosions and landslides may occur. The risk of implementing on-site rescue is high, and incidents of rescue workers being killed also occur from time to time.

[0003] The occurrence of serious accidents in recent years shows that casualties caused by the complex and changeable environment at the accident site has gradually become one of the main factors. For example, in accidents such as fires and explosions of hazardous chemicals, the first thing to do is to quickly understand the on-site environment and eliminate major hidden dangers. In addition, during the rescue process, the supply of rescue equipment, materials and medicines is often insufficient, and their transportation and storage also cost a lot of manpower, affecting the efficiency of rescue.

[0004] In recent years, robots for rescue site surveys and rescue material supply have mainly been controlled remotely. The command center uses remote control to enter the scene, and the robot does not have the ability to plan paths and navigate autonomously. In terms of on-site surveys, the command center uses the real-time images sent back by the robot to understand the situation at the accident site, and combines thermal imagers, temperature sensors and other sensors to understand the specific situation at the accident site, which places high demands on operators.

[0005] Therefore, in terms of rescue site investigation and rescue material supply, the existing technical shortcomings mainly include the following aspects:

[0006] First, most of the existing rescue robots do not have the ability to navigate autonomously. They mainly use remote control to remotely control the robots through the real-time images sent back, and the operators remotely control the robots to conduct on-site surveys. In addition, the existing rescue robots do not have the ability to perceive and identify the environment, and they mainly rely on the operators to judge the conditions on the scene, which may result in certain human errors and place high demands on the operators. Secondly, the robots do not have the ability to predict possible dangers on the scene during operation, and they rely on the remote control of the operators, which can easily cause irreparable losses; secondly, in terms of material supply, the existing technology mainly uses human transportation, and some medicines, flammable and explosive items are stored in professional lockers; when there is a lack of rescue materials, rescue personnel need to be dispatched to the rescue site. For accidents such as fires and explosions, dispatching rescue personnel poses certain safety risks. Summary of the invention

[0007] The purpose of the present invention is to provide a rescue robot to at least partially solve the technical problems existing in the prior art. This purpose is achieved through the following technical solutions:

[0008] The present invention provides a method for on-site investigation, safety control and material supply based on a rescue robot, the method comprising:

[0009] Acquire a planned path to the rescue site based on the prior map, and send an operation instruction to the chassis control system based on the planned path, so that the rescue robot enters the accident site according to the planned path according to the operation instruction;

[0010] Obtain environmental data and real-time image data from the accident site, use the on-site edge computing unit to run a multimodal prediction algorithm, and complete the on-site environmental survey. Then, based on the identification results of the dangerous objects and the depth information of the visual sensor, make a dangerous object hazard index evaluation and display it in the real-time image. And transmit the environmental data and real-time image data to the command center;

[0011] The command center formulates rescue strategies based on real-time image data and completes the robot's movement goals and material supply based on the rescue strategies;

[0012] According to the rescue mission, the on-site robot runs the path planning algorithm and autonomously navigates to the target location to deliver materials. It adopts different travel speeds according to the different danger indexes and can re-plan the path when there is a high risk.

[0013] Further, the method of acquiring a planned path to enter the rescue site based on the prior map and sending an operation instruction to the chassis control system based on the planned path so that the rescue robot enters the accident site according to the planned path according to the operation instruction specifically includes:

[0014] Establish an optimal collision-free path from the starting point to the target point based on the prior map;

[0015] Sending an operation instruction to the chassis control system based on the optimal collision-free path, so that the rescue robot enters the accident scene along the optimal collision-free path according to the operation instruction;

[0016] At the same time, in the process of entering the accident scene, dynamic obstacles in the optimal collision-free path are avoided in real time.

[0017] Furthermore, establishing an optimal collision-free path from the starting point to the target point according to the prior map specifically includes:

[0018] In the pre-established three-dimensional prior map, according to the actual rescue needs, multiple locations are randomly marked between the starting point and the target location where the accident scene is located, and the three-dimensional coordinate information corresponding to each location is obtained;

[0019] Sort the obtained multiple three-dimensional coordinates by serial numbers, and determine other serial numbers connected to each serial number;

[0020] Among all paths that meet the conditions, the path with the shortest distance is selected as the optimal collision-free path.

[0021] Furthermore, the selecting the path with the shortest distance as the optimal collision-free path from among all paths that meet the conditions specifically includes:

[0022] Use the global search algorithm to search all paths between the starting point and the end point;

[0023] Eliminate those with more steps;

[0024] Calculate the path length of each stage, accumulate and select the shortest path as the optimal collision-free path.

[0025] Furthermore, the receiving command center formulates a rescue strategy based on the real-time image data, and completes on-site investigation and material supply based on the rescue strategy, specifically including:

[0026] In the source code file of the voice command, add a control command containing the rescue strategy;

[0027] Receive voice commands and complete on-site investigation, material query, map information query and rescue material supply based on the rescue strategy in the voice commands.

[0028] Furthermore, the step of obtaining the environmental data and real-time image data of the accident scene further includes:

[0029] The acquired real-time image data is segmented into instances of people and dangerous objects.

[0030] Furthermore, the acquired real-time image data is segmented into instances of people and dangerous objects, etc., specifically including:

[0031] The acquired real-time image data is assumed to be an RGB image T and a depth map S of the image T;

[0032] Construct the feature matrix R(T, S) of each RoI area:

[0033] R(T,S)=[a(t)*b(s)]

[0034] Where a(t) and b(s) represent RGB and depth data respectively;

[0035] The gate function G1 is set by the following formula:

[0036]

[0037] In the formula, is the weight parameter;

[0038] Set the gate function G2 to back up the error area that was discarded by the error;

[0039] By providing the original feature map, the feature map is constructed using the following formula

[0040]

[0041] In the formula, is the weight parameter, The final channel of is the same as the depth map b(S);

[0042] The data information to be retained is filtered by the following formula:

[0043]

[0044] Combination and The output result P (i) .

[0045] The segmented image data is used to identify people and dangerous objects. The design uses a multi-classifier fusion prediction method. The multimodal prediction system includes two steps: First, using the same data set D train Train to obtain the model parameters of each prediction model. Next, each prediction method will use the parameters found in the first step to train the target data set D Predict-set In this step, for each trajectory in D and each prediction time horizon of interest ΔT, each predictor is used to produce an estimate of the future position, using The prediction results of each method are then used to calculate the weight of each predicted value, which in turn are used to define the rules that multiple forecasting systems use to select which forecasting methods should be used.

[0046] For a single classifier module, its training is first supervised learning to train the model parameters of each prediction method of the multimodal prediction system. Assuming there are n available prediction methods, for each method i = 1...n, the parameters of the method are defined as Z i ,make m i is the number of adjustable parameters of method i. The possible values ​​of the independent parameters are restricted to a given discrete set, with the goal of Identify a set of allocation parameters that minimizes the training error for each prediction method.

[0047] Because the accuracy of the forecast period may vary significantly as the forecast time horizon changes. Consider a set of discrete forecast time horizons of length k, defined as H. We denote each time horizon value as h P , where P = 1...k. Therefore, the goal is to find the most appropriate parameter value assignment Z for each time range in H i If the total training error of method i is given by time range h P , then the parameter assignment Z i and input data D is defined as E i , the parameter assignments that specify the discrete set that achieve the highest performance for this method and time horizon are defined as follows:

[0048]

[0049] Therefore, the training process will result in k×n set assignments Define d as a single trajectory contained in D, is a set of time steps of trajectory d for which a prediction can be produced by predictor i, with a training error E i It is defined as the average of these prediction errors:

[0050]

[0051] in is the prediction of predictor i, which is the parameter value Z i function.

[0052] For the output results of multiple classifiers, DS evidence theory, let Θ = {θ1, θ2, …, θ n} represents all possible sets in the rescue scene environment, θ i is the conclusion drawn by the system. The basic probability distribution function, confidence function and likelihood function in DS evidence theory are defined as follows:

[0053]

[0054]

[0055]

[0056] In formula (3), A is a hypothesis in the recognition framework. m(A) is a basic probability distribution function. In formula (4), Bel(A) is the sum of the basic distribution probabilities of all subsets A, and Pl(A) is the sum of the basic distribution probabilities of all intersecting subsets. Since BEL is independent of the unified recognition framework, they can form a subset 2 Θ→ Common protocol on [0, 1], any conflict can be quantified using Dempster’s combinatorial rule. For all And given n probability distribution functions m1, m2, ...m n , Dempster's law is calculated using equations (6) and (7):

[0057]

[0058]

[0059] Where K represents the conflict measure of the belief function.

[0060] The classification algorithms are integrated, multiple classifiers are trained, and the classifier model is obtained. Then, the information fusion technology based on DS evidence theory is used to integrate the results of multiple classifiers, and the Dempster combination rule is used to integrate the same items. Among the results after the operation, the one with the highest accuracy is selected as the output recognition result in the fused target information.

[0061] The present invention also provides a rescue robot-based on-site investigation and material supply system for implementing the above method, the system comprising:

[0062] A path planning unit, used to obtain a planned path to enter the rescue site based on a priori map, and send an operation instruction to the chassis control system based on the planned path, so that the rescue robot enters the accident site according to the planned path according to the operation instruction;

[0063] A rescue data acquisition unit is used to acquire environmental data and real-time image data at the accident site, and transmit the environmental data and real-time image data to the command center;

[0064] The rescue command receiving unit is used to receive the rescue strategy formulated by the command center based on real-time image data, and complete on-site investigation and material supply based on the rescue strategy.

[0065] The present invention also provides a terminal device, the device comprising: a data collector, a processor and a memory;

[0066] The data collector is used to collect data; the memory is used to store one or more program instructions; the processor is used to execute one or more program instructions to perform the method as described above.

[0067] The present invention also provides a computer-readable storage medium, wherein the computer storage medium contains one or more program instructions, and the one or more program instructions are used to execute the method as described above.

[0068] The method for on-site investigation and material supply based on a rescue robot provided by the present invention obtains a planned path to enter the rescue site based on a priori map, and sends an operation instruction to the chassis control system based on the planned path, so that the rescue robot enters the accident site according to the planned path according to the operation instruction; obtains environmental data and real-time image data of the accident site, and transmits the environmental data and real-time image data to the command center; receives the rescue strategy formulated by the command center according to the real-time image data, and completes on-site investigation and material supply based on the rescue strategy. In the process of implementing the rescue, when the supply of rescue equipment, materials and medicines is insufficient, the rescue robot is used for fixed-point transportation; at the same time, the rescue robot, as a special rescue equipment, can autonomously reach multiple designated locations, provide corresponding support, and cooperate with rescue personnel to complete the rescue mission. As a professional rescue material supply robot, for materials that need special preservation, the robot provides constant temperature, refrigeration, sealing and other conditions to ensure that the relevant items are foolproof. In addition, the rescue personnel can operate the robot through voice. If the specified items are required, the storage compartment will automatically open after the instruction is issued to ensure the immediate use of the items. Thereby solving the technical problem of low efficiency of on-site investigation and material supply of rescue robots in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference numerals are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0070] Figure 1 A flowchart of a specific implementation of the on-site investigation and material supply method based on the rescue robot provided by the present invention;

[0071] Figure 2 for Figure 1 A flowchart of the method shown in the usage scenario;

[0072] Figure 3 A structural block diagram of a specific implementation of the on-site investigation and material supply system based on the rescue robot provided by the present invention;

[0073] Figure 4 for Figure 3 An architectural diagram of the system shown;

[0074] Figure 5 This is a flowchart of the multiple classifier fusion method based on DS evidence theory. DETAILED DESCRIPTION

[0075] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0076] The present invention is based on the on-site investigation and material supply method of the rescue robot. It aims at the problems of chaotic site, difficult investigation and difficult material supply, and combines knowledge in the fields of artificial intelligence, simultaneous positioning and mapping technology, voice recognition and mechanical engineering to solve the problems of difficult and untimely rescue site investigation and material supply in accident rescue.

[0077] The method is based on a rescue robot, which mainly includes hardware such as 3D laser sensors, RGB-D image sensors, nine-axis attitude sensors, ultrasonic sensors, thermal imagers, field controllers, touch screens, smart lockers, chassis and aluminum alloy shells. In terms of structural design, a 3D laser sensor is mounted on the top of the robot to collect point cloud information in the surrounding environment in real time. An RGB-D image sensor and a thermal imager are mounted at the front and rear positions respectively. The RGB-D image sensor can be used to obtain deep visual information and real-time picture information in the environment. The thermal imager is used to detect possible overheating hazards of trapped personnel and equipment at the accident site. Based on the above information, the command center can formulate an effective rescue plan based on the relevant data collected.

[0078] In a specific embodiment, the present invention provides a method for on-site investigation and material supply based on a rescue robot, such as Figure 1 As shown, the method comprises the following steps:

[0079] S1: Acquire a planned path to enter the rescue site based on a priori map, and send an operation instruction to the chassis control system based on the planned path, so that the rescue robot enters the accident site according to the planned path according to the operation instruction.

[0080] Specifically, in step S1, when performing path planning through the prior map, an optimal collision-free path from the starting point to the target point is established according to the prior map; then, based on the optimal collision-free path, an operation instruction is sent to the chassis control system so that the rescue robot enters the accident scene according to the optimal collision-free path according to the operation instruction. At the same time, in the process of entering the accident scene, dynamic obstacles in the optimal collision-free path are avoided in real time.

[0081] The step of establishing an optimal collision-free path from a starting point to a target point according to a priori map specifically includes:

[0082] In the pre-established three-dimensional prior map, multiple locations are randomly marked between the starting point and the target location where the accident scene is located, and the corresponding three-dimensional coordinate information of each location is obtained;

[0083] Sort the obtained multiple three-dimensional coordinates by serial numbers, and determine other serial numbers connected to each serial number;

[0084] Among all the paths that meet the conditions, the path with the shortest distance is selected as the optimal collision-free path. Specifically, all paths between the starting point and the end point are searched using a global search algorithm; those with more steps are eliminated; the path length of each stage is calculated, and the shortest path is selected as the optimal collision-free path after accumulation.

[0085] That is to say, in the path planning and navigation system, first the global path planning is to establish an optimal collision-free path from the starting point to the target point according to the prior map, and then in the local path planning, the dynamic obstacles in the path are avoided in real time. Currently, the more commonly used global path planning algorithms are A* and Dijkstra algorithm, and the local path planning algorithms are TEB and DWA algorithm. Based on the use of Dijkstra algorithm and DWA algorithm, the present invention aims at the problem that the autonomous mobile robot can only realize autonomous navigation of two target points in the past, and realizes a method for autonomous navigation of multiple target points through the improvement of the path planning algorithm. The algorithm can set the end point of the robot and other multiple target points to be passed before reaching the end point. After the algorithm is optimized, the time taken by the robot to reach the end point after passing all the target points is the shortest.

[0086] The specific operation process can be divided into the following steps: In the established three-dimensional map, randomly mark multiple locations according to the specific situation. These locations can be rescue locations, forks and corners in the map, and try to distribute them evenly. Since each location has corresponding three-dimensional coordinate information, the target locations can be sorted according to the three-dimensional coordinates. Then determine the serial number connected to it in each serial number. Before starting the navigation function, first determine the end point and the rescue location that the robot is going to on the way. After that, the algorithm will automatically list all paths that meet the conditions and select the one with the shortest distance.

[0087] The specific implementation of the algorithm mainly includes: in the path planning stage, the global search algorithm is used to search all paths between the starting point and the end point, and the ones with more steps are eliminated. Then the path length of each stage is calculated, and the shortest path is selected as the final result after accumulation. In the navigation stage, the move_base function package in ROS is combined to improve its algorithm, establish a python type node, and use the callback mechanism in the subscriber. The callback function passes a pointer to the action message. After receiving the messages published by the cmd_vel, move_base / global_plan, move_base / goal and odom_rf2o topics, the callback function is called. At this time, according to the planned path, the robot passes through each serial number area in turn. After obtaining the Twist message published by the cmd_vel topic, the BaseController node controls the lower computer and drives the robot to move. If multiple locations need to be surveyed or rescue supplies are provided for multiple target locations, it can effectively save task time and improve rescue efficiency. When performing survey tasks, in order to increase the efficiency of task execution, it is set that every time the robot reaches the next numbered area 10m away, it will plan the path from the current position to the target point of the next numbered area, so that it can maintain a uniform speed between each numbered area to ensure rescue efficiency.

[0088] S2: Acquire environmental data and real-time image data at the accident site, and transmit the environmental data and real-time image data to the command center; specifically, add control instructions containing the rescue strategy to the source code file of the voice command; receive the voice command, and complete on-site investigation, material query, map information query and rescue material supply based on the rescue strategy in the voice command.

[0089] In actual use, the robot's actions can be controlled through voice recognition. For example, the iFlytek voice SDK package can be modified accordingly, and corresponding control instructions can be added to its source code file according to actual use. By using the voice function, rescuers can directly control the robot, query supplies, map information, and take rescue supplies from the storage compartment.

[0090] S3: Receive the rescue strategy formulated by the command center based on real-time image data, and complete on-site investigation and material supply based on the rescue strategy.

[0091] In order to improve the picture quality, the environmental data and real-time picture data of the accident scene are obtained, and then the following is further included:

[0092] The acquired real-time image data is segmented into instances of people and dangerous objects.

[0093] The real-time image data is segmented into instances of people and dangerous objects, etc., specifically including:

[0094] The acquired real-time image data is assumed to be an RGB image T and a depth map S of the image T;

[0095] Construct the feature matrix R(T, S) of each Roi area:

[0096] R(T,S)=[a(t)*b(s)]

[0097] Where a(t) and b(s) represent RGB and depth data respectively;

[0098] The gate function G1 is set by the following formula:

[0099]

[0100] In the formula, is the weight parameter;

[0101] Set the gate function G2 to back up the error area that was discarded by the error;

[0102] By providing the original feature map, the feature map is constructed using the following formula

[0103]

[0104] In the formula, is the weight parameter, The final channel of is the same as the depth map b(S);

[0105] The data information to be retained is filtered by the following formula:

[0106]

[0107] Combination and The output result P (i) .

[0108] That is to say, during object recognition and segmentation, the present invention uses the RGBD image data obtained by the depth vision sensor to perform instance segmentation of people and dangerous objects. Its use effect is better indoors, but in special scenes such as accident scenes, the depth image will be blurred, and the difference between two adjacent instance objects is still small. Even after depth normalization, the instance segmentation effect will be poor. Therefore, considering the above problems, we use a residual compensation mechanism and integrate it into the current more common instance segmentation framework Mask RCNN in an end-to-end manner.

[0109] Specifically, the input of the structure is the RGB image T and the corresponding depth map S, and the feature matrix R(T, S) of each RoI area is first constructed.

[0110] R(T,S)=[a(t)*b(s)]

[0111] In the formula, a(t) and b(s) represent RGB and depth data respectively. Depth data can be used to enhance the instantiation effect. Since the depth vision sensor has a certain range when processing depth data, the original depth information may contain noise, which will affect the final prediction effect. A filtering process is required to optimize the data information in the depth map. Therefore, the gate function G1 is set to extract judgment information such as instance boundaries in the depth map and suppress the noise area.

[0112]

[0113] In the formula, is the weight parameter. However, in this filtering process, there may be some important areas that are mistakenly discarded by G1. Therefore, a compensation mechanism is used so that some important areas that are mistakenly discarded by G1 will have the opportunity to be backed up in the next unit. The gate function G2 is set to back up the mistakenly discarded erroneous areas, and 1-G1 is used to filter out the important areas. G2 is similar to G1, except that the relevant weight parameters are different. A feature map is constructed by providing the original feature map. To achieve information integrity.

[0114]

[0115] In the formula, is the weight parameter, The final channel of is the same as the depth map b(S). Next, the data information to be retained is filtered.

[0116]

[0117] Finally, combined and The output result P (i) .

[0118] The segmented image data is used to identify people and dangerous objects. The design uses a multi-classifier fusion prediction method. The multimodal prediction system includes two steps: First, using the same data set D train Train to obtain the model parameters of each prediction model. Next, each prediction method will use the parameters found in the first step to train the target data set D Predict-setIn this step, for each trajectory in D and each prediction time horizon of interest ΔT, each predictor is used to produce an estimate of the future position, using The prediction results of each method are then used to calculate the weight of each predicted value, which in turn are used to define the rules that multiple forecasting systems use to select which forecasting methods should be used.

[0119] For a single classifier module, its training is first supervised learning to train the model parameters of each prediction method of the multimodal prediction system. Assuming there are n available prediction methods, for each method i = 1...n, the parameters of the method are defined as Z i ,make m i is the number of adjustable parameters of method i. The possible values ​​of the independent parameters are restricted to a given discrete set, with the goal of Identify a set of allocation parameters that minimizes the training error for each prediction method.

[0120] Because the accuracy of the forecast period may vary significantly as the forecast time horizon changes. Consider a set of discrete forecast time horizons of length k, defined as H. We denote each time horizon value as h P , where P = 1...k. Therefore, the goal is to find the most appropriate parameter value assignment Z for each time range in H i If the total training error of method i is given by time range h P , then the parameter assignment Z i and input data D is defined as E i , the parameter assignments that specify the discrete set that achieve the highest performance for this method and time horizon are defined as follows:

[0121]

[0122] Therefore, the training process will result in k×n set assignments Define d as a single trajectory contained in D, is a set of time steps of trajectory d for which a prediction can be produced by predictor i, with a training error E i It is defined as the average of these prediction errors:

[0123]

[0124] in is the prediction of predictor i, which is the parameter value Z i function.

[0125] For the output results of multiple classifiers, DS evidence theory, let Θ = {θ1, θ2, …, θ n} represents all possible sets in the rescue scene environment, θ i is the conclusion drawn by the system. The basic probability distribution function, confidence function and likelihood function in DS evidence theory are defined as follows:

[0126]

[0127]

[0128]

[0129] In formula (3), A is a hypothesis in the recognition framework. m(A) is a basic probability distribution function. In formula (4), Bel(A) is the sum of the basic distribution probabilities of all subsets A, and Pl(A) is the sum of the basic distribution probabilities of all intersecting subsets. Since BEL is independent of the unified recognition framework, they can form a subset 2 Θ → Common protocol on [0, 1], any conflict can be quantified using Dempster’s combinatorial rule. For all And given n probability distribution functions m1, m2, ...m n , Dempster's law is calculated using equations (6) and (7):

[0130]

[0131]

[0132] Where K represents the conflict measure of the belief function.

[0133] The classification algorithms are integrated, multiple classifiers are trained, and the classifier model is obtained. Then, the information fusion technology based on DS evidence theory is used to integrate the results of multiple classifiers, and the Dempster combination rule is used to integrate the same items. Among the results after the operation, the one with the highest accuracy is selected as the output recognition result in the fused target information.

[0134] The following describes the overall solution in combination with actual rescue scenarios. Figure 2 As shown in the figure, when a fire, explosion or other accident occurs, before implementing rescue, the first thing to do is to survey the internal environment of the accident site, detect the source of danger, search and rescue the trapped people, etc., so as to reasonably arrange the entry of rescue personnel and rescue the trapped people to evacuate safely. Before the rescue personnel enter, the robot first takes the place of the rescue personnel to initially understand the accident situation. The command center remotely controls the robot to enter the scene and establish a three-dimensional map of the accident site. The three-dimensional map is uploaded to the command center, through which the command center can roughly grasp the overall situation of the accident.

[0135] At the same time, the RGB-D image sensors at the front and back ends of the robot are also collecting real-time image information at the scene. The real-time images at the scene are segmented using deep learning algorithms to identify buildings, vehicles, personnel, flames, and various equipment at the scene. Dangerous objects such as flammable and explosive materials are marked as dangerous and the danger level is determined. The thermal imager can display the thermal image of the scene in real time. With the integrated three-dimensional map information, the command center can have a detailed grasp of the overall situation of the accident scene.

[0136] During this process, the robot can effectively find trapped people. When trapped people are detected in the real-time picture, the command center can immediately arrange rescue personnel to rescue them. For trapped people who are blocked, they can be found in time through thermal imaging. In addition, some equipment that may have overheating risks can also be found and removed in time.

[0137] Using the established three-dimensional map, the robot can realize autonomous navigation according to the map.

[0138] In terms of the evaluation of the hazard index of dangerous objects, the design uses a hazard index evaluation method based on the category and distance of dangerous objects, and evaluates the hazard index based on the identification results of dangerous objects and the depth information of the visual sensor. When keeping the same distance from the robot, different dangerous objects will present different hazard indexes. Assume that when performing the task, the coordinate position of the center point (TCP) of the robot in the global coordinate system is The speed is v tcp , braking time is T b If the coordinate value of the dangerous object is The corresponding speed is v danger , then the danger threshold can be defined as formula (8), which is also the minimum distance between the robot and the dangerous object.

[0139]

[0140] To ensure safety, the robot should satisfy equation (9) in the next time step.

[0141] |H o -R t |>D t-d (9)

[0142] Equation (9) defines the danger zone in the rescue area. When there is no danger in this danger zone, the robot can run at maximum power. Obviously, D t-d It changes with the speed of the robot. Define the position of the robot in the global map as R m , the speed is v m, combined with the definition of the danger threshold, the danger threshold can be calculated by formula (10):

[0143]

[0144] Calculate the danger index of dangerous goods in the rescue area. According to the above definition, the danger index of dangerous goods can be expressed as formula (11).

[0145]

[0146] According to the danger index, the robot running speed v c The control can be expressed as:

[0147]

[0148] Among them, f(s danger -s slow ) represents the relationship between the danger index and the robot speed in equations (8), (9) and (10).

[0149] During the real-time rescue process of rescuers, if an emergency occurs, rescue materials need to be arranged immediately. At this moment, the command center only needs to determine the location of the rescuers in the three-dimensional map, and the robot can automatically go to the target area. If obstacles are encountered during the process, the robot can avoid them autonomously. If problems such as road congestion occur, the robot can re-plan the path to ensure the smooth completion of the rescue mission. In special circumstances, the command center can arrange the robot to go to multiple target areas at one time. Using the multi-point positioning and navigation algorithm, the robot can ensure that the path is the shortest and the time spent is the least when reaching the target area.

[0150] After the robot reaches the target area, the rescuers can communicate with the robot through voice. After raising the demand for materials, the storage compartment will automatically open for the rescuers to quickly take them out, and the specific item information will be displayed on the touch screen. In addition, the rescuers can also check their location through the touch screen. When they need to go to the next rescue location or evacuate the scene, they can operate the robot to help them quickly go to the next rescue location or evacuate. If the rescuers do not understand the specific operation method. The command center can remotely operate the robot to help the rescuers complete the relevant operations.

[0151] After the mission is completed, the robot can autonomously return to the command center or perform related patrol tasks according to the specific situation. When performing patrol tasks, the command center marks multiple target areas that the robot must pass through in the three-dimensional map, and then the robot can autonomously plan an optimal path that passes through all target areas. During the patrol process, using the robot's object recognition function, the command center can observe various things at the accident scene in real time, and promptly feedback the specific conditions of undiscovered trapped persons and dangerous items to the command center, and the command center can take immediate action based on the feedback information.

[0152] In the above specific implementation, the on-site investigation and material supply method based on the rescue robot provided by the present invention obtains the planned path to enter the rescue site based on the prior map, and sends an operation instruction to the chassis control system based on the planned path, so that the rescue robot enters the accident site according to the planned path according to the operation instruction; obtains the environmental data and real-time picture data of the accident site, and transmits the environmental data and real-time picture data to the command center; receives the rescue strategy formulated by the command center based on the real-time picture data, and completes the on-site investigation and material supply based on the rescue strategy. In the process of implementing the rescue, when the supply of rescue equipment, materials and medicines is insufficient, the rescue robot is used for fixed-point transportation; at the same time, the rescue robot, as a special rescue equipment, can autonomously reach multiple designated locations, provide corresponding support, and cooperate with rescue personnel to complete the rescue mission. As a professional rescue material supply robot, for materials that need special preservation, the robot provides constant temperature, refrigeration, sealing and other conditions to ensure the safety of related items. And the rescue personnel can operate the robot through voice. If the specified items are required, the storage compartment will automatically open after the instruction is issued to ensure the immediate use of the items. This solves the technical problem of low efficiency of on-site investigation and material supply of rescue robots in the prior art.

[0153] In addition to the above method, the present invention also provides a rescue robot-based on-site investigation and material supply system for implementing the above method. In a specific implementation, Figure 3 As shown, the system comprises:

[0154] The path planning unit 100 is used to obtain a planned path to enter the rescue site based on the prior map, and send an operation instruction to the chassis control system based on the planned path, so that the rescue robot enters the accident site according to the planned path according to the operation instruction;

[0155] The rescue data acquisition unit 200 is used to acquire environmental data and real-time image data at the accident site, perceive the environmental data at the disaster site in real time through a multi-classifier fusion algorithm, and transmit the environmental data and real-time image data to the command center;

[0156] The rescue instruction receiving unit 300 is used to receive the rescue strategy formulated by the command center according to the real-time image data, and complete the on-site investigation and material supply based on the rescue strategy.

[0157] In actual usage scenarios, such as Figure 4 As shown, the system includes a field controller (host computer) in terms of hardware construction, an RGB-D image sensor, a 3D laser sensor, a thermal imager, an ultrasonic sensor, and a nine-axis attitude sensor that communicate with the field controller to realize data transmission. These sensors send their detected data to the field controller. The field controller integrates and transmits these data to the command center, receives control instructions fed back by the command center, and controls the single-chip control board (lower computer) based on the control instructions, and then drives motor 1, motor 2, motor 3, and motor 4 respectively through the driver board (including multiple, controlling motor 1, motor 2, motor 3, and motor 4 respectively).

[0158] In the above specific implementation, the on-site investigation and material supply system based on the rescue robot provided by the present invention obtains the planned path to enter the rescue site based on the prior map, and sends an operation instruction to the chassis control system based on the planned path, so that the rescue robot enters the accident site according to the planned path according to the operation instruction; obtains the environmental data and real-time picture data of the accident site, and transmits the environmental data and real-time picture data to the command center; receives the rescue strategy formulated by the command center based on the real-time picture data, and completes the on-site investigation and material supply based on the rescue strategy. In the process of implementing the rescue, when the supply of rescue equipment, materials and medicines is insufficient, the rescue robot is used for fixed-point transportation; at the same time, the rescue robot, as a special rescue equipment, can autonomously reach multiple designated locations, provide corresponding support, and cooperate with rescue personnel to complete the rescue mission. As a professional rescue material supply robot, for materials that need special preservation, the robot provides constant temperature, refrigeration, sealing and other conditions to ensure the safety of related items. And the rescue personnel can operate the robot by voice. If the specified items are required, the storage compartment will automatically open after the instruction is issued to ensure the immediate use of the items. This solves the technical problem of low efficiency of on-site investigation and material supply of rescue robots in the prior art.

[0159] The present invention also provides a terminal device, the device comprising: a data collector, a processor and a memory;

[0160] The data collector is used to collect data; the memory is used to store one or more program instructions; the processor is used to execute one or more program instructions to perform the method as described above.

[0161] Corresponding to the above embodiment, an embodiment of the present invention further provides a computer storage medium, which includes one or more program instructions. The one or more program instructions are used to be executed by a binocular camera depth calibration system as described above.

[0162] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0163] Although the terms first, second, third, etc. can be used in the text to describe multiple elements, components, regions, layers and / or sections, these elements, components, regions, layers and / or sections should not be limited by these terms. These terms can only be used to distinguish an element, component, region, layer or section from another region, layer or section. Unless the context clearly indicates, terms such as "first", "second" and other numerical terms do not imply order or sequence when used in the text. Therefore, the first element, component, region, layer or section discussed below can be referred to as the second element, component, region, layer or section without departing from the teaching of the example embodiments.

[0164] In the embodiment of the present invention, the processor may be an integrated circuit chip having the signal processing capability. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0165] The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware.

[0166] The storage medium may be a memory, which may be, for example, a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory.

[0167] Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.

[0168] The volatile memory may be a random access memory (RAM) which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus DRAM (DRRAM).

[0169] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0170] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention can be implemented using a combination of hardware and software. When software is used, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. Storage media can be any available media that can be accessed by general or special-purpose computers.

[0171] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for on-site investigation and material supply based on a rescue robot, characterized in that: The method comprises: Acquire a planned path to the rescue site based on the prior map, and send an operation instruction to the chassis control system based on the planned path, so that the rescue robot enters the accident site according to the planned path according to the operation instruction; Obtain environmental data and real-time image data from the accident site, use the on-site edge computing unit to run a multimodal prediction algorithm, and complete the on-site environmental survey; then, based on the identification results of dangerous objects and the depth information of the visual sensor, make a dangerous object hazard index evaluation, display it in the real-time image, and transmit the environmental data and real-time image data to the command center; The command center formulates rescue strategies based on real-time image data and completes on-site surveys and material supply based on the rescue strategies; The on-site robot runs the path planning algorithm according to the rescue strategy, autonomously navigates to the target location to supply materials, and adopts different travel speeds according to different danger indexes. When there is a high risk, the path can be replanned; The step of obtaining environmental data and real-time image data of the accident scene further includes: The acquired real-time image data is segmented into instances of people and dangerous objects, including: The acquired real-time image data is assumed to be an RGB image T and a depth map S of the image T; Construct the feature matrix R(T,S) of each RoI area: R(T,S)=[a(t)*b(s)] Where a(t) and b(s) represent RGB and depth data respectively; The gate function G1 is set by the following formula: In the formula, is the weight parameter; Set the gate function G2 to back up the error area that was discarded by the error; By providing the original feature map, the feature map is constructed using the following formula In the formula, is the weight parameter, The final channel of is the same as the depth map b(S); The data information to be retained is filtered by the following formula: Combination and The output result P (i) .

2. The method according to claim 1, characterized in that: The step of acquiring a planned path to a rescue site based on a priori maps, and sending an operation instruction to a chassis control system based on the planned path, so that the rescue robot enters the accident site according to the planned path according to the operation instruction, specifically includes: Establish an optimal collision-free path from the starting point to the target point based on the prior map; Sending an operation instruction to the chassis control system based on the optimal collision-free path, so that the rescue robot enters the accident scene along the optimal collision-free path according to the operation instruction; At the same time, in the process of entering the accident scene, the dynamic obstacles in the optimal collision-free path are avoided in real time. The obstacle avoidance method based on the danger index evaluation of the dangerous object category and distance is adopted in the obstacle avoidance process; the danger index evaluation method based on the dangerous object category and distance is specifically to evaluate the danger index according to the recognition result of the dangerous object and the depth information of the visual sensor; when performing the task, the coordinate position of the center point of the robot in the global coordinate system is set to The speed is v tcp , braking time is t b ; If the coordinate value of the dangerous object is The corresponding speed is v danger , then the danger threshold can be defined as formula (8), which is also the minimum distance between the robot and the dangerous object: To ensure safety, the robot should satisfy equation (9) in the next time step: |H o -R t |>D t-d (9) Formula (9) defines the dangerous area in the rescue area. When there is no dangerous object in this dangerous area, the robot can run at the maximum power, D t-d The robot's position in the global map is R. m , the speed is v m , combined with the definition of the danger threshold, the danger threshold can be calculated by formula (10): Where i represents the i-th dangerous item; Calculate the danger index of dangerous goods in the rescue area. The danger index of dangerous goods can be expressed as formula (11): According to the danger index, the robot running speed v c The control can be expressed as: Among them, f(s danger -s slow ) represents the relationship between the danger index and the robot speed in equations (8), (9) and (10).

3. The method according to claim 2, characterized in that The step of establishing an optimal collision-free path from the starting point to the target point according to the prior map specifically includes: In the pre-established three-dimensional prior map, according to the actual rescue needs, multiple locations are randomly marked between the starting point and the target location where the accident scene is located, and the three-dimensional coordinate information corresponding to each location is obtained; Sort the obtained multiple three-dimensional coordinates by serial numbers, and determine other serial numbers connected to each serial number; Among all paths that meet the conditions, a path with the shortest distance is selected as the optimal collision-free path.

4. The method according to claim 3, characterized in that The step of selecting the path with the shortest distance from all paths that meet the conditions as the optimal collision-free path specifically includes: Use the global search algorithm to search all paths between the starting point and the end point; Eliminate those with more steps; Calculate the path length of each stage, accumulate and select the shortest path as the optimal collision-free path.

5. The method according to claim 1, characterized in that The command center formulates rescue strategies based on real-time image data and completes on-site surveys and material supply based on the rescue strategies, including: In the source code file of the voice command, add a control command containing the rescue strategy; Receive voice commands and complete on-site investigation, material query, map information query and rescue material supply based on the rescue strategy in the voice commands.

6. The method according to claim 1, characterized in that Using the segmented image data to identify people and dangerous objects, the multimodal prediction system includes two steps: first, using the same dataset D train Train to obtain the model parameters of each prediction model; next, each prediction method will use the parameters found in the first step to train the target data set D Predict-set Make predictions. In this step, for the target dataset D Predict-set For each trajectory in and each prediction time horizon of interest ΔT, each predictor is used to produce an estimate of the future position, The prediction results of each method are then used to calculate the weight of each prediction value, which in turn are used to define the rules used by multiple prediction systems to select which prediction methods should be used; For a single classifier module, its training is first supervised learning to train the model parameters of each prediction method of the multimodal prediction system; assuming that there are n available prediction methods, for each method i = 1, ..., n, the parameters of the method are defined as Z i ,make m i is the number of adjustable parameters of method i; The goal is to restrict the possible values ​​of the independent parameters to a given discrete set, with the goal of Identify a set of allocation parameters that minimize the training error for each prediction method; Because the accuracy of the forecast period may vary significantly with the change of the forecast time horizon; consider a set of discrete forecast time horizons with length k, defined as H; we denote each time horizon value as h P , where P = 1, ..., k; therefore, the goal is to find the most appropriate parameter value assignment Z for each time range in H i ; If the total training error of method i is given by time horizon h P , then the parameter assignment Z i and input data D is defined as E i , the parameter assignments that specify the discrete set that achieve the highest performance for this method and time horizon are defined as follows: Therefore, the training process will result in k×n set assignments Define d as a single trajectory contained in D, is a set of time steps of trajectory d for which a prediction can be produced by predictor i, with a training error E i It is defined as the average of these prediction errors: in is the prediction of predictor i, which is the parameter value Z i Function of The classification algorithms are integrated, multiple classifiers are trained, and the classifier model and the prediction of predictor i are obtained; then the information fusion technology based on DS evidence theory is used to integrate the results of multiple predictors and the Dempster combination rule is used to integrate the same items. Among the results after the operation, the one with the highest accuracy is selected as the output recognition result in the fused target information; For the output results of multiple classifiers, DS evidence theory, let Θ = {θ1,θ2,…,θ n } represents all possible sets in the rescue scene environment, θ i is the conclusion drawn by the system; the basic probability distribution function, confidence function and likelihood function in DS evidence theory, these three functions are defined as follows: In formula (3), A is a hypothesis in the recognition framework. m(A) is a basic probability distribution function; in formula (4), Bel(A) is the sum of the basic distribution probabilities of all subsets A, and Pl(A) is the sum of the basic distribution probabilities of all intersecting subsets; since BEL is independent of the unified recognition framework, they can form a subset 2 Θ → Common protocol on [0,1], any conflict can be quantified using Dempster's combinatorial rule; for all And given n probability distribution functions m1,m2,…m n , Dempster's law is calculated using equations (6) and (7): In the formula, K represents the conflict measure of the belief function; When objects are identified in the on-site environment, the expert database will give the dangerousness level D of the objects. level (0-1).

7. A rescue robot-based on-site investigation and material supply system, used to implement the method according to any one of claims 1 to 6, characterized in that: The system comprises: A rescue data acquisition unit is used to acquire environmental data and real-time image data at the accident site, and transmit the environmental data and real-time image data to the command center; The rescue command receiving unit is used to receive the rescue strategy formulated by the command center based on real-time image data, and complete on-site investigation and material supply based on the rescue strategy.

8. A terminal device, characterized in that: The terminal device includes: a data collector, a processor and a memory; The data collector is used to collect data; the memory is used to store one or more program instructions; the processor is used to execute one or more program instructions to execute the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer storage medium includes one or more program instructions, and the one or more program instructions are used to execute the method according to any one of claims 1-6.

Images