A three-dimensional reconstruction and path planning method, device, equipment and storage medium

Abstract

The embodiments of the present application relate to the field of multimodal perception technology of robots, and in particular to a three-dimensional reconstruction and path planning method, device, and storage medium. The method comprises: constructing a three-dimensional perception array based on electronic skin; then, obtaining a three-dimensional grid space and extracting a contour vector from the grid space; performing three-dimensional decomposition on the contour vector to obtain contour data and smoothing it; next, synthesizing the contour data into a three-dimensional contour and extracting time series data; obtaining the three-dimensional spatial information of the object based on the time series data; based on the relative position of the robot, correcting the three-dimensional spatial information and obtaining the current global environment feature vector; finally, obtaining path planning information based on the current global environment feature vector and a reinforcement learning algorithm. The three-dimensional reconstruction and path planning method provided by the embodiments of the present application can improve the response speed, realize real-time and accurate three-dimensional reconstruction of objects and path planning in unknown complex environments.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of robot multimodal perception technology, and in particular to a three-dimensional reconstruction and path planning method, device, equipment, and storage medium. Background Art

[0002] The development of modern sensor technology and artificial intelligence has put forward higher requirements for perception devices. A robot with a single sensor can only obtain a single external information. This requires the integration of a large number of comprehensive sensors of various types and directions on the robot, and the fusion of external data collected by multiple sensors to obtain more accurate environmental information, thereby establishing a complete external environment model.

[0003] In structured environments, most existing sensor fusion methods use vision devices or lidar to reconstruct and perceive the surrounding environment in three dimensions, effectively capturing features of the external environment. However, in unstructured environments, especially those with complex, confined spaces or those subject to light interference, traditional vision- and lidar-based methods cannot accurately perceive the external environment, resulting in visual and radar blind spots. The placement of vision devices is limited by the robot's three-dimensional structure, making it impossible to fully perceive the robot's surroundings. The high cost and large size of lidar also restrict its application in confined spaces. The accuracy of its three-dimensional modeling cannot be guaranteed in complex lighting conditions or in rain, fog, or dust. Summary of the Invention

[0004] The embodiments of the present application provide a three-dimensional reconstruction and path planning method, device, equipment and storage medium to improve response speed, realize real-time and accurate three-dimensional reconstruction of objects and path planning in unknown complex environments.

[0005] To solve the above technical problems, in the first aspect, an embodiment of the present application provides a three-dimensional reconstruction and path planning method, including: constructing a three-dimensional sensing array of infrared proximity sensors based on electronic skin; obtaining a three-dimensional grid space based on the three-dimensional sensing array, and extracting a contour vector from the grid space; performing three-dimensional decomposition on the contour vector to obtain contour data, and smoothing the contour data; synthesizing the smoothed contour data into a three-dimensional contour, and extracting time series data of the array sensor corresponding to the three-dimensional contour; obtaining three-dimensional spatial information of the object based on the time series data; correcting the three-dimensional spatial information based on the relative posture of the robot, and obtaining the current global environment feature vector from the corrected three-dimensional spatial information; obtaining path planning information based on the current global environment feature vector and a reinforcement learning algorithm.

[0006] In some exemplary embodiments, the grid space is a quantifiable and discretizable space formed by intersection nodes of detection ranges of multiple infrared proximity sensors in a three-dimensional sensing array.

[0007] In some exemplary embodiments, smoothing the contour data includes: performing a median absolute deviation outlier method on the contour data to obtain contour data with discrete values removed; and filtering the contour data with discrete values removed.

[0008] In some exemplary embodiments, the time series data is used to represent a curve of distance data sensed by each infrared proximity sensor over a fixed period of time.

[0009] In some exemplary embodiments, the three-dimensional spatial information includes the size and center coordinates of the object in the three-dimensional space with the electronic skin array as the spatial origin, the object's moving speed, the object's moving direction, and the object's moving acceleration.

[0010] In some exemplary embodiments, the relative posture of the robot is obtained through the inertial sensor of the electronic skin, including: sampling the robot acceleration and angular velocity data by the inertial sensor; based on the robot acceleration and the angular velocity data, the relative posture of the robot is calculated by the extended Kalman filter algorithm.

[0011] In some exemplary embodiments, the reinforcement learning algorithm adopts a deep deterministic policy gradient algorithm based on a value function and a policy function, takes the global dynamic environment as the network input through an end-to-end learning method, and directly outputs the robot's path planning information.

[0012] On the second aspect, the embodiment of the present application also provides a three-dimensional reconstruction and path planning device, including a multimodal perception module and a data processing module; the multimodal perception module includes a temperature acquisition module, a humidity acquisition module, a pressure acquisition module, a ranging module, an acceleration acquisition module and an angular velocity acquisition module; the multimodal perception module is used to realize data acquisition, and the data includes collected temperature, humidity, pressure, obstacle distance, robot surface acceleration and robot surface angular velocity; the data processing module is used to process the data collected by the multimodal perception module to obtain three-dimensional spatial information of the object; and based on the relative posture of the robot, the three-dimensional spatial information is corrected, and the current global environment feature vector is obtained from the corrected three-dimensional spatial information; based on the current global environment feature vector and the reinforcement learning algorithm, path planning information is obtained.

[0013] In addition, the present application also provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned three-dimensional reconstruction and path planning method.

[0014] In addition, the present application also provides a computer-readable storage medium storing a computer program, characterized in that when the computer program is executed by a processor, the above-mentioned three-dimensional reconstruction and path planning method is implemented.

[0015] The technical solution provided by the embodiments of the present application has at least the following advantages:

[0016] An embodiment of the present application provides a three-dimensional reconstruction and path planning method, apparatus, device and storage medium, the method comprising: first, constructing a three-dimensional sensing array of infrared proximity sensors based on electronic skin; then, based on the three-dimensional sensing array, obtaining a three-dimensional grid space, and extracting a contour vector from the grid space; next, performing three-dimensional decomposition on the contour vector to obtain contour data, and smoothing the contour data; next, synthesizing the smoothed contour data into a three-dimensional contour, and extracting time series data of the array sensor corresponding to the three-dimensional contour; then, based on the time series data, obtaining three-dimensional spatial information of the object; next, correcting the three-dimensional spatial information based on the relative position of the robot, and obtaining a current global environment feature vector from the corrected three-dimensional spatial information; finally, obtaining path planning information based on the current global environment feature vector and a reinforcement learning algorithm.

[0017] The three-dimensional reconstruction and path planning method provided by the present application can sense physical quantities such as temperature, distance, acceleration, and pressure through the use of multimodal electronic skin integrated sensors. It is small in size and can be networked and expanded over a large area, so as to better adhere to the complex three-dimensional surfaces and joints of the robot. The multimodal electronic skin used in the present application can achieve full coverage perception of the external environment and can obtain complete environmental characteristics in confined environments and unknown complex environments after disasters. In addition, the data processing method used in the present application can smooth and filter the signals collected by the infrared proximity sensor array, and has a fast calculation speed, and can obtain accurate three-dimensional surface contours of objects, meeting the needs of fast real-time performance. At the same time, the reinforcement learning algorithm used in the present application can directly output the input global environment feature vector as the robot's path planning information, realizing a direct closed loop of the system's environmental perception and decision-making control, thereby realizing path planning for the entire environment in unknown, complex, and confined environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] One or more embodiments are exemplarily described by the pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Unless otherwise stated, the pictures in the drawings do not constitute proportional limitations.

[0019] Figure 1 A flowchart of a three-dimensional reconstruction and path planning method provided in one embodiment of the present application;

[0020] Figure 2 This is a diagram of the electronic skin infrared proximity sensor array according to an embodiment of the present application;

[0021] Figure 3 This is a schematic diagram of a multimodal electronic skin sensing an object in an embodiment of the present application;

[0022] Figure 4 This is a transverse cross-sectional view of an intersection node of the sensing range of the infrared proximity sensor array in an embodiment of the present application;

[0023] Figure 5 This is a time series data graph generated by the infrared proximity sensor array in the embodiment of the present application;

[0024] Figure 6 This is a transverse cross-sectional view of the outline of an object sensed by the infrared proximity sensor array in an embodiment of the present application;

[0025] Figure 7 This is an X-axis exploded view of the three-dimensional profile in the embodiment of the present application;

[0026] Figure 8 This is a Y-axis exploded view of the three-dimensional profile in the embodiment of the present application;

[0027] Figure 9 This is a Z-axis exploded view of the three-dimensional profile in the embodiment of the present application;

[0028] Figure 10 The three-dimensional contour image of the object being measured after data processing in the embodiment of the present application;

[0029] Figure 11 The raw data collected by a single sensor in the infrared proximity sensor array in the embodiment of the present application;

[0030] Figure 12 The data is the result of MAD processing on the raw data collected by a single sensor in the embodiment of the present application;

[0031] Figure 13 The data after the MAD processing is subjected to IIR filtering in the embodiment of the present application;

[0032] Figure 14 A block diagram of the reinforcement learning algorithm used in the embodiments of this application;

[0033] Figure 15 A schematic structural diagram of a three-dimensional reconstruction and path planning device provided in one embodiment of the present application;

[0034] Figure 16 A schematic structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0035] As can be seen from the background technology, the current existing three-dimensional reconstruction and path planning methods cannot achieve accurate perception of the external environment in unstructured environments, and there are visual and radar blind spots. The accuracy of three-dimensional modeling cannot be guaranteed in complex lighting environments and rain, fog and dust environments.

[0036] Robot electronic skin (E-Skin) is one of the key technologies for realizing autonomous intelligence of robots. Electronic skin integrates a large number of sensors with its extremely small volume. The electronic skin integrated on the complex three-dimensional surface or movable joints of the robot can enable the robot to obtain multi-dimensional information perception capabilities of its own surface and external environment, and can make intelligent decisions based on the perceived information. A related art discloses a robot sensor that integrates a TOF (time of flight) sensor and a capacitive proximity sensor. The sensor is placed on multiple surfaces at the tip joints of a hinged robot. The distance of the object is detected by the TOF sensor, and the capacitance detected by the capacitive proximity sensor is used to identify the type of object, and the drive of the joint robot is controlled according to the identified object type. However, the robot sensor cannot recognize the outline of the surrounding objects, does not perform three-dimensional reconstruction of the surrounding environment to perceive the object type, and lacks a high-response, high-precision three-dimensional reconstruction environment perception method.

[0037] To solve the above technical problems, an embodiment of the present application provides a three-dimensional reconstruction and path planning method, comprising: first, constructing a three-dimensional sensing array of infrared proximity sensors based on electronic skin; then, based on the three-dimensional sensing array, obtaining a three-dimensional grid space and extracting a contour vector from the grid space; next, performing three-dimensional decomposition on the contour vector to obtain contour data, and smoothing the contour data; then, synthesizing the smoothed contour data into a three-dimensional contour, and extracting the time series data of the array sensor corresponding to the three-dimensional contour; next, obtaining the three-dimensional spatial information of the object based on the time series data; then, correcting the three-dimensional spatial information based on the relative position of the robot, and obtaining the current global environment feature vector from the corrected three-dimensional spatial information; finally, obtaining path planning information based on the current global environment feature vector and a reinforcement learning algorithm. An embodiment of the present application provides a three-dimensional reconstruction and path planning method, which improves the response speed and realizes real-time and accurate three-dimensional reconstruction of objects and path planning in unknown complex environments.

[0038] The following detailed description of the various embodiments of the present application is provided in conjunction with the accompanying drawings. However, those skilled in the art will appreciate that many technical details are provided in the various embodiments of the present application to facilitate a better understanding of the present application. However, even without these technical details and the various variations and modifications based on the following embodiments, the technical solutions claimed in the present application can still be implemented.

[0039] See Figure 1, the embodiment of the present application provides a three-dimensional reconstruction and path planning method, comprising the following steps:

[0040] Step S1: construct a three-dimensional sensing array of infrared proximity sensors based on electronic skin.

[0041] Step S2: Based on the three-dimensional sensing array, a three-dimensional grid space is obtained, and a contour vector is extracted from the grid space.

[0042] Step S3: Decompose the contour vector in three dimensions to obtain contour data, and perform smoothing on the contour data.

[0043] Step S4: synthesize the smoothed contour data into a three-dimensional contour, and extract the time series data of the array sensor corresponding to the three-dimensional contour.

[0044] Step S5: Obtain three-dimensional spatial information of the object based on the time series data.

[0045] Step S6: Based on the relative position of the robot, the three-dimensional space information is corrected, and the current global environment feature vector is obtained from the corrected three-dimensional space information.

[0046] Step S7: Obtain path planning information based on the current global environment feature vector and the reinforcement learning algorithm.

[0047] Specifically, the electronic skin in step S1 of this embodiment utilizes a multimodal electronic skin integrated core sensor. This system integrates four core sensors capable of sensing temperature, distance, acceleration, and pressure. It is compact and capable of large-scale networking and expansion, allowing for better adhesion to complex three-dimensional surfaces and joints of robots. These core sensors include a temperature sensor, a pressure sensor, an infrared proximity sensor, and a six-axis accelerometer, enabling real-time, high-precision, low-power, high-response, and highly sensitive data acquisition.

[0048] The three-dimensional sensing array of the electronic skin infrared proximity sensor in step S1 of the embodiment of the present application is as follows Figure 2 As shown in the figure, 1 is a three-dimensional sensing array, namely the electronic skin array, and 2 is an infrared proximity sensor.

[0049] Figure 3 shows a basic schematic diagram of the electronic skin's infrared proximity sensor array sensing an external object and reconstructing its three-dimensional contour. This diagram illustrates the operation of the electronic skin's infrared proximity sensor array in detecting an external cylindrical object. Reference numeral 3 represents the multimodal electronic skin, and reference numeral 4 represents the object being measured in the environment.

[0050] In the embodiment of the present application, in step S2, a three-dimensional grid space is obtained based on a three-dimensional sensing array, and a gridded digital representation of the entire three-dimensional space needs to be obtained to obtain the grid space.

[0051] In some embodiments, the grid space is a quantifiable and discretizable space formed by intersection nodes of detection ranges of multiple infrared proximity sensors in a three-dimensional sensing array.

[0052] The grid space in this embodiment is as follows Figure 4 As shown, the grid space is a quantifiable and discretizable space composed of the intersection nodes of the detection ranges of each infrared proximity sensor in the three-dimensional sensing array of the electronic skin infrared proximity sensor.

[0053] In step S3 of the present application, the contour vector is decomposed in three dimensions to obtain contour data, and the contour data is smoothed. Specifically, the contour vector is decomposed in three axes (X, Y, and Z) to achieve contour smoothing in a two-dimensional cross section.

[0054] In some embodiments, smoothing the contour data includes: performing a median absolute deviation outlier method on the contour data to obtain contour data with discrete values removed; and filtering the contour data with discrete values removed.

[0055] Specifically, the contour data is processed using the MAD (Median Absolute Deviation) outlier method to remove discrete values; then, the contour data is processed using the IIR (Infinite Impulse Response) filter to smooth the curve contour.

[0056] In some embodiments, the time series data is used to represent a curve of distance data sensed by each infrared proximity sensor over a fixed period of time.

[0057] The time series data in this embodiment is as follows Figure 5 As shown, it represents the curve of the distance data sensed by each sensor over a fixed period of time.

[0058] The visualization result of the contour vector formed by the change points in the grid space in this embodiment is as follows: Figure 6 As shown in the figure, the visualization results of the contour vector on the two-dimensional section are displayed, where the value of each change point represents the detection confidence, that is, the number of sensors that detect the object at the node in the grid space, and the two-dimensional section visualization result is distinguished by the color depth of the color block.

[0059] The three-dimensional contour vector in this embodiment can be decomposed into three-axis two-dimensional contour vectors to simplify the amount of data processing and the difficulty of calculation. The decomposition into the X-axis two-dimensional contour vector schematic diagram is as follows Figure 7 As shown, it is decomposed into a two-dimensional contour vector diagram along the Y axis as shown in Figure 8 As shown, it is decomposed into a two-dimensional contour vector diagram along the Z axis as shown in Figure 9 shown.

[0060] In the actual data acquisition process, the sampled data of the proximity sensor array will produce certain errors due to the changes in the surrounding environment. For this reason, we use the MAD outlier method to remove discrete values in the data, and then use the IIR filter to filter the sampled data to further smooth the contour vector.

[0061] During the test, the infrared proximity sensor array data is collected separately, such as Figure 10 As shown. Figure 10 For the sensor time series data in , we use the MAD (median absolute deviation) method to handle outliers. Specifically, the formula for outlier processing is shown as follows:

[0062] MAD=median(|X i -median(X)|)

[0063] Among them, median is the median function, X i is the data point at the i-th position, and X is the sum of the time series data intercepted by the fixed sliding window length.

[0064] First, calculate the residual between the current time series data point and its median, that is, get the median of the absolute value of the deviation, and select the MAD as the threshold to perform discrete value filtering of the data. Compare the current time series data point with 3 times the threshold. When the value exceeds 3 times the threshold, it is considered an outlier and smoothing is performed to obtain the result as shown below. Figure 11 shown.

[0065] Next, we perform IIR filtering to smooth the curve and use the bilinear transformation method to design the IIR infinite impulse response filter. The design steps are as follows:

[0066] The digital frequency ω index is converted into Convert to analog frequency Ω indicator;

[0067] Design the system function H(s) of the analog filter according to its technical specifications;

[0068] The H(s) of the analog filter is converted into the H(z) of the digital filter using the bilinear transformation method.

[0069] The indicators of the designed IIR low-pass filter are shown in Table 1 below:

[0070] Table 1 IIR filter indicators

[0071]

[0072] The time series data after smoothing by IIR filter is as follows Figure 12 shown.

[0073] All the three-axis two-dimensional contour vectors after data smoothing are merged to obtain a three-dimensional feature vector visualization diagram of the surrounding object contour. Figure 13 shown.

[0074] After obtaining the time series data, the object's three-dimensional spatial information is calculated based on the time series data. In some embodiments, the three-dimensional spatial information includes the object's size and center coordinates, the object's movement speed, the object's movement direction, and the object's movement acceleration within a three-dimensional space with the electronic skin array as the spatial origin.

[0075] The robot's relative position is calculated using the e-skin's Inertial Measurement Unit (IMU) sensor, which corrects the object's real-time spatial information to obtain the current global environment feature vector. The IMU sensor is a combination of an accelerometer and a gyroscope. It detects acceleration and angular velocity to indicate motion and intensity.

[0076] In some embodiments, the robot's relative position and posture are acquired through the electronic skin's inertial sensors, including: sampling the robot's acceleration and angular velocity data from the inertial sensors; and calculating the robot's relative position and posture using an extended Kalman filter algorithm based on the robot's acceleration and angular velocity data. Specifically, the robot's three-axis acceleration and three-axis angular velocity are obtained from the electronic skin's IMU sensors, and the robot's relative position and posture trajectory is calculated using an EKF (Extended Kalman Filter) algorithm.

[0077] In some embodiments, the reinforcement learning algorithm uses a deep deterministic policy gradient algorithm (DDPG) based on a value function and a policy function to take the global dynamic environment as network input through an end-to-end learning method and directly output the robot's path planning information.

[0078] Specifically, the reinforcement learning algorithm in this embodiment adopts the DDPG algorithm based on value function and policy function, takes the global dynamic environment as network input through end-to-end learning, directly outputs the robot's path planning information, and obtains the optimal decision strategy by maximizing the cumulative reward of successful obstacle avoidance after the robot interacts with the dynamic environment. The design structure diagram is shown in the figure below. Figure 14 No.

[0079] Among them, the learning steps of the DDPG algorithm are as follows:

[0080] Randomly initialize the critic network parameters Q θ , and the actor network parameters π θ ;

[0081] Initialize the target network parameters θ′←θ, Initialize the experience replay array β;

[0082] For each time step, execute the loop:

[0083] 1. Exploring Noise Next, select Action, where

[0084] 2. Get the observed reward r for the next state s′ corresponding to the action, and store (s, a, r, s′) as a tuple in β;

[0085] 3. When the collected data reaches a certain level, sample the smallest batch of training samples (s, a, r, s′) from β and input them into the targetcritic network to calculate Q θ′ (s′a) value, through y←r+γQ θ′ (s′a) Calculate the target Q value;

[0086] 4. Calculate the currentQ value through the critic network, calculate the cross entropy loss and update the critic network; the cross entropy loss is shown as follows:

[0087] θ←argmin θ N -1 ∑(yQ θ (s,a)) 2

[0088] 5. Use the descending gradient method to maximize the Q value and update the actor network;

[0089] 6. End the loop.

[0090] See Figure 15 The embodiment of the present application also provides a three-dimensional reconstruction and path planning device, including a multimodal perception module 101 and a data processing module 102; the multimodal perception module 101 includes a temperature acquisition module 1011, a humidity acquisition module 1012, a pressure acquisition module 1013, a ranging module 1014, an acceleration acquisition module 1015 and an angular velocity acquisition module 1016; the multimodal perception module 101 is used to realize data acquisition, and the data includes collected temperature, humidity, pressure, obstacle distance, robot surface acceleration and robot surface angular velocity; the data processing module 102 is used to process the data collected by the multimodal perception module to obtain three-dimensional spatial information of the object; and based on the relative posture of the robot, the three-dimensional spatial information is corrected, and the current global environment feature vector is obtained from the corrected three-dimensional spatial information; based on the current global environment feature vector and the reinforcement learning algorithm, path planning information is obtained.

[0091] Specifically, the multimodal sensing module 101 is composed of an IC (integrated circuit) temperature sensor, a MEMS pressure sensor, an infrared-based MEMS (micro-electromechanical system) proximity sensor, and a MEMS six-axis accelerometer, to achieve high-precision, low-power, high-response, and high-sensitivity data acquisition.

[0092] As an example, the IC temperature sensor can be the Texas Instruments (TI) HDC2010 low-power digital temperature and humidity sensor. This integrated circuit offers high precision, low cost, and a compact size. Integrated on the bottom of the electronic skin module, the IC temperature sensor minimizes environmental influences and provides high stability.

[0093] As an example, the MEMS pressure sensor may be Honeywell's FMAMSDXX025WC2C3 digital pressure sensor, which has a small size, high integration, good consistency, and extremely high measurement accuracy.

[0094] Compared with traditional proximity sensors that use the mutual capacitance sensing principle, infrared-based MEMS proximity sensors use VCNL proximity sensors. Based on the optical measurement principle, they use the linear relationship between reflected light and reflected distance to measure distance. They are not easily affected by the environment, and have high detection accuracy and good stability. The integrated packaging used greatly reduces the size of the proximity sensor.

[0095] As an example, the MEMS six-axis accelerometer may be the BMI160 six-axis accelerometer from BOSCH, which can accurately measure the acceleration and angular velocity of an object and characterize the surface vibration information of the object.

[0096] The data processing module 102 is used to process the data of the signals collected by the proximity sensor array to obtain the three-dimensional contour feature vector of the object.

[0097] The data processing module implements the method of the above embodiment, such as Figure 1As shown, steps S1 to S7 are executed. The method of the above embodiment is a typical object three-dimensional reconstruction and path planning method based on multimodal sensing electronic skin. First, a detection distance signal output by the electronic skin infrared proximity sensor array is obtained; then, the detection distance signal is sampled according to a preconfigured sampling frequency to obtain a sampling signal; based on the sampling signal, grid space change point data of the entire three-dimensional space is extracted as a contour vector; the contour vector is decomposed into three axes of X, Y, and Z, and pre-processed on a two-dimensional section; the decomposed three-axis two-dimensional contour vector is processed by the MAD outlier method to remove discrete values; the decomposed three-axis two-dimensional contour vector is processed by IIR filtering to obtain a smooth curve contour; all processed three-axis two-dimensional contour vectors are resynthesized into a three-dimensional contour, and the array sensor time series data corresponding to the three-dimensional contour is extracted; the three-dimensional spatial information of the object is calculated based on the time series data, including the size and center coordinates of the object in the three-dimensional space, the object's moving speed, the object's moving direction, and the object's moving acceleration; the robot's three-axis acceleration and three-axis angular velocity are obtained based on the electronic skin's IMU (inertial motion unit) sensor, and the robot's relative position trajectory is calculated using the EKF algorithm. Then, the real-time spatial information of the object is corrected in combination with the robot's relative position trajectory to obtain the current global environment feature vector; finally, the global environment feature vector is input into the end-to-end reinforcement learning algorithm and the path planning information is directly output, realizing a direct closed loop of the system's environmental perception and decision-making control.

[0098] refer to Figure 16 Another embodiment of the present application provides an electronic device, comprising: at least one processor 110; and a memory 111 communicatively connected to the at least one processor; wherein the memory 111 stores instructions that can be executed by the at least one processor 110, and the instructions are executed by the at least one processor 110 so that the at least one processor 110 can execute any of the above method embodiments.

[0099] The memory 111 and the processor 110 are connected using a bus. The bus may include any number of interconnected buses and bridges. The bus connects various circuits of one or more processors 110 and the memory 111. The bus may also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. These are all well known in the art and are therefore not described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver may be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor 110 is transmitted over a wireless medium via an antenna. Furthermore, the antenna receives data and transmits the data to the processor 110.

[0100] The processor 110 is responsible for managing the bus and general processing, and may also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. The memory 111 may be used to store data used by the processor 110 when performing operations.

[0101] Another embodiment of the present application relates to a computer-readable storage medium storing a computer program, which implements the above method embodiment when executed by a processor.

[0102] That is, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a program, which is stored in a storage medium and includes a number of instructions for causing a device (which may be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps in the above-mentioned methods of each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0103] Based on the above technical solution, the embodiment of the present application provides a three-dimensional reconstruction and path planning method, device, equipment and storage medium, which includes: first, based on electronic skin, constructing a three-dimensional sensing array of infrared proximity sensors; then, based on the three-dimensional sensing array, obtaining a three-dimensional grid space, and extracting a contour vector from the grid space; next, performing three-dimensional decomposition on the contour vector to obtain contour data, and smoothing the contour data; next, synthesizing the smoothed contour data into a three-dimensional contour, and extracting the time series data of the array sensor corresponding to the three-dimensional contour; then, based on the time series data, obtaining the three-dimensional spatial information of the object; next, based on the relative posture of the robot, correcting the three-dimensional spatial information, and obtaining the current global environment feature vector from the corrected three-dimensional spatial information; finally, based on the current global environment feature vector and the reinforcement learning algorithm, obtaining path planning information.

[0104] The three-dimensional reconstruction and path planning method provided by the present application can sense physical quantities such as temperature, distance, acceleration, and pressure through the use of multimodal electronic skin integrated sensors. It is small in size and can be networked and expanded over a large area, so as to better adhere to the complex three-dimensional surfaces and joints of the robot. The multimodal electronic skin used in the present application can achieve full coverage perception of the external environment and can obtain complete environmental characteristics in confined environments and unknown complex environments after disasters. In addition, the data processing method used in the present application can smooth and filter the signals collected by the infrared proximity sensor array, and has a fast calculation speed, and can obtain accurate three-dimensional surface contours of objects, meeting the needs of fast real-time performance. At the same time, the reinforcement learning algorithm used in the present application can directly output the input global environment feature vector as the robot's path planning information, realizing a direct closed loop of the system's environmental perception and decision-making control, thereby realizing path planning for the entire environment in unknown, complex, and confined environments.

[0105] Those skilled in the art will appreciate that the above-described embodiments are specific examples for implementing the present application, and that in actual applications, various changes in form and detail may be made thereto without departing from the spirit and scope of the present application. Any person skilled in the art may make changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be subject to the scope defined in the claims.

Claims

1. A three-dimensional reconstruction and path planning method, characterized in that: include: Based on electronic skin, a three-dimensional sensing array of infrared proximity sensors is constructed; Acquiring a three-dimensional grid space based on the three-dimensional sensing array, and extracting a contour vector from the grid space; Performing three-dimensional decomposition on the contour vector to obtain contour data, and performing smoothing processing on the contour data; synthesizing the smoothed contour data into a three-dimensional contour, and extracting the time series data of the array sensor corresponding to the three-dimensional contour; Based on the time series data, obtaining three-dimensional spatial information of the object; Based on the relative position and posture of the robot, the three-dimensional spatial information is corrected, and a current global environment feature vector is obtained from the corrected three-dimensional spatial information; Acquiring path planning information based on the current global environment feature vector and a reinforcement learning algorithm; The electronic skin uses a multimodal electronic skin integrated with core sensors, including a temperature sensor, a pressure sensor, an infrared proximity sensor, and a six-axis acceleration sensor. The core sensors are attached to the robot's three-dimensional surface and joints and can sense four physical quantities: temperature, distance, acceleration, and pressure. The reinforcement learning algorithm adopts a deep deterministic policy gradient algorithm based on value function and policy function, takes the global dynamic environment as network input through end-to-end learning, and directly outputs the robot's path planning information.

2. The three-dimensional reconstruction and path planning method according to claim 1, characterized in that: The grid space is a quantifiable and discretizable space formed by the intersection nodes of the detection ranges of multiple infrared proximity sensors in a three-dimensional sensing array.

3. The three-dimensional reconstruction and path planning method according to claim 1, characterized in that: The smoothing process on the contour data comprises: Processing the profile data using a median absolute deviation outlier method to obtain profile data with discrete values removed; The contour data from which the discrete values are removed is subjected to filtering processing.

4. The three-dimensional reconstruction and path planning method according to claim 1, characterized in that: The time series data is used to represent a curve of distance data sensed by each infrared proximity sensor over a fixed period of time.

5. The three-dimensional reconstruction and path planning method according to claim 1, characterized in that: The three-dimensional spatial information includes the size and center coordinates of the object in the three-dimensional space with the electronic skin array as the spatial origin, the object's moving speed, the object's moving direction, and the object's moving acceleration.

6. The three-dimensional reconstruction and path planning method according to claim 1, characterized in that: The relative position of the robot is obtained through the inertial sensor of the electronic skin, including: The inertial sensor samples and obtains the robot acceleration and angular velocity data; Based on the robot acceleration and the angular velocity data, the relative position and posture of the robot are calculated by an extended Kalman filter algorithm.

7. An electronic device, characterized in that: include: at least one processor; as well as, A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so as to enable the at least one processor to execute the three-dimensional reconstruction and path planning method as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the three-dimensional reconstruction and path planning method according to any one of claims 1 to 6 is implemented.

Images