Scene reconstruction method and system and storage medium

Abstract

The invention provides a scene reconstruction method and system and a storage medium, the method is executed at a user side, the user side is connected with a robot control side, and the method comprises the following steps: transmitting a robot control instruction and a radar board control instruction to the robot control side to obtain robot motion data and millimeter wave original signals, robot coordinates of the robot motion data are converted into world coordinates to obtain a coordinate conversion relation; the data processing module combines data obtained after the millimeter wave original signals are subjected to fast Fourier transform in a distance dimension and a speed dimension, and inputs a constructed thermodynamic diagram into a preset neural network to output effective signal point positions; and performing fast Fourier transform on speed and distance data acquired by antennas in different directions to obtain a horizontal angle and a pitch angle, transmitting the obtained point cloud data to the model construction module, converting the point cloud data into world coordinates based on a coordinate conversion relationship, and transmitting the obtained scene reconstruction model data to the user interface module for display.

Description

Technical Field

[0001] The present invention relates to the technical field of scene reconstruction, and in particular, to a scene reconstruction method, system, and storage medium. Background Art

[0002] With the rapid development of intelligent driving and intelligent robot technologies, scene reconstruction technology, as an important means of perceiving the environment, plays a key role in these fields; accurate and efficient environment reconstruction methods are crucial for the automatic navigation and braking tasks of vehicles and robots.

[0003] In the prior art, the environment reconstruction method mainly uses cameras and lidar to scan the environment and obtain point cloud data for modeling. However, cameras and lidar are greatly affected by light and the environment, and the signal acquisition performance will significantly decrease under harsh weather conditions such as dark environments and foggy weather, thus limiting the application scenarios of the scene reconstruction system; in addition, the point cloud obtained by traditional millimeter-wave processing methods is relatively sparse and has many noise points, which affects the effect of scene reconstruction. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a scene reconstruction method, system, and storage medium to eliminate or improve one or more defects existing in the prior art and solve the problem that accurate and efficient scene reconstruction cannot be achieved in the prior art.

[0005] One aspect of the present invention provides a scene reconstruction method, and the method includes the following steps:

[0006] Transmit the robot control instruction and the radar board control instruction to the robot control terminal; the robot control instruction controls the movement of the robot and collects robot movement data through a preset sensor in the robot, so as to obtain a coordinate conversion relationship by converting the robot coordinates of the robot movement data into world coordinates; the radar board control instruction controls the radar board and collects millimeter-wave raw signals;

[0007] Receive the millimeter-wave raw signal and perform fast Fourier transforms on the millimeter-wave raw signals corresponding to each antenna in the radar board in the range dimension and the velocity dimension respectively, and construct a heat map based on the data of each antenna in the range and velocity dimensions; detect the heat map via a preset neural network and output the positions of the valid signal points in the heat map; arrange the velocity-range data of the positions of the valid signal points received by the horizontal antennas on the radar board together, and perform a fast Fourier transform to obtain the horizontal angle of the positions of the valid signal points; arrange the velocity-range data of the positions of the valid signal points received by the vertical antennas on the radar board together, and perform a fast Fourier transform to obtain the elevation angle of the positions of the valid signal points; summarize the range, velocity, horizontal angle, and elevation angle corresponding to each position of the valid signal points to obtain point cloud data;

[0008] Based on the coordinate transformation relationship, convert the point cloud data from the robot coordinate to the world coordinate and display the obtained scene reconstruction model data.

[0009] In some embodiments, after the robot control instruction and the radar board control instruction are transmitted to the robot control terminal, it further includes:

[0010] Use serial communication to write the radar board control instruction including the radar board sampling rate, signal start frequency, and signal bandwidth into the radar board;

[0011] Use serial communication to write the robot control instruction including the robot motion path control instruction, motion speed control instruction, and motion state control instruction into the robot.

[0012] In some embodiments, the robot control instruction controls the robot to move and collects robot motion data through a preset sensor in the robot, and the coordinate transformation relationship obtained by converting the robot coordinate of the robot motion data into the world coordinate includes:

[0013] The preset sensor collects the robot motion data and calculates the dynamic position offset and dynamic angle offset of the robot after motion according to the robot motion data; obtains the static position offset and static angle offset according to the position of the radar board on the robot base;

[0014] Calculate the rotation matrix for angle conversion according to the dynamic coordinate offset and the static coordinate offset, and calculate the translation matrix for position conversion according to the dynamic position offset and the static position offset;

[0015] Obtain the coordinate transformation relationship according to the rotation matrix and the translation matrix.

[0016] In some embodiments, the form of the millimeter-wave original signal is a three-dimensional matrix including a distance dimension, a velocity dimension, and an angle dimension. The process of obtaining the three-dimensional matrix includes:

[0017] Obtain a preset number of transmitting antennas and receiving antennas in the radar board; the transmitting antennas are used to sequentially emit a preset number of detection signals according to a signal transmission period; the receiving antennas are used to receive the echo signals generated after the detection signals hit the surface of the object to be measured;

[0018] The preset mixer in the radar board mixes the detection signals with the corresponding echo signals to obtain a plurality of intermediate-frequency signals, and samples the plurality of intermediate-frequency signals to obtain a preset number of sampling points;

[0019] Calculate the distance according to the frequency of the intermediate-frequency signals obtained within the signal transmission period, calculate the velocity according to the phase difference of the frequency peaks, and calculate the angle according to the phase difference of the intermediate-frequency signals and the fixed distances of different antennas.

[0020] In some embodiments, the structure of the preset neural network includes an embedding layer, multiple transformer layers, and an artificial neural network layer; each of the transformer layers is connected to the artificial neural network layer in a residual short connection; the process of detecting the heat map through the preset neural network and outputting the positions of valid signal points in the heat map includes:

[0021] Preprocess the heat map through the embedding layer and convert the heat map into a feature map;

[0022] The feature map is input into multiple transformer layers, and each transformer layer obtains a feature representation through an attention mechanism;

[0023] Fuse the multiple feature representations through the artificial neural network layer, output the predicted value of each pixel point in the heat map through the classifier of the artificial neural network layer, and use the signal points with predicted values higher than a preset threshold as valid signal points.

[0024] In some embodiments, the steps of converting the point cloud data from the robot coordinate to the world coordinate based on the coordinate conversion relationship and obtaining the scene reconstruction model data include:

[0025] After filtering out the point cloud data outside the preset coordinate range, find the coordinate conversion relationship for converting the point cloud data from the robot coordinate to the world coordinate. If it does not exist, directly exit. If it exists, convert the point cloud data from the robot coordinate to the world coordinate according to the coordinate conversion relationship;

[0026] Insert the point cloud data converted to the world coordinate into the preset data storage model in the model construction module and publish the generated scene reconstruction model data to the map topic.

[0027] In some embodiments, presenting the obtained scene reconstruction model data includes:

[0028] Using the RVIZ tool to subscribe to the map topic containing the scene reconstruction model data and perform visual display of the scene reconstruction model.

[0029] On the other hand, the present invention also provides a scene reconstruction system, which includes:

[0030] A client for executing the scene reconstruction method as described in any one of the above, including a model construction module, a data processing module, and a user interface module; the data processing module is used to obtain point cloud data according to the millimeter-wave raw signal; the model construction module is used to convert the point cloud data from the robot coordinate to the world coordinate through the coordinate conversion relationship and obtain scene reconstruction model data; the user interface display module is used to present the scene reconstruction model data;

[0031] A robot control terminal, including a robot and radar board control module and a radar signal acquisition module; the robot and radar board control module receives robot control instructions and radar board control instructions, controls the movement of the robot through the robot control instructions and collects robot movement data through preset sensors in the robot, obtains the coordinate conversion relationship by converting the robot coordinate of the robot movement data into the world coordinate; controls the radar board and collects millimeter-wave raw signals through the radar board control instructions.

[0032] In some embodiments, the system further includes:

[0033] An exception prompt module for sending an alarm prompt when data transmission errors, data processing errors, and scene reconstruction errors occur.

[0034] On the other hand, the present invention also provides a computer-readable storage medium, on which computer programs / instructions are stored, and when the programs / instructions are executed by a processor, the steps of the method described in any one of the above are implemented.

[0035] In the scene reconstruction method and system of the present invention, obtaining the coordinate conversion relationship for converting the robot coordinate to the world coordinate can convert the point cloud data into the world coordinate system to ensure the consistency of the point cloud data in the world coordinate system and construct a scene reconstruction model convenient for users to observe; using millimeter-wave sensing has the advantages of high privacy, low power consumption, and anti-interference, and can still maintain good sensing ability in bad weather to achieve occasion reconstruction throughout the day and all hours; using a preset neural network for effective signal point detection, improving the feature extraction ability and ensuring the operation performance of the system, finally obtaining dense point cloud data and suppressing the number of noises; the model construction module can generate an accurate scene reconstruction model according to the converted point cloud data.

[0036] Additional advantages, objects, and features of the present invention will be partly described below, and will partly become apparent to those of ordinary skill in the art after studying the following, or can be learned from the practice of the present invention. The objects and other advantages of the present invention can be achieved and obtained by the structure specifically pointed out in the specification and the drawings.

[0037] Those skilled in the art will understand that the objects and advantages achievable by the present invention are not limited to those specifically described above, and the above and other objects achievable by the present invention will be more clearly understood according to the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings described herein are used to provide a further understanding of the present invention, form a part of this application, and do not limit the present invention. In the drawings:

[0039] Figure 1 is a schematic flowchart of the scenario reconstruction method according to an embodiment of the present invention.

[0040] Figure 2 is a schematic structural diagram of the scenario reconstruction system according to an embodiment of the present invention.

[0041] Figure 3 is a schematic flowchart of the processing of millimeter-wave raw signals to point cloud data according to an embodiment of the present invention.

[0042] Figure 4 is a schematic structural diagram of the preset neural network according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] To make the objects, technical solutions, and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in conjunction with the embodiments and the drawings. Herein, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but do not limit the present invention.

[0044] Herein, it should also be noted that in order to avoid obscuring the present invention due to unnecessary details, only the structures and / or processing steps closely related to the solution of the present invention are shown in the drawings, while other details less related to the present invention are omitted.

[0045] It should be emphasized that the term "comprising / including" when used herein refers to the presence of features, elements, steps, or components, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0046] Herein, it should also be noted that if not specifically stated, the term "connection" in this article can not only refer to direct connection, but also represent indirect connection with an intermediate.

[0047] In the following, embodiments of the present invention will be described with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0048] In the prior art, the environmental reconstruction method mainly uses cameras and lidar to scan the environment and obtain point cloud data for modeling. However, cameras and lidar are greatly affected by light and the environment, and the signal acquisition performance will significantly decline under harsh weather conditions such as dark environments and foggy weather, thus limiting the application scenarios of the scene reconstruction system; in addition, the point clouds obtained by traditional millimeter-wave processing methods are relatively sparse and have many noise points, affecting the effect of scene reconstruction; the present invention proposes a scene reconstruction method, system and storage medium, and the method includes: transmitting robot control instructions and radar board control instructions to the robot control terminal; the robot control instructions control the movement of the robot and collect robot movement data through a preset sensor in the robot, so as to obtain a coordinate conversion relationship by converting the robot coordinates of the robot movement data into world coordinates; the radar board control instructions control the radar board and collect millimeter-wave raw signals; receiving the millimeter-wave raw signals and performing fast Fourier transforms on the millimeter-wave raw signals corresponding to each antenna in the radar board in the distance dimension and the speed dimension respectively, and constructing a heat map based on the data of each antenna in the distance and speed dimensions; detecting the heat map through a preset neural network and outputting the positions of valid signal points in the heat map; arranging the speed-distance data of the positions of the valid signal points received by the horizontal antennas on the radar board together, and obtaining the horizontal angle of the positions of the valid signal points after performing a fast Fourier transform; arranging the speed-distance data of the positions of the valid signal points received by the vertical antennas on the radar board together, and obtaining the elevation angle of the positions of the valid signal points after performing a fast Fourier transform; summarizing the distance, speed, horizontal angle and elevation angle corresponding to each position of the valid signal points to obtain point cloud data; converting the point cloud data from robot coordinates to world coordinates based on the coordinate conversion relationship and displaying the obtained scene reconstruction model data.

[0049] Figure 1 It is a schematic flow chart of the scene reconstruction method according to an embodiment of the present invention. Figure 2 It is a schematic structural diagram of the scene reconstruction system according to an embodiment of the present invention. Specifically, the present application provides a scene reconstruction method, and the scene reconstruction method includes the following steps S101 to S103:

[0050] Step S101: Transmit the robot control instruction and the radar board control instruction to the robot control terminal; the robot control instruction controls the movement of the robot and collects the robot movement data through a preset sensor in the robot, so as to obtain the coordinate conversion relationship by converting the robot coordinates of the robot movement data into world coordinates; the radar board control instruction controls the radar board and collects the millimeter wave raw signal.

[0051] Step S102: Receive the millimeter wave raw signal and perform fast Fourier transform on the millimeter wave raw signal corresponding to each antenna in the radar board in the distance dimension and the velocity dimension respectively, and construct a heat map based on the data of each antenna in the distance and velocity dimensions; pass the heat map through a preset neural network for detection to output the position of the valid signal points in the heat map; arrange the velocity-distance data of the position of the valid signal points received by the horizontal antennas on the radar board together, and perform fast Fourier transform to obtain the horizontal angle of the position of the valid signal points; arrange the velocity-distance data of the position of the valid signal points received by the vertical antennas on the radar board together, and perform fast Fourier transform to obtain the pitch angle of the position of the valid signal points; summarize the distance, velocity, horizontal angle and pitch angle corresponding to each position of the valid signal points to obtain the point cloud data.

[0052] Step S103: Convert the point cloud data from the robot coordinates to the world coordinates based on the coordinate conversion relationship and display the obtained scene reconstruction model data.

[0053] In step S101, the scene reconstruction method is executed on the user side. The user side is connected to the robot control side and deployed in a distributed manner. The user side includes a model building module, a data processing module, and a user interface module. Each module realizes data communication through the Robot Operating System (ROS) framework. The robot control side includes a robot and radar board control module and a radar signal acquisition module, with the ROS framework as the software foundation. In the ROS framework, nodes are used to represent processes. After writing the names of the nodes to be run and the parameters required for each node in a preset format in a file, the ROS framework automatically parses and starts the corresponding nodes. The user interface module sends control information including robot control instructions and radar board control instructions to control the robot and the radar board. The control information is transmitted by the ROS framework to the robot and radar control module. The robot and radar control module controls the robot through the robot control instructions. The robot uses a Turtlebot2 robot. The robot and radar control module transmits the radar board control instructions to the radar information acquisition module, and the radar signal acquisition module transmits them to the radar board to perform signal acquisition. The radar board has a transmitting antenna and a receiving antenna. The transmitting antenna emits a millimeter-wave detection signal (chirp signal) whose frequency linearly increases with time. After the millimeter-wave detection signal hits the surface of the object to be measured, an echo signal is generated and received by the receiving end. The mixer deployed on the radar board mixes the millimeter-wave detection signal and the echo signal to obtain an intermediate-frequency signal. The frequency of the intermediate-frequency signal is obtained according to the Fast Fourier Transform (FFT), and the distance from the object to be measured to the radar board is calculated based on the motion characteristics of the millimeter wave. A transmitting antenna periodically emits multiple millimeter-wave detection signals. After an echo signal is generated when reaching an object to be detected, the intermediate-frequency signal is calculated and the Fast Fourier Transform is performed to obtain multiple frequency peaks. The speed of the object to be detected is calculated based on the phases of the multiple frequency peaks and the Doppler effect. When two transmitting antennas reach an object to be detected, there is a distance difference. The distance difference generates a phase difference in the intermediate-frequency signal. The angle formed by the object and the antennas is calculated based on the distance difference and the fixed distance between the two antennas.

[0054] In some embodiments, the form of the millimeter-wave raw signal is a three-dimensional matrix including a distance dimension, a speed dimension, and an angle dimension. The process of obtaining the three-dimensional matrix includes steps S1011 to S1013:

[0055] Step S1011: Obtain a preset number of transmitting antennas and receiving antennas in the radar board. The transmitting antenna is used to sequentially emit a preset number of detection signals according to the signal emission cycle. The receiving antenna is used to receive the corresponding echo signals generated after the detection signals hit the surface of the object to be measured.

[0056] Step S1012: The preset mixer of the radar board mixes the detection signal with the corresponding echo signal to obtain multiple intermediate frequency signals, and samples the multiple intermediate frequency signals to obtain a preset number of sampling points.

[0057] Step S1013: Calculate the distance based on the frequency of the intermediate frequency signal obtained within the signal emission period, calculate the speed based on the phase difference of the frequency peaks, and calculate the angle based on the phase difference of the intermediate frequency signal and the fixed distance between different antennas.

[0058] Specifically, in each emission period, each of the TX transmitting antennas among the TX transmitting antennas periodically transmits D millimeter-wave detection signals and uses the RX receiving antennas for reception. Samples are taken for the intermediate frequency signals obtained for each millimeter-wave detection signal to obtain R sampling points. The number of signal samples in each emission period is R * D * (TX * RX); the R sampling points are samples of the intermediate frequency signals at different times and the round-trip time of the millimeter-wave detection is obtained, and the distance is obtained based on the round-trip time; when periodically transmitting D millimeter-wave detection signals, there is a frequency shift for each obtained intermediate frequency signal, and the speed is obtained based on the frequency shift; the TX transmitting antennas and the RX receiving antennas simulate TX * RX pairs of virtual receiving antennas. There is a distance difference and an intermediate frequency signal phase difference when each two pairs of virtual receiving antennas reach an object to be detected, and the angle is obtained based on the distance between the two pairs of virtual receiving antennas; therefore, organizing in the form of R * D * (TX * RX) can obtain a three-dimensional array form in the distance dimension, speed dimension, and angle dimension; the robot and the radar control module organize the millimeter-wave raw signal into a ROS message type for easy transmission to the user side.

[0059] In some embodiments, the robot control instruction controls the movement of the robot and collects the robot movement data through the preset sensors in the robot, so as to obtain the coordinate conversion relationship by converting the robot coordinates of the robot movement data into world coordinates, including steps S1 - S3:

[0060] Step S1: The preset sensors collect the robot movement data and calculate the dynamic position offset and dynamic angle offset of the robot after movement based on the robot movement data; obtain the static position offset and static angle offset based on the position of the radar board on the robot base.

[0061] Step S2: Calculate the rotation matrix for angle conversion based on the dynamic coordinate offset and the static coordinate offset, and calculate the translation matrix for position conversion based on the dynamic position offset and the static position offset.

[0062] Step S3: Obtain the coordinate conversion relationship based on the rotation matrix and the translation matrix.

[0063] In some embodiments, after the robot control instruction and the radar board control instruction are transmitted to the robot control terminal, it further includes:

[0064] Write the radar board control instructions including the radar board sampling rate, signal start frequency, and signal bandwidth into the radar board by means of serial port communication.

[0065] Write the robot control instructions including the robot motion path control instruction, motion speed control instruction, and motion state control instruction into the robot by means of serial port communication.

[0066] Specifically, the preset sensor adopts a robot odometer sensor. Based on the coordinate transformation mechanism provided by the robot operating system, the coordinate transformation mechanism includes a dynamic coordinate transformation mechanism and a static coordinate transformation mechanism. The dynamic coordinate transformation mechanism enables the robot odometer sensor to calculate the coordinate transformation relationship according to the position offset of the robot compared to when it starts up and the angular offsets of the yaw angle, pitch angle, and roll angle. The static coordinate transformation relationship is to obtain the position offset and angular offset from the radar board to the robot base according to the detailed installation position from the radar board to the robot base; further, obtain the coordinate transformation relationship and publish it to the coordinate transformation (Transform, TF) topic in the form of a robot operating system message.

[0067] In step S102, Figure 3 This is a schematic diagram of the processing flow of millimeter-wave raw signals to point cloud data according to an embodiment of the present invention. Obtain the millimeter-wave raw signals through the subscription method of the robot operating system and perform signal processing. Perform fast Fourier transforms on the distance dimension and speed dimension of the millimeter-wave raw data respectively and combine the data of each antenna in the distance and speed dimensions. Use speed and distance as the horizontal and vertical coordinates, and use the signal intensity to mark the color depth to construct a heat map of the millimeter-wave raw signals corresponding to all antennas; input the heat map into a preset neural network for effective point signal detection. The preset neural network adopts a SegFormer neural network (lightweight self-attention semantic segmentation network) to improve the quantity and quality of the point cloud in the scene reconstruction scenario; obtain point cloud data after angle estimation. Figure 4 This is a schematic diagram of the structure of the preset neural network according to an embodiment of the present invention. In some embodiments, the structure of the preset neural network includes an embedding layer, multiple transformer layers, and an artificial neural network layer; each transformer layer is connected in a residual short connection with the artificial neural network layer; the process of detecting the effective signal point positions in the heat map through the preset neural network includes steps S1021 to S1023:

[0068] Step S1021: Preprocess the heat map through the embedding layer and convert the heat map into a feature map;

[0069] Step S1022: Input the feature map into multiple transformer layers, and each transformer layer obtains a feature representation through an attention mechanism;

[0070] Step S1023: Fuse multiple feature representations through an artificial neural network layer, output the predicted value of each pixel point in the heat map through the classifier of the artificial neural network layer, and use the signal points with predicted values higher than the preset threshold as valid signal points.

[0071] Specifically, the SegFormer neural network combines a transformer with a multi-layer perceptron (MLP); further, the form of array indexing is used to represent the positions of valid signal points detected in the heat map by the SegFormer neural network. In the heat map, speed and distance are used as the horizontal and vertical coordinates, and the color depth is marked with signal intensity. The SegFormer neural network outputs a [10, 20] array, indicating that there is a valid point signal at the position of x = 10 and y = 20 in the heat map.

[0072] Further, the data in the angle dimension of the millimeter-wave raw signal is reordered according to the physical arrangement of the antennas. The speed-distance data received by the horizontal antennas on the radar board are arranged together to obtain horizontal data, and the speed-distance data received by the vertical antennas are arranged together to obtain vertical data. The horizontal data is subjected to a fast Fourier transform to obtain the horizontal angle, and the vertical data is subjected to a fast Fourier transform to obtain the pitch angle. First, the horizontal data received by the horizontal antennas is subjected to a fast Fourier transform to obtain the horizontal angle spectrum, and a peak search is performed on the horizontal angle spectrum to obtain different horizontal angles corresponding to the signal points of multiple horizontal angle peaks; next, the pitch angle corresponding to the horizontal angle peak is calculated. There are multiple horizontal antenna arrays in the millimeter-wave radar. After performing a fast Fourier transform on the horizontal data received by multiple horizontal antennas in the horizontal dimension, multiple peaks are obtained and the signal phases corresponding to the peaks are different. After performing a fast Fourier transform on the vertical data received by the vertical antennas at the peak positions, a pitch angle spectrum is obtained, and a peak search is performed on the pitch angle spectrum to obtain different pitch angles corresponding to the signal points of multiple peaks; a 4×4 virtual antenna array, including 4 horizontal arrays, and each horizontal array includes 4 antennas; point cloud data in the robot coordinate system is obtained based on the distance, speed, horizontal angle, and pitch angle. In step S103, the coordinate transformation relationship between the robot and the radar control module and the point cloud data of the data processing module are obtained by subscription. The preset modeling tool in the model construction module transforms each point cloud in the point cloud data from the robot coordinate to the world coordinate. The preset modeling tool uses the OctoMap modeling tool, and the OctoMap modeling tool is an open-source library for three-dimensional environment modeling; in some embodiments, the steps of transforming the point cloud data from the robot coordinate to the world coordinate based on the coordinate transformation relationship and obtaining the scene reconstruction model data include S1031 to S1032:

[0073] Step S1031: After filtering out the point cloud data outside the preset coordinate range, search for the coordinate transformation relationship that transforms the point cloud data from the robot coordinates to the world coordinates. If it does not exist, exit directly. If it does exist, transform the point cloud data from the robot coordinates to the world coordinates according to the coordinate transformation relationship.

[0074] Step S1032: inserting the point cloud data converted to world coordinates into the preset data storage model in the model building module and publishing the generated scene reconstruction model data to the map topic.

[0075] Specifically, the preset data storage model is a model for storing the OctoMap modeling tool in memory, which is based on the octree data structure and is a special model format for storing the OctoMap modeling tool in memory; the preset modeling tool stores the point cloud data converted into world coordinates in the preset data storage model, obtains the scene reconstruction model data and publishes it to the map topic, and the map topic can be subscribed by the user interface module and send the scene reconstruction model data to the user interface module. In some embodiments, displaying the obtained scene reconstruction model data includes: using the RVIZ tool to subscribe to the map topic containing the scene reconstruction model data and perform a visual display of the scene reconstruction model. Specifically, the RVIZ tool is a graphical tool that comes with the robot operating system, which can realize the subscription to the map topic, obtain the scene reconstruction model data and display the scene reconstruction model data on the user end.

[0076] On the other hand, the present invention also provides a scene reconstruction system, the system comprising:

[0077] The user end is used to execute any of the scene reconstruction methods as described above, including a model construction module, a data processing module and a user interface module; the data processing module is used to obtain point cloud data based on the millimeter wave original signal; the model construction module is used to convert the point cloud data from the robot coordinates into the world coordinates through the coordinate transformation relationship and obtain the scene reconstruction model data; the user interface display module is used to display the scene reconstruction model data.

[0078] The robot control end includes a robot and radar board control module and a radar signal acquisition module; the robot and radar board control module receives robot control instructions and radar board control instructions, controls the robot movement through the robot control instructions and collects the robot motion data through the preset sensors in the robot, and obtains the coordinate conversion relationship by converting the robot coordinates of the robot motion data into world coordinates; controls the radar board through the radar board control instructions and collects the millimeter wave original signal.

[0079] In some embodiments, the scene reconstruction system further includes:

[0080] An exception prompt module is used to issue an alarm prompt when data transmission errors, data processing errors, and scenario reconstruction errors occur. The exception prompt module can notify the user in a timely manner when a failure occurs, so that the user can handle the failure according to the alarm prompt, improving the working efficiency of the model reconstruction system.

[0081] On the other hand, the present invention also provides a computer-readable storage medium, on which a computer program / instructions are stored. When the program / instructions are executed by a processor, the steps of any of the above methods are implemented.

[0082] The present invention will be described below in conjunction with a specific embodiment:

[0083] Deploy the entire system in a distributed manner: the robot control end and the user end. Among them, the robot control end uses the ROS framework as the software foundation and is deployed on a low-power computing platform, which includes a Raspberry Pi 4B (8G memory version).

[0084] 1. The robot control end includes a radar signal acquisition module and a robot and radar control module.

[0085] The radar signal acquisition module realizes the acquisition of millimeter-wave raw signals. Specifically, this module will first control the millimeter-wave radar board and write the configuration file data of the radar board into the radar board in the way of serial communication. The configuration file data of the radar board includes the sampling rate, signal start frequency, and signal bandwidth of the radar; control the radar board to collect millimeter-wave raw signals through the configuration file data of the radar board; re-arrange and organize the data in the distance dimension, speed dimension, and angle dimension every time a frame of data is collected; a frame refers to a complete signal emission cycle, and each transmitting antenna sequentially emits radar detection signals.

[0086] The functions of the robot and radar control module include: organizing the millimeter-wave raw signals obtained by the radar signal acquisition module into the message type of the robot operating system for subsequent transmission to the user end for further processing; using a third-party control program to receive the robot control instructions from the user end and control the robot entity. In this aspect, the Turtlebot2 robot is used, and the robot motion data provided by the robot odometer sensor is read in real time. The robot motion data includes the position offset of the robot compared to the startup position and the angle offsets of the yaw angle, pitch angle, and roll angle. Use the dynamic coordinate conversion mechanism provided by the robot operating system to calculate the coordinate conversion relationship with the startup position with the robot as the coordinate origin, and publish it to the coordinate conversion topic in the form of a robot operating system message for the user end to perform further processing.

[0087] 2. The client reconstructs the scene model based on the millimeter-wave raw data and coordinate transformation relationship obtained from the robot control end and combines with the modeling algorithm. The client includes a data processing module, a model construction module, and a user interface module; data communication between each module is achieved through the Robot Operating System framework.

[0088] The data processing module obtains the millimeter-wave raw signal collected by the radar signal acquisition module through the subscription method of the Robot Operating System. After performing a fast Fourier transform on the data in the distance dimension and then performing a fast Fourier transform again in the velocity dimension, considering the distance and velocity data under each antenna dimension as a whole, a heat map is obtained. The heat map is input into the SegFormer neural network to detect the positions of valid signal points and give the coordinate positions of the valid signals detected by the neural network in the form of array indices. The array index is the position where the valid signal is located; for a heat map with a size of 128*128, the underlying data is actually a 128*128 array. The signal at the position of the array where x = 10 and y = 20 is a valid signal point. The SegFormer neural network performs a binary classification task to determine whether each pixel point in the heat map is a valid signal point and outputs a 128*128 boolean-type array. In the boolean-type array, each pixel point uses a true value to represent being classified as a valid signal point and a false value to represent not being classified as a valid signal point; at the position of x = 10 and y = 20, it is a true value. Next, traverse the entire boolean-type array and record the positions of all true values. Therefore, an array of [10, 20] is output, indicating that there is a valid signal at the position of x = 10 and y = 20 in the heat map. This position needs to be used in the subsequent angle estimation operation.

[0089] At the positions of the detected valid signal points, the data in the angle dimension is reordered again. According to the physical arrangement of the antennas, the horizontal data received by the horizontal antennas are arranged together and a fast Fourier transform is performed to obtain some waveform peaks, and the horizontal angle is obtained based on the waveform peaks; the vertical data received by the vertical antennas are arranged together and a fast Fourier transform is performed to obtain some waveform peaks, and the vertical angle is obtained based on the waveform peaks; finally, multiple valid signal points are obtained. Each valid signal point contains information on distance, velocity, horizontal angle, and pitch angle. These information are converted to the robot coordinate system to obtain the point cloud data within the current radar board signal range. In the data processing module, the signal processing flow from the millimeter-wave raw signal to the point cloud data includes data preprocessing, valid signal detection, and angle calculation.

[0090] The model construction module performs coordinate transformation on the point cloud data obtained by the data processing module and the motion data of the robot or vehicle, so that the point cloud coordinates are transformed to the world coordinate system. At the same time, the modeling module is used to insert the transformed point cloud data into a preset storage model and update the preset storage model data with the latest points, so as to reconstruct more real - world scene information. The preset storage model is a data storage model for storing data of the OctoMap modeling tool in memory, which is a dedicated model format maintained by the OctoMap modeling tool in memory and is based on the octree data structure.

[0091] The specific process from point cloud data to scene reconstruction model includes: The coordinate system where the point cloud data generated by the radar board is located has the position of the radar board as the origin, and the radar board is mounted on the movable robot base during the operation of the entire system. Therefore, the robot coordinates of the point cloud are not fixed, and the robot coordinates of the point cloud must be transformed to a fixed world coordinate system; This is achieved using the coordinate transformation (Transform, TF) mechanism provided by the robot operating system. Coordinate transformation is divided into dynamic coordinate transformation and static coordinate transformation. Dynamic coordinate transformation is used to achieve coordinate transformation between the world coordinate system and the robot coordinate system. A third - party library is used to run an independent node to read the odometer sensor data in the robot in real - time, and the data including the position offset of the robot relative to the starting point of operation, and the angular offsets of yaw angle, pitch angle, and roll angle are obtained from the current robot coordinate position, horizontal deflection angle, and speed provided by the odometer and published to a topic named coordinate transformation; In addition, it also includes the static coordinate transformation from the radar board to the robot base. According to the detailed installation position of the radar board, the static coordinate transformation mechanism provided by the robot operating system is started, and the static position offset and static angle offset are used as transformation parameters for coordinate transformation; The coordinate transformation mechanism of the robot operating system only provides the transformation relationship between different coordinate systems and does not perform the specific transformation process; When constructing the scene model, the OctoMap modeling tool will first check whether there is a transformation between the robot coordinates and the world coordinates. If it exists, the point cloud data will be transformed from the robot coordinates to the world coordinates according to this transformation relationship; The OctoMap modeling tool is an open - source library based on the octree data structure and is used for three - dimensional environment modeling.

[0092] In the OctoMap modeling tool, the process of inserting point cloud data and constructing the map topic is as follows: After filtering out the point cloud data outside the designated range according to the preset coordinate range, find the conversion relationship between the robot coordinates and the world coordinates of the point cloud data. If not found, directly exit; if found, convert the point cloud data from robot coordinates to world coordinates according to the coordinate conversion relationship; the preset range is specified in the form of parameters when the OctoMap modeling tool is started; after converting the point cloud data to the world coordinate system, insert it into the preset storage model maintained inside the OctoMap modeling tool to obtain the scene reconstruction model data, and publish the scene reconstruction model data to the map topic for other visualization programs to subscribe to and visualize.

[0093] Specifically, this solution is developed using the ROS framework. The ROS framework has high flexibility and scalability, and can conveniently integrate each module to achieve efficient data transmission and processing; in the model construction module, obtain the corresponding data by subscribing to the topics of the point cloud data and the robot motion data published by the point cloud generation module; subscription is a mechanism for data communication in the robot operating system. The point cloud data topic is provided with a low-level message queue architecture by the robot operating system, and the data content transmitted is a specific data type representing the point cloud; then perform coordinate conversion on the point cloud data according to the robot motion data to ensure the consistency of the point cloud data in the world coordinate system; finally, insert the converted point cloud data into the OctoMap modeling tool to gradually construct a three-dimensional scene reconstruction model; the OctoMap modeling tool is an efficient three-dimensional space modeling library that can effectively process point cloud data and generate accurate scene models.

[0094] The user interface module is implemented using the RVIZ tool provided by the robot operating system. The RVIZ tool can achieve subscription and visualization display of the map topic.

[0095] In summary, the present invention provides a scene reconstruction method, system, and storage medium. The scene reconstruction method is executed on a user side including a model construction module, a data processing module, and a user interface module. The user side is connected to a robot control side including a robot and a radar board control module and a radar signal acquisition module. The method includes: transmitting robot control instructions and radar board control instructions to a robot control port to obtain robot motion data and millimeter-wave raw signals, and obtaining a coordinate conversion relationship by converting the robot coordinates of the robot motion data into world coordinates; receiving the millimeter-wave raw signals by the user side, and the data processing module respectively performs fast Fourier transforms on the millimeter-wave raw signals corresponding to each antenna in the radar board in the distance dimension and the velocity dimension, and constructs a heat map based on the data of each antenna in the distance and velocity dimensions; the heat map outputs the positions of valid signal points in the heat map via a preset neural network; arranging the velocity-distance data of the positions of valid signal points received by the horizontal antennas and the vertical antennas on the radar board together, respectively performing fast Fourier transforms, and taking the angles corresponding to the waveform peaks as the horizontal angle and the pitch angle of the positions of valid signal points; summarizing the distance, velocity, horizontal angle, and pitch angle corresponding to each position of valid signal points to obtain point cloud data; transmitting the point cloud data to the model construction module, converting the point cloud data from robot coordinates into world coordinates based on the coordinate conversion relationship, and obtaining scene reconstruction model data; and transmitting the scene reconstruction model data to the user interface module for display.

[0096] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the foregoing edge computing server deployment method are implemented. The computer-readable storage medium may be a tangible storage medium, such as a random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0097] Those of ordinary skill in the art should understand that the various exemplary components, systems, and methods described in connection with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Specifically, whether to implement in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, and so on. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium or transmitted through a data signal carried in a carrier wave over a transmission medium or a communication link.

[0098] It should be clear that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order between steps after understanding the spirit of the present invention.

[0099] In the present invention, features described and / or illustrated for one embodiment can be used in the same or a similar manner in one or more other embodiments, and / or combined with the features of other embodiments or replace the features of other embodiments.

[0100] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, various changes and modifications can be made to the embodiments of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A scene reconstruction method, characterized in that: The method comprises the following steps: The robot control instruction and the radar board control instruction are transmitted to the robot control end; the robot control instruction controls the robot movement and collects the robot movement data through the preset sensor in the robot, so as to obtain the coordinate conversion relationship by converting the robot coordinate of the robot movement data into the world coordinate; the radar board control instruction controls the radar board and collects the millimeter wave original signal; Receive the millimeter-wave original signal and perform fast Fourier transform on the millimeter-wave original signal corresponding to each antenna in the radar board in the distance dimension and the speed dimension respectively, and construct a heat map based on the data of each antenna in the distance and speed dimensions; detect the heat map through a preset neural network to output the effective signal point position in the heat map; arrange the speed distance data of the effective signal point position received by the horizontal direction antenna on the radar board together, and obtain the horizontal angle of the effective signal point position after fast Fourier transform; arrange the speed distance data of the effective signal point position received by the vertical direction antenna on the radar board together, and obtain the pitch angle of the effective signal point position after fast Fourier transform; summarize the distance, speed, horizontal angle and pitch angle corresponding to each effective signal point position to obtain point cloud data; Based on the coordinate transformation relationship, the point cloud data is transformed from the robot coordinates to the world coordinates and the obtained scene reconstruction model data is displayed.

2. The scene reconstruction method according to claim 1, characterized in that: After the robot control instructions and radar board control instructions are transmitted to the robot control end, it also includes: Writing the radar board control instructions including the radar board sampling rate, signal starting frequency and signal bandwidth into the radar board by means of serial communication; The robot control instructions including robot motion path control instructions, motion speed control instructions and motion state control instructions are written into the robot by means of serial communication.

3. The scene reconstruction method according to claim 1, characterized in that: The robot control instruction controls the robot movement and collects the robot movement data through the preset sensor in the robot, so as to obtain the coordinate transformation relationship by transforming the robot coordinate of the robot movement data into the world coordinate, including: The preset sensor collects the robot motion data and calculates the dynamic position offset and dynamic angle offset of the robot after the robot moves according to the robot motion data; and obtains the static position offset and static angle offset according to the position of the robot base where the radar board is located; Calculating a rotation matrix for angle conversion according to the dynamic coordinate offset and the static coordinate offset, and calculating a translation matrix for position conversion according to the dynamic position offset and the static position offset; A coordinate transformation relationship is obtained according to the rotation matrix and the translation matrix.

4. The scene reconstruction method according to claim 1, characterized in that: The millimeter wave original signal is in the form of a three-dimensional matrix including a distance dimension, a speed dimension, and an angle dimension. The process of obtaining the three-dimensional matrix includes: Obtain a preset number of transmitting antennas and receiving antennas in the radar board; the transmitting antenna is used to sequentially send out a preset number of detection signals according to a signal transmission cycle; the receiving antenna is used to receive the detection signal and generate a corresponding echo signal after hitting the surface of the object to be detected; The preset mixer of the radar board mixes the detection signal with the corresponding echo signal to obtain a plurality of intermediate frequency signals, and samples the plurality of intermediate frequency signals to obtain a preset number of sampling points; The distance is calculated according to the frequency of the intermediate frequency signal obtained during the signal transmission period, the speed is calculated according to the phase difference of the frequency peak, and the angle is calculated according to the phase difference of the intermediate frequency signal and the fixed distances of different antennas.

5. The scene reconstruction method according to claim 1, characterized in that: The structure of the preset neural network includes an embedding layer, a multi-layer transformer layer and an artificial neural network layer; each transformer layer is connected to the artificial neural network layer by residual short connection; The process of detecting and outputting the effective signal point positions in the heat map via a preset neural network comprises: Preprocessing the heat map through the embedding layer and converting the heat map into a feature map; The feature map is input into multiple layers of transformer layers, and each transformer layer obtains feature representation through an attention mechanism; The multiple feature representations are subjected to feature fusion through the artificial neural network layer, the predicted value of each pixel point in the thermal map is output through the classifier of the artificial neural network layer, and the signal points in the predicted values ​​that are higher than a preset threshold are taken as valid signal points.

6. The scene reconstruction method according to claim 1, characterized in that: The step of converting the point cloud data from robot coordinates to world coordinates based on the coordinate conversion relationship and obtaining scene reconstruction model data comprises: After filtering out the point cloud data outside the preset coordinate range, searching for a coordinate transformation relationship for transforming the point cloud data from the robot coordinates to the world coordinates, and directly exiting when the coordinate transformation relationship does not exist; and transforming the point cloud data from the robot coordinates to the world coordinates according to the coordinate transformation relationship when the coordinate transformation relationship exists; The point cloud data converted to world coordinates is inserted into the preset data storage model in the model building module and the generated scene reconstruction model data is published on the map topic.

7. The scene reconstruction method according to claim 6, characterized in that: The obtained scene reconstruction model data is displayed including: The RVIZ tool is used to subscribe to the map topic containing the scene reconstruction model data and to perform a visual display of the scene reconstruction model.

8. A scene reconstruction system, characterized in that: The system includes: A user terminal, used to execute the scene reconstruction method according to any one of claims 1 to 7, comprising a model construction module, a data processing module and a user interface module; the data processing module is used to obtain point cloud data according to the millimeter wave original signal; the model construction module is used to convert the point cloud data from the robot coordinates to the world coordinates through the coordinate conversion relationship and obtain scene reconstruction model data; the user interface display module is used to display the scene reconstruction model data; The robot control end includes a robot and radar board control module and a radar signal acquisition module; the robot and radar board control module receives robot control instructions and radar board control instructions, controls the robot movement through the robot control instructions and collects robot movement data through preset sensors in the robot, and obtains a coordinate conversion relationship by converting the robot coordinates of the robot movement data into world coordinates; controls the radar board and collects millimeter wave original signals through the radar board control instructions.

9. The scene reconstruction system according to claim 8, characterized in that: The system further comprises: The abnormal prompt module is used to issue an alarm prompt when data transmission errors, data processing errors and scene reconstruction errors occur.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method as claimed in any one of claims 1 to 7 are implemented.

Images