A fusion positioning data processing method in the solution verification stage

Abstract

The present invention provides a fusion positioning data processing method in a scheme verification stage, comprising: a data communication forwarding end process pre-processing a plurality of received sensor data respectively; for each first-category sensor data occupying a small number of bytes, the data communication forwarding end process forwards the encapsulated first-category sensor data to a data communication receiving end process by using a message queue; for the second-category sensor data occupying a large number of bytes, the data communication forwarding end process forwards the encapsulated second-category sensor data to the data communication receiving end process by using a shared memory; dead reckoning processing is performed on the received first-category sensor data, and dedistortion processing is performed on the received second-category sensor data; a result obtained after the dead reckoning processing and a result obtained after the dedistortion processing are sent to a fusion positioning process in the form of ROS format messages; and the fusion positioning process performs fusion positioning based on all received ROS messages.

Description

Technical Field

[0001] The present invention is used for fused positioning data processing and communication in the solution verification stage of intelligent driving vehicles, and specifically relates to a fused positioning data processing method in the solution verification stage. Background Art

[0002] Developing positioning capabilities for intelligent vehicles is a complex systems project. For underground parking scenarios, a fusion positioning solution is employed. The data sources for the fusion positioning module primarily include IMU sensors, wheel speed sensors, and high-definition surround-view camera images. During the solution validation phase, the actual sensor performance and algorithm model effectiveness must be verified. Different solutions must also be rapidly deployed and compared. Considerations must also be given to data access between the sensors and the vehicle itself, internal data communication within the vehicle itself, data communication between the local output and downstream units, and the coordination and debugging of the positioning module with other modules. The IMU and wheel speed sensors each have a small data volume and a high frequency (50-100 Hz). The IMU is sensitive to angle changes, while the wheel speed sensor is sensitive to speed changes. High-definition surround-view camera images are large in data volume, requiring high data transmission bandwidth and consuming significant resources for raw image parsing and processing. Compared to single-sensor positioning, a multi-sensor combination theoretically yields more accurate and robust results. The combination of multiple sensor inputs ensures that at least one input is reliable and effective in all scenarios. High-definition surround-view cameras suffer from poor image quality in low light or in scenes with drastic lighting changes, and fisheye cameras exhibit severe distortion. Image quality is very poor in areas with significant distortion, making it impossible to directly provide an effective mapping of the real environment. Therefore, a multi-sensor positioning module is more applicable to specific scenarios. Accurate positioning results, used as regulatory input, can theoretically improve the performance of intelligent vehicle systems and provide a better user experience. Summary of the Invention

[0003] The present invention provides a fusion positioning data processing method for the solution verification stage. Targeting the application scenario of underground garages, the present invention comprehensively considers system costs, avoids the use of expensive vehicle-mounted sensors such as lidar in sensor selection, and also abandons solutions that require special modifications to the factory side. Taking into account the actual application effect of the system, the advantages of various sensors are integrated to ensure that at least one reliable input is provided under different circumstances in the solution verification stage.

[0004] The present invention provides a method for processing fused positioning data in a solution verification phase, comprising:

[0005] The data communication forwarding end process pre-processes the received multiple sensor data respectively, and encapsulates the pre-processed multiple sensor data respectively;

[0006] For each first-category sensor data that occupies a small number of bytes, the data communication forwarding end process uses a message queue to forward the encapsulated first-category sensor data to the data communication receiving end process; for each second-category sensor data that occupies a large number of bytes, the data communication forwarding end process uses a shared memory to forward the encapsulated second-category sensor data to the data communication receiving end process;

[0007] The data communication receiving end process performs dead reckoning processing on the received first type of sensor data and performs dedistortion processing on the received second type of sensor data;

[0008] The data communication receiving end process sends the results obtained after the dead reckoning processing and the results obtained after the dedistortion processing to the fusion positioning process in the form of ROS messages;

[0009] The fusion positioning process performs fusion positioning based on all received ROS messages.

[0010] Preferably, the data communication forwarding end process opens up threads for each of the received sensor data to perform pre-processing to remove abnormal values ​​of the data.

[0011] Preferably, the first category of sensor data includes wheel speed sensor data and IMU sensor data. When the data communication forwarding end process uses a message queue to forward the encapsulated first category sensor data, the key value of the message queue of each category of first sensor data is set to a corresponding preset fixed parameter.

[0012] Preferably, the second type of sensor data includes: surround view camera image data, and when the data communication forwarding end process forwards the packaged second type of sensor data in a shared memory manner,

[0013] The data communication forwarding end process writes the received second-category sensor data alternately into two different memories;

[0014] When writing to each memory is completed, the data communication forwarding end process sends a trigger message containing the timestamp corresponding to the image and the corresponding memory ID to the process communication receiving module of the data communication receiving end process through the message queue;

[0015] After receiving the trigger message, the process communication receiving module of the data communication receiving end process locks the corresponding memory area based on the memory ID, reads the data, and releases the lock after completing the data reading.

[0016] Preferably, for different types of first-category sensor data, the dead reckoning module of the data communication receiving end process reads the IMU data from the message queue of the process communication receiving module of the data processing process and pushes the IMU data into the IMU queue and pushes the wheel speed data into the wheel speed queue;

[0017] If the number of elements in the wheel speed queue is greater than 2, the dead reckoning module takes the first element from the wheel speed queue and copies the second element;

[0018] The dead reckoning module uses the timestamp of the first element of the queue to perform a binary search;

[0019] The dead reckoning module finds the first element in the IMU queue whose timestamp is greater than the timestamp of the first element in the wheel speed queue;

[0020] The dead reckoning module finds the previous element before the first element in the IMU queue;

[0021] The dead reckoning module reads the wheel speed pulse factor and correction parameters from the wheel speed queue, loads the dead reckoning model selected by the user, and performs dead reckoning using the previous element, wheel speed pulse factor, and correction parameters found in the MU queue.

[0022] The dead reckoning module sends the results obtained after dead reckoning processing to the fusion positioning process in ROS format.

[0023] Preferably, for the second type of sensor data, the fisheye image dedistortion module of the data communication receiving end process pushes the image data read from the memory of the process communication receiving module of the data processing process into the image queue;

[0024] If the number of elements in the wheel speed queue is greater than 2, the fisheye image dedistortion module takes the fisheye YUV image at the head of the queue from the image queue and parses the fisheye YUV image at the head of the queue into RGB format;

[0025] The fisheye image dedistortion module determines and extracts the camera image to be processed from the fisheye YUV image parsed into RGB format;

[0026] The fisheye image dedistortion module obtains the fisheye YUV dedistortion model selected by the user;

[0027] The fisheye image dedistortion module sends the extracted camera images to be processed to the fisheye YUV dedistortion model via ROS messages and establishes listeners corresponding to the number of cameras to be processed.

[0028] When the listener detects the camera image to be processed, it will call back and trigger the fisheye image dedistortion module to read the distortion parameters of the corresponding camera to be processed, and solve the dedistortion mapping table of the corresponding camera to be processed based on the read distortion parameters;

[0029] The fisheye image dedistortion module uses the fisheye YUV dedistortion model based on the dedistortion mapping table of the camera to be processed to dedistort the camera image to be processed;

[0030] The fisheye image dedistortion module sends the dedistortion results to the fusion positioning process in ROS format.

[0031] Preferably, the step of determining the camera image to be processed from the fisheye YUV image parsed into RGB format by the fisheye image dedistortion module includes:

[0032] If the user specifies the camera parameters to be processed, the fisheye image dedistortion module combines the camera parameters specified by the user to determine the camera image to be processed from the fisheye YUV image parsed into RGB format; or

[0033] If the user does not specify the camera parameters to be processed, the fisheye image dedistortion module will parse the entire fisheye YUV image in RGB format and determine it as the camera image to be processed specified by the user.

[0034] Preferably, the visual positioning module in the fusion positioning process 17 uses the dedistortion result obtained after the dedistortion processing to perform visual positioning, and then sends the visual positioning message to the EKF fusion positioning module. The EKF fusion positioning module outputs the fusion positioning result based on the visual positioning message and the track calculation result obtained after the track calculation processing, and outputs the fusion positioning result to the regulation and control simulation module in the fusion positioning process 17 in the form of UDP for processing.

[0035] The beneficial effects of the present invention are:

[0036] For the application scenario of underground garages, comprehensive consideration was given to system costs. Expensive on-board sensors such as lidar were avoided in sensor selection, and solutions requiring special modifications to the factory were discarded. Taking into account the actual application effect of the system and the advantages of each sensor, efforts were made to ensure at least one reliable input in different situations. The fusion positioning data processing and communication system in the verification phase of this solution mainly consists of ordinary IMU sensors, wheel speed sensors, high-definition fisheye cameras, data transmission equipment, local industrial computers, and data processing and communication software. In terms of communication, shared memory and message queues are used for camera input to ensure efficiency and security. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a framework diagram for integrating positioning data processing and communication in this embodiment;

[0038] Figure 2 This is a diagram showing the implementation framework of sensor process communication in this embodiment;

[0039] Figure 3 This is a framework diagram for wheel speed and IMU data processing in this embodiment;

[0040] Figure 4 This is a framework diagram of image data processing in this embodiment. DETAILED DESCRIPTION

[0041] Embodiments of the present disclosure are described herein. However, it should be understood that the disclosed embodiments are merely examples and that other embodiments may take various and optional forms. The drawings are not necessarily drawn to scale and may exaggerate or minimize some features to illustrate details of specific components. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a representative basis for teaching those skilled in the art to use the present invention in various ways. As will be understood by those of ordinary skill in the art, the various features shown and described with reference to any one of the figures may be combined with features shown in one or more of the other figures to produce embodiments that are not explicitly shown or described. The combinations of features shown provide representative embodiments for typical applications. However, various combinations and variations of features consistent with the teachings of the present disclosure may be expected for specific applications or implementations.

[0042] The present invention provides a fusion positioning data processing method in the scheme verification stage. For the application scenario of underground garages, the system cost is comprehensively considered, and the use of expensive vehicle-mounted sensors such as lidar is avoided in the selection of sensors. The solution that requires special modification of the factory side is also abandoned. Taking into account the actual application effect of the system, the advantages of various sensors are integrated to ensure that there is at least one reliable input in different situations. The fusion positioning data processing and communication system in the scheme verification stage is mainly composed of ordinary IMU sensors, wheel speed sensors, high-definition fisheye cameras, data transmission equipment, local industrial computers and data processing and communication software. In terms of communication, shared memory and message queue methods are used for camera input to ensure efficiency and safety.

[0043] The raw sensor data goes from the sensor to the switch first, and then to the industrial computer via the crystal head network cable. The event triggering method is used. When the data communication forwarding end process of the industrial computer receives the relevant data, it is pre-processed. According to the time sequence judgment of the two frames before and after, the data range statistics within a period of time, etc., it is filtered according to the set threshold to obtain sensor data with normal time series. In order to keep the frequency of each sensor data basically consistent with the frequency of the raw data sent by the device, for different sensor data, the data communication forwarding end process opens up a thread for processing. Simple pre-processing is completed in this step, mainly including the elimination of data anomalies and time conversion.

[0044] After removing outliers, each thread encapsulates the data and prepares for process communication. The data communication forwarding end process and the data communication forwarding end process use message queues for process communication for wheel speed data and IMU data, which occupy a small number of bytes. The creation of message queues for different data is managed through custom class objects. When creating this class object, the key value of the message queue generated is a fixed parameter to ensure consistency between the receiving end and the current forwarding end. The data communication forwarding end process and the data communication forwarding end process use shared memory to read and write surround view camera data, which occupies a large number of bytes. However, the issue of efficient collaboration between the sending and reading ends needs to be resolved, and the timing of reading and writing needs to be resolved. To solve this problem, two pieces of memory are opened up, and the writing and reading operations are performed in a loop, avoiding the low reading and writing efficiency of only one piece of memory and the resource consumption caused by multiple pieces of memory; the message queue is used for triggering. After the forwarding end completes the memory write operation, the customized shared memory image related information is forwarded through the message queue. After receiving the image message, the receiving end indicates that the memory write is completed, locks the memory area, reads the relevant data, and then releases the lock. The memory area after being read is then written again, solving the coordination problem between the writing end and the reading end of the shared memory.

[0045] The processing content of the data communication forwarding end process mainly includes the track calculation of the wheel speed sensor and IMU, the dedistortion of the surround view image, and the message assembly after the data processing is completed, which is sent to the fusion positioning process through ROS. The EKF fusion positioning module 18 in the fusion positioning process outputs the fusion to the regulation and control simulation module 20 in UDP mode for joint debugging after the fusion is completed. This regulation and control simulation module 20 can be expanded by configuring the camera to be processed, the track calculation data source, the track calculation model and the camera dedistortion model.

[0046] Regarding wheel speed and IMU data communication, the data communication forwarding process manages a message queue through a custom class object. During object initialization, a message queue interface consistent with the forwarding process is created based on constant key values. Upon receiving IMU and wheel speed sensor data, the forwarding process creates separate threads to retrieve the data from the message queue and place it in the corresponding containers. To prevent simultaneous pushes from different threads into the same container and potential confusion, locks are acquired and released where possible. Regarding data processing, the thread processing wheel speed data retrieves elements promptly, provided the data container is not empty. Based on the wheel speed time, the IMU data container is iterated over and, using a custom lambda expression, the IMU data closest to the wheel speed time is found. Leveraging the wheel speed sensor's high displacement accuracy and the IMU's high rotation accuracy, these data are used as input for dead reckoning, effectively combining the strengths of different sensors. During dead reckoning, different models can be selected based on the user's preferences, and odometry results are automatically generated in TUM format, making it easy to compare different models and input sources to determine the optimal one.

[0047] In terms of image process communication, the image thread message queue receives relevant messages, indicating that the memory has been written and can be retrieved. In order to prevent the situation where the data written during the previous program run is not fully retrieved, a custom class object is used to manage the shared memory. The parameters of the shared memory are constants during initialization. During runtime, the flag bit passed in determines whether the memory data manipulation needs to be reset when it is first created, as well as whether it can only read or write. After reading the memory data, the lock on the memory is released to facilitate writing, which greatly improves the efficiency and security of the program. In terms of image data processing, users can select the processing of the four-camera data in the configuration file, including: 1. Which cameras to process. For the positioning module, in order to save computing resources, only the front or rear cameras can be processed, of course, all camera images can also be processed; 2. Fisheye images have large areas with poor image quality and cannot establish a good mapping with the real environment. Dedistortion is required. The user can configure which model to use for dedistortion.

[0048] In the data communication forwarding process, how to send the processed data to other required modules? In order to improve the efficiency of the verification phase, limited resources and energy are focused on solution and algorithm improvements. With the help of the ROS system, which is the most commonly used system in the robotics field, the messages are encapsulated into the standard ROS message format. The required modules only need to listen to the relevant topics, eliminating a lot of interface definition and coordination work, which can significantly improve the efficiency of the verification phase.

[0049] During the fusion positioning process, the visual positioning module receives the dedistorted ROS image information and then sends the positioning results to the EKF fusion positioning module in the form of ROS messages. The EKF fusion positioning module also receives ROS messages for wheel speed and IMU track calculation results. The fusion positioning results are given to the regulation and control simulation module in the form of UDP.

[0050] Figure 1 This describes the overall framework for fused positioning data processing and communication. Multi-sensor raw data 2 is transmitted from switch 1 via the RTSP protocol to the local industrial computer's data preprocessing process 3. Data preprocessing process 3 parses and extracts the data (wheel speed, IMU, and fisheye camera images) transmitted from the switch according to a predefined data structure 4. Abnormal data is then removed 5. After simple filtering, the data is forwarded via message queues or shared memory for process communication 6.

[0051] The process communication receiving module of the data processing process 7 receives or reads the data 8 and then passes the wheel speed and IMU data to the dead reckoning module. The dead reckoning module reads the track input configuration 9, selects the dead reckoning model 10, performs the dead reckoning 11 based on these configurations and inputs, and sends the result 12 in the form of ROS.

[0052] At the same time, the fisheye image dedistortion module of the data processing process 7 reads the distortion camera configuration 13 and dedistortion model configuration 14 selected by the user from the process communication receiving module, then performs fisheye image dedistortion 15, and finally sends the dedistorted image 16 in the form of ROS.

[0053] The visual positioning module 19 in the fusion positioning process 17 subscribes to the ROS image for visual positioning, and the EKF fusion positioning module 18 in the fusion positioning process 17 subscribes to the track calculation message and the visual positioning message to output the fusion positioning result, and gives the positioning result to the regulation and control simulation module 20 in the form of UDP.

[0054] Figure 2The specific implementation of image communication between two processes is described in detail. First, the data communication receiving process 21 is started. It creates shared memory and a message queue object 22 according to a predetermined class. The parameters of the created objects are constant member variables and cannot be modified externally. The data communication receiving process 21 is started first to determine whether there is any memory area that has not been read since the last program execution ended 23. If the memory has not been read completely since the last program execution ended, the data in the memory area is deleted. Then, the data communication forwarding process 29 is waited for to send a trigger signal 25. If the memory ID transmitted in the form of a message queue is received, it is verified 26. If the ID verification fails, an error is reported 27. If it passes, the memory is locked, the data is read, and the lock is released 28. The data communication forwarding process 29 is started and two shared memories 32 are created that can only be written to. This is done to balance efficiency and security. That is, while one memory is writing, the other memory, which has already been written to, can be read. After the memory is written to, the memory ID and image time information are sent to the message queue 34. The receiving end receives the message, indicating that the memory is written and can be read.

[0055] Figure 3 The processing framework of the surround-view camera image data by the data communication receiving end process 21 is described in detail. The surround-view camera image data read from the shared memory by the fisheye image dedistortion module in the data communication receiving end process 21 is pushed into the image queue 35. Before image processing, it is determined whether the image queue is empty 36. If it is empty, it continues to wait for the image to enter 37. If it is not empty, it takes out the fisheye YUV image at the head of the queue from the image queue 38, parses the image into RGB format, and obtains the camera parameters to be processed (corresponding to the camera parameters specified by the user) Figure 1 In the dedistortion camera configuration 13, it is specified whether the images of the front, rear, left, right, or a few cameras need to be processed. The parameter settings here are mainly based on the usage scenario and computing power. If the specified camera parameters to be processed are read 41, they are processed as specified 42. If not, all camera images are processed by default 43. Then the corresponding camera images are extracted 44. Next, the extracted images are sent to the fisheye image dedistortion module 45 via ROS messages. The fisheye image dedistortion module 45 receives the dedistortion model selected by the user 46, establishes a listener corresponding to the number of cameras to be processed 47, and calls back 48 when a fisheye image of a specific subject is received. The distortion parameters corresponding to different cameras are read 49, and then a distortion mapping table is calculated 50. With the mapping table, the fisheye image can be dedistorted 51. Finally, the dedistorted image is sent as ROS 52.

[0056] Figure 4The dead reckoning framework is described in detail. IMU data transmitted from the message queue is pushed into the IMU queue 53, and wheel speed data is pushed into the wheel speed queue 54. The size of both queue elements is checked to see if they are both greater than a threshold 55. If not, the loop exits 56. If they are, the first element in the wheel speed queue is retrieved and the second element is copied 57. A binary search is then performed on the IMU using the timestamp of the first element in the wheel speed queue 58. If a matching element is found in the IMU queue 59, the IMU data is retrieved 61. Otherwise, the loop exits. The wheel speed pulse factor and correction parameters are read 62, and the user-selected dead reckoning model is loaded 63. Finally, dead reckoning is performed 64.

[0057] Although exemplary embodiments have been described above, it is not intended that these embodiments describe all possible forms encompassed by the claims. The words used in the specification are descriptive rather than restrictive, and it should be understood that various changes may be made without departing from the spirit and scope of the present disclosure. As previously described, the features of the various embodiments may be combined to form further embodiments of the present invention that may not be explicitly described or shown. Although various embodiments may have been described as providing advantages or being superior to other embodiments or prior art implementations in terms of one or more desired characteristics, those of ordinary skill in the art recognize that, depending on the specific application and implementation, one or more features or characteristics may be compromised to achieve the desired overall system properties. These properties may include, but are not limited to, cost, strength, durability, life cycle cost, appearance, size, manufacturability, functional robustness, and the like. Thus, embodiments described as being less desirable than other embodiments or prior art implementations in terms of one or more characteristics are not outside the scope of the present disclosure and may be expected to be used in specific applications.

Claims

1. A method for processing fused positioning data in the solution verification phase, characterized in that: include: The data communication forwarding end process pre-processes the received multiple sensor data respectively, and encapsulates the pre-processed multiple sensor data respectively; For each first-category sensor data that occupies a small number of bytes, the data communication forwarding end process uses a message queue to forward the encapsulated first-category sensor data to the data communication receiving end process; for each second-category sensor data that occupies a large number of bytes, the data communication forwarding end process uses a shared memory to forward the encapsulated second-category sensor data to the data communication receiving end process; The data communication receiving end process performs dead reckoning processing on the received first type of sensor data and performs dedistortion processing on the received second type of sensor data; The data communication receiving end process sends the results obtained after the dead reckoning processing and the results obtained after the dedistortion processing to the fusion positioning process in the form of ROS messages; The fusion positioning process performs fusion positioning based on all received ROS messages.

2. The method for processing fused positioning data in the solution verification phase according to claim 1, characterized in that: The data communication forwarding end process opens up threads for each of the multiple sensor data received to perform preprocessing to remove abnormal values ​​from the data.

3. The method for processing fused positioning data in the solution verification phase according to claim 1, characterized in that: The first category of sensor data includes wheel speed sensor data and IMU sensor data. When the data communication forwarding end process uses a message queue to forward the encapsulated first category sensor data, the key value of the message queue of each category of first sensor data is set to a corresponding preset fixed parameter.

4. The method for processing fused positioning data in the solution verification phase according to claim 1, characterized in that: The second type of sensor data includes: surround view camera image data. When the data communication forwarding end process forwards the packaged second type of sensor data using shared memory, The data communication forwarding end process writes the received second-category sensor data alternately into two different memories; When writing to each memory is completed, the data communication forwarding end process sends a trigger message containing the timestamp corresponding to the image and the corresponding memory ID to the process communication receiving module of the data communication receiving end process through the message queue; After receiving the trigger message, the process communication receiving module of the data communication receiving end process locks the corresponding memory area based on the memory ID, reads the data, and releases the lock after completing the data reading.

5. The method for processing fused positioning data in the solution verification phase according to claim 3, characterized in that: For different types of first-category sensor data, the dead reckoning module of the data communication receiving end process reads the IMU data from the message queue of the process communication receiving module of the data processing process and pushes it into the IMU queue, and pushes the wheel speed data into the wheel speed queue; If the number of elements in the wheel speed queue is greater than 2, the dead reckoning module takes the first element from the wheel speed queue and copies the second element; The dead reckoning module uses the timestamp of the first element of the queue to perform a binary search; The dead reckoning module finds the first element in the IMU queue whose timestamp is greater than the timestamp of the first element in the wheel speed queue; The dead reckoning module finds the previous element before the first element in the IMU queue; The dead reckoning module reads the wheel speed pulse factor and correction parameters from the wheel speed queue, loads the dead reckoning model selected by the user, and performs dead reckoning using the previous element, wheel speed pulse factor, and correction parameters found in the MU queue. The dead reckoning module sends the results obtained after dead reckoning processing to the fusion positioning process in ROS format.

6. The method for processing fused positioning data in the solution verification phase according to claim 4, characterized in that: For the second type of sensor data, the fisheye image dedistortion module of the data communication receiving end process pushes the image data read from the memory of the process communication receiving module of the data processing process into the image queue; If the number of elements in the wheel speed queue is greater than 2, the fisheye image dedistortion module takes the fisheye YUV image at the head of the queue from the image queue and parses the fisheye YUV image at the head of the queue into RGB format; The fisheye image dedistortion module determines and extracts the camera image to be processed from the fisheye YUV image parsed into RGB format; The fisheye image dedistortion module obtains the fisheye YUV dedistortion model selected by the user; The fisheye image dedistortion module sends the extracted camera images to be processed to the fisheye YUV dedistortion model via ROS messages and establishes listeners corresponding to the number of cameras to be processed. When the listener detects the camera image to be processed, it will call back and trigger the fisheye image dedistortion module to read the distortion parameters of the corresponding camera to be processed, and solve the dedistortion mapping table of the corresponding camera to be processed based on the read distortion parameters; The fisheye image dedistortion module uses the fisheye YUV dedistortion model based on the dedistortion mapping table of the camera to be processed to dedistort the camera image to be processed; The fisheye image dedistortion module sends the dedistortion results to the fusion positioning process in ROS format.

7. The method for processing fused positioning data in the solution verification phase according to claim 6, characterized in that: The fisheye image dedistortion module determines the camera image to be processed from the fisheye YUV image parsed into RGB format, including the following steps: If the user specifies the camera parameters to be processed, the fisheye image dedistortion module combines the camera parameters specified by the user to determine the camera image to be processed from the fisheye YUV image parsed into RGB format; or If the user does not specify the camera parameters to be processed, the fisheye image dedistortion module will parse the entire fisheye YUV image in RGB format and determine it as the camera image to be processed specified by the user.

8. The method for processing fused positioning data in the solution verification phase according to claim 1, characterized in that: The visual positioning module in the fusion positioning process uses the dedistortion result obtained after dedistortion processing to perform visual positioning, and then sends the visual positioning message to the EKF fusion positioning module. The EKF fusion positioning module outputs the fusion positioning result based on the visual positioning message and the track calculation result obtained after track calculation processing, and outputs the fusion positioning result to the regulation and control simulation module in the fusion positioning process in the form of UDP for its processing.

Images