Obstacle avoidance verification method and device
By acquiring and correcting simulated binocular images in diverse simulation scenarios, a fused depth map that integrates with the real environment is generated. This solves the efficiency and environmental coverage problems of obstacle avoidance algorithm training and verification in existing technologies, and achieves high-performance and highly robust obstacle avoidance capabilities.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XIAN LINGKONG ELECTRONICS TECH CO LTD
- Filing Date
- 2026-04-29
- Publication Date
- 2026-06-05
AI Technical Summary
Existing methods for training and validating binocular camera obstacle avoidance algorithms have limitations in terms of efficiency, cost, cycle time, and environmental coverage, making it difficult to meet the needs of intelligent agents for high-performance and robust obstacle avoidance capabilities.
By conducting simulation tests in diverse simulation scenarios, simulated binocular images are acquired. Through calibration and fusion of depth map models, a fused depth map that combines with the real environment is generated and converted into a video stream, which is then pushed to a streaming media server for obstacle avoidance algorithm verification.
It reduces the cost and time required for training and validating obstacle avoidance algorithms, covers extreme weather and complex dynamic scenarios, and improves the generalization ability and robustness of obstacle avoidance algorithms.
Smart Images
Figure CN122157211A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of autonomous obstacle avoidance technology, and in particular to an obstacle avoidance verification method and apparatus. Background Technology
[0002] With the rapid development of artificial intelligence and automation technologies, autonomous navigation and obstacle avoidance capabilities have become core requirements for the safe operation of intelligent agents such as drones, unmanned vehicles, and autonomous driving systems in complex and dynamic environments. Among these, obstacle avoidance technology based on visual perception has attracted significant attention due to its advantages such as rich information and low cost. In particular, the method of acquiring depth information using binocular cameras is one of the mainstream technical approaches for achieving environmental perception and obstacle detection. This technology calculates the depth map of the scene by simulating the parallax principle of the human eye, thereby identifying obstacles and planning safe paths.
[0003] Currently, there are three main methods for training and validating obstacle avoidance algorithms using binocular cameras: First, offline training based on limited samples. Training samples come from publicly available internet datasets or images of specific scenes captured by binocular cameras in controlled / local environments. These images need to be labeled before being used to train supervised learning models, resulting in low efficiency. Furthermore, these images have a narrow coverage area, making it difficult to reflect the extreme diversity (e.g., cluttered backgrounds, reflections from special materials, small obstacles) and dynamic complexity (e.g., moving objects, sudden changes in lighting) of the real world. This leads to insufficient generalization ability and poor robustness of the model when facing unseen complex scenes.
[0004] The second method is field verification based on physical platforms. After the algorithm is developed, it is tested in the field on a real physical platform (such as a car or drone prototype) equipped with a binocular camera. By running the algorithm in a real environment, its obstacle avoidance behavior is observed and its performance is evaluated. However, physical testing is costly and risky, so developers usually tend to reduce the number of tests and simplify the test scenarios, making it difficult to conduct comprehensive and sufficient verification, especially failing to cover all potential edge cases and dangerous operating conditions.
[0005] Thirdly, there is scenario verification in constrained environments. This method is usually carried out in pre-selected, relatively simple static or semi-static environments (such as open indoor spaces or flat outdoor roads). However, test sites that accurately simulate adverse weather conditions such as low light at night, strong backlight, rain, snow, fog, and haze are expensive, time-consuming, and extremely inflexible, and still cannot reproduce the diverse environments that are widely present in the real world. Summary of the Invention
[0006] This application provides an obstacle avoidance verification method and apparatus, which solves the problem that existing training and verification methods for binocular camera obstacle avoidance algorithms have limitations in terms of efficiency, cost, cycle and environmental coverage, making it difficult to meet the current demand of intelligent agents for high-performance and robust obstacle avoidance capabilities.
[0007] In a first aspect, embodiments of this application provide an obstacle avoidance verification method, including: In diverse simulation scenarios, simulation tests are conducted using motion models with mounted camera components to acquire multiple sets of simulated binocular images. The simulated binocular image is corrected, and its simulated depth map is determined by the disparity of the simulated binocular image; Construct a depth map model and train it with real-world depth maps collected from the real environment to obtain a real-world depth map model; The simulated depth map is input into the physical depth map model to fuse real-world features and obtain a fused depth map. The fused depth map is converted into a video stream and pushed to the streaming media server. The obstacle avoidance algorithm obtains the fused depth map by accessing the streaming media server to verify the obstacle avoidance algorithm.
[0008] In conjunction with the first aspect, in one possible implementation, the simulation test is conducted in a diversified simulation scenario using a motion model with mounted camera components to acquire multiple sets of simulated binocular images, including: Based on testing requirements, a diverse simulation scenario is constructed using 3D simulation software, and multiple motion models are defined within it. Enable multiple camera components in the 3D simulation software, configure their camera intrinsics, and set the camera spacing baseline; The camera components are mounted onto the motion model, and the motion model is driven to move according to a preset trajectory to acquire multiple sets of simulated binocular images.
[0009] In conjunction with the first aspect, in one possible implementation, the correction of the simulated binocular image includes: Distortion correction is performed on the simulated binocular images using distortion coefficients to eliminate image distortion; and / or, the left and right camera components are corrected to the same imaging plane using a rotation matrix so that the x-axis of the corrected camera component imaging plane remains horizontal with the camera spacing baseline.
[0010] In conjunction with the first aspect, in one possible implementation, determining the simulated depth map using the disparity of the simulated binocular images includes: The disparity value is calculated based on the projection coordinates of the simulated binocular image; The simulated depth map corresponding to the simulated binocular image is determined by using camera intrinsic parameters, camera spacing baseline, and disparity value.
[0011] In conjunction with the first aspect, in one possible implementation, training the object depth map model using real-world depth maps collected from the actual environment includes: Add obstacle feature points to the real-world depth map captured by the physical camera in the real environment; The actual depth values between the real camera and the obstacle, along with the labeled real-scene depth map, are input into the depth map model for training. After multiple rounds of optimization, the real-scene depth map model is obtained.
[0012] In conjunction with the first aspect, in one possible implementation, the step of inputting the simulated depth map into the physical depth map model to fuse real-world features and obtain a fused depth map includes: The simulated depth map is input into the trained physical depth map model. The physical depth map model is used to supplement the obstacle feature points of the simulated depth map, and the depth values corresponding to the pixels are corrected and the display colors are completed to obtain the fused depth map.
[0013] In conjunction with the first aspect, in one possible implementation, the step of converting the fused depth map into a video stream and pushing it to a streaming media server, and the obstacle avoidance algorithm obtaining the fused depth map by accessing the streaming media server for obstacle avoidance algorithm verification, includes: The fused depth map is encapsulated frame by frame into a standardized video stream; The standardized video stream is pushed to the streaming media server via video push, and an independent data channel is configured for each obstacle avoidance algorithm to achieve the sharing and distribution of the fused depth map.
[0014] Secondly, embodiments of this application provide an obstacle avoidance verification device, characterized in that it includes: The acquisition module is used to conduct simulation tests in diverse simulation scenarios by using motion models with mounted camera components to acquire multiple sets of simulated binocular images. The correction module is used to correct the simulated binocular image and determine its simulated depth map based on the disparity of the simulated binocular image. The training module is used to build a depth map model. It is trained with real-world depth maps collected from the real environment to obtain a real-world depth map model. The fusion module is used to input the simulated depth map into the physical depth map model to fuse real-world features and obtain a fused depth map. The verification module is used to convert the fused depth map into a video stream and push it to the streaming media server. The obstacle avoidance algorithm obtains the fused depth map by accessing the streaming media server to verify the obstacle avoidance algorithm.
[0015] Thirdly, embodiments of this application provide an apparatus comprising: a processor; a memory for storing processor-executable instructions; wherein, when the processor executes the executable instructions, it implements the method as described in the first aspect or any possible implementation of the first aspect.
[0016] Fourthly, embodiments of this application provide a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium including storage for storing a computer program or instructions that, when executed, cause the method described in the first aspect or any possible implementation of the first aspect to be implemented.
[0017] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: This application embodiment uses a motion model with a mounted camera component to simulate and acquire multiple sets of simulated binocular images. This allows for the rapid acquisition of simulated binocular images covering various test scenarios, different motion trajectories, and different obstacle types, avoiding the high cost and high risk of physical testing. By fusing depth maps, the simulated binocular images can be combined with real-world depth maps. This not only compensates for the shortcomings of simulated binocular depth maps in terms of detail texture and real-world environmental noise, but also solves the problems of long acquisition cycles and limited scene coverage of real-world depth maps. The generated fused depth map is closer to the visual perception effect of the real world. By encapsulating the fused depth map into a standardized video stream and pushing it to a streaming media server, it can support the simultaneous access and verification of multiple obstacle avoidance algorithms, greatly improving the efficiency and flexibility of obstacle avoidance algorithm verification. This effectively solves the problem that existing binocular camera obstacle avoidance algorithm training and verification methods are limited in terms of efficiency, cost, cycle, and environmental coverage, making it difficult to meet the current intelligent agent's demand for high-performance and highly robust obstacle avoidance capabilities. This not only reduces the cost and cycle of obstacle avoidance algorithm training and verification, but also covers edge cases such as extreme weather and complex dynamic scenarios, effectively improving the generalization ability and robustness of obstacle avoidance algorithm verification. Attached Figure Description
[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 A flowchart of an obstacle avoidance verification method provided in an embodiment of this application; Figure 2 An example diagram of the simulated depth map provided in the embodiments of this application; Figure 3An example diagram of the fusion depth map provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of an obstacle avoidance verification device provided in an embodiment of this application. Detailed Implementation
[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0021] The following description of some technologies involved in the embodiments of this application is provided to aid understanding and should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, some descriptions of well-known functions and structures are omitted in the following description.
[0022] Figure 1 This is a flowchart of an obstacle avoidance verification method provided in an embodiment of this application, including steps 101 to 105. Figure 1 This is merely one execution order shown in the embodiments of this application and does not represent the only execution order for an obstacle avoidance verification method. The execution order can be adjusted to achieve the desired final result. Figure 1 The steps shown can be performed in parallel or in reverse order.
[0023] Step 101: In a diversified simulation scenario, simulation tests are conducted using motion models with mounted camera components to acquire multiple sets of simulated binocular images. In this embodiment, a diversified simulation scenario is constructed using 3D simulation software based on testing requirements, and multiple motion models are defined within it; multiple camera components in the 3D simulation software are enabled, and their intrinsic camera parameters are configured, and a camera spacing baseline is set; the camera components are mounted onto the motion models respectively, and the motion models are driven to move according to a preset trajectory to acquire multiple sets of simulated binocular images.
[0024] Specifically, firstly, based on the testing requirements of obstacle avoidance algorithms, diverse simulation scenarios with varying terrains, obstacle distributions, and weather conditions are built using 3D simulation software (such as Unity3D or Unreal Engine). Within these diverse simulation scenarios, various types of motion models, such as cars and drones, are defined to adapt to different test scenarios. Then, two sets of camera components in the 3D simulation software are activated, configured with intrinsic camera parameters consistent with those of physical cameras, including focal length, field of view, principal point, and distortion coefficients. Simultaneously, a baseline for the camera spacing between the two sets of camera components is set according to testing requirements. The configured camera components are then mounted onto different motion models, allowing the camera components to move with the motion models. Finally, each motion model is driven to move within the diverse simulation scenarios according to a preset motion trajectory for obstacle avoidance algorithm testing. Simultaneously, the camera components are controlled to acquire images at a set frame rate to obtain multiple sets of simulated binocular images adapted to different test scenarios.
[0025] Furthermore, the construction of diversified simulation scenarios needs to cover common and extreme scenarios in the actual application of obstacle avoidance algorithms, including terrain environments such as flat roads, rugged terrain, and urban building areas; obstacle distribution environments such as pedestrians, vehicles, and roadblocks; and weather and lighting environments such as sunny days, rainy days, foggy days, and nighttime, to ensure that the collected simulated binocular images have scene diversity. The definition of the motion model needs to match the applicable carrier of the obstacle avoidance algorithm, and its motion parameters can be adjusted according to the test requirements. The camera intrinsic parameter configuration of the camera component needs to be completely consistent with the camera parameters of the actual physical camera used in the application. The camera spacing baseline can be adapted to the depth detection requirements of the test scenario, and the two sets of camera components need to be arranged in parallel. The preset trajectory of the motion model needs to cover various motion states such as straight driving, turning, and speed change. The set frame rate of the camera component needs to meet the real-time detection requirements of the obstacle avoidance algorithm. During the motion model's movement along the preset trajectory, the camera component continuously collects images from the left and right perspectives and stores them in groups to form multiple sets of simulated binocular images. Each set of simulated binocular images contains two simulated binocular images, one on the left and one on the right, from the same moment and perspective.
[0026] Step 102: Correct the simulated binocular image and determine its simulated depth map using the disparity of the simulated binocular image. In this embodiment, distortion correction is performed on the simulated binocular image using distortion coefficients to eliminate image distortion; and / or, the left and right camera components are corrected to the same imaging horizontal plane using a rotation matrix, so that the x-axis of the corrected camera component imaging horizontal plane remains horizontal with the camera spacing baseline.
[0027] Specifically, the acquired simulated binocular images are first distorted based on the radial and tangential distortion coefficients of the camera components to eliminate image distortion caused by lens optical characteristics. Then, by calculating the stereo correction rotation matrix, the left and right sets of camera components are corrected to the same imaging plane so that the x-axis of the imaging plane of the corrected camera components remains horizontal with the camera spacing baseline, thus completing the stereo correction of the simulated binocular images.
[0028] Furthermore, distortion correction is based on the radial and tangential distortion coefficients obtained from the camera component calibration. These coefficients are then substituted into the distortion correction formula to correct the coordinates of each pixel in the simulated binocular image, eliminating distortions such as radial warp and tangential shift, thereby restoring the realistic scene imaging effect and ensuring the geometric accuracy of a single simulated binocular image. Specifically: , .
[0029] In the formula, The x-axis coordinates of the simulated binocular image before distortion correction. The vertical axis coordinates of the simulated binocular image before distortion correction are given. This represents the horizontal axis coordinate of the simulated binocular image after distortion correction. This represents the mid-ordinate of the simulated binocular image after distortion correction. This is a rotation matrix used to describe directional relationships. ) is the radial distortion coefficient, ( () represents the tangential distortion coefficient.
[0030] Specifically, stereo calibration first calculates the rotation matrix based on the relative poses of the left and right camera components. This rotation matrix is then used to perform a spatial geometric transformation on the left and right simulated binocular images, ensuring that the stereo-calibrated left and right camera components are on the same imaging plane. Furthermore, the horizontal x-axis of this imaging plane is perfectly parallel to the baseline connecting the optical centers of the two cameras, achieving row alignment of the left and right simulated binocular images and correcting vertical pixel misalignment. The specific formula is as follows: , , .
[0031] In the formula, , These are the points where the camera spacing baseline intersects the imaging plane, and are called poles. Here, is the z-axis coordinate point after stereo correction, T is the coordinate parallel transformation from the left camera component to the right camera component, and t is the translation vector. The rotation matrix is calculated from the intrinsic parameter matrices of the left and right camera components.
[0032] After stereo calibration, the coordinates of the two camera components are rotated to the same imaging plane, and the projection effect satisfies the following relationship: , .
[0033] In the formula, B is the baseline of the camera spacing. The horizontal coordinates of the simulated binocular image before correction. Z represents the camera focal length, and Z represents the depth value of the simulated stereo image. These are the coordinates of the points on the horizontal axis (x-axis) of the disparity map. The coordinates of the points on the vertical axis (y-axis) of the disparity map are given. This represents the horizontal axis coordinate of the simulated binocular image after distortion correction.
[0034] In this embodiment, the disparity value is calculated based on the projection coordinates of the simulated binocular image; the simulated depth map (e.g., ...) corresponding to the simulated binocular image is determined using camera intrinsic parameters, camera spacing baseline, and the disparity value. Figure 2 The image shown is an example of one of the simulated depth maps.
[0035] Specifically, after completing distortion correction and stereo correction, the corresponding pixels of the left and right simulated binocular images are matched in the same horizontal row, the disparity value of each pixel is calculated and a disparity map is generated, and then the disparity value is converted into the depth value of the corresponding scene point according to the depth calculation formula. Finally, a simulated depth map of the same size as the simulated binocular image is generated, and the depth value of each pixel represents the actual distance from the scene point to the camera component.
[0036] For example, the depth calculation formula is as follows: ,in, .
[0037] In the formula, For depth value, B is the camera focal length, B is the camera distance baseline, and d is the parallax value. The x-axis coordinates of the disparity map are... The coordinates of the vertical axis of the disparity map are given.
[0038] In addition, the generated simulated depth map can be validated to remove abnormal pixels whose depth values exceed a reasonable range (such as less than 0.5 meters or greater than 50 meters) and mark them as invalid areas so that subsequent physical depth map models can be processed accordingly.
[0039] Step 103: Construct a depth map model and train it with real-world depth maps collected from the real environment to obtain a real-world depth map model. In this embodiment, obstacle feature points are labeled on the real-world depth maps collected by the real camera in the real environment; the actual depth values between the real camera and the obstacles and the labeled real-world depth maps are input into the depth map model for training, and the real-world depth map model is obtained after multiple rounds of optimization.
[0040] Specifically, a depth map model adapted for deep feature learning is built. Then, obstacle feature points are labeled on real-world depth maps collected by physical stereo cameras in various real-world environments. The actual depth values corresponding to each obstacle feature point of the physical camera (including visible light stereo cameras and infrared stereo cameras) are extracted. The labeled real-world depth maps and the matched actual depth values are input into the depth map model for training and learning. Multiple rounds of optimization are completed by iteratively adjusting the model parameters to obtain a physical depth map model with obstacle feature recognition and depth value fitting capabilities.
[0041] Furthermore, the acquisition of real-scene depth maps is accomplished using a physical binocular camera with camera intrinsic parameters consistent with those of the camera components in diverse simulation scenarios. This covers different weather and lighting conditions, including sunny days, rainy days, and nighttime, as well as real-world scenes with varying obstacle distributions such as pedestrians, vehicles, roadblocks, and buildings, ensuring the diversity and comprehensiveness of the training data. Obstacle feature point annotation precisely marks key locations such as the edges, contours, and centers of various obstacles in the real-scene depth map. Simultaneously, the ranging function of the physical camera is used to extract and record the actual depth value between each obstacle feature point and the imaging plane of the physical camera. This results in multiple sets of training data with one-to-one correspondence between the real-scene depth map, obstacle feature points, and actual depth values, which are then proportionally divided into training and validation sets.
[0042] The depth map model is built based on the feature extraction and data fitting requirements of binocular depth maps, ensuring the model's ability to learn the correlation between pixel features and depth values in the binocular depth map. Training data is then input into the depth map model for iterative training. Forward propagation learns the mapping relationship between pixel features of the binocular depth map (i.e., real-world depth map) and actual depth values, while backpropagation continuously adjusts the model weights and bias parameters. After each training round, a validation set is used to verify the feature recognition accuracy and depth value fitting precision of the depth map model. The model structure and parameters are optimized to address issues such as recognition bias and depth value error. Through multiple rounds of training and tuning, the depth map model acquires stable cross-scene obstacle feature recognition and depth value correction capabilities, ultimately resulting in a real-world depth map model that meets the requirements of simulation depth map fusion.
[0043] For example, the depth map model is DivFusion (an image fusion technique combining scene brightness deentanglement and texture-contrast enhancement). DivFusion maintains the original fusion main network structure and only adapts the input data format. During the training phase, the sample set used (i.e., the sample set composed of real-world depth maps) consists of visible light images and infrared images to improve the generalization ability of the trained real-world depth map model. The visible light images are color, with 3 channels, a size of 640×640, and dimensions [B, 3, 640, 640]. The infrared images are grayscale images, with 1 channel, a size of 640×640, and dimensions [B, 1, 640, 640], where B represents the batch size. To balance texture preservation and target saliency of the real-world depth images, a gradient preservation term and a structural similarity term are added to the loss function of the depth map model to construct a joint loss function, as follows: , in, , , .
[0044] In the formula, The loss function represents the depth map model (or physical depth map model). Represents structural similarity loss. Indicates strength loss. Represents gradient loss, , , The weights represent the intensity loss, structural similarity loss, and gradient loss, used to balance the impact of each loss term on model training. Their values range from (0,1). This represents the fused image output by the depth map model (or physical depth map model). This represents the input visible light image (i.e., the real-world depth map). The second modality image refers to the infrared image or depth map acquired based on the actual task. The structural similarity index measures the degree of similarity between two images in terms of brightness, contrast, and structure, ensuring that the fused image is structurally close to both the visible light image and the second modality image. This represents the gradient operator, used to extract edge and texture information from visible light images. L1 norm is used to measure the difference in pixel intensity between the fused image and the input visible light image, so that the fused image retains as much brightness and salient information as possible from the visible light image.
[0045] In this application, the AdamW (Adam optimizer with weight decay) optimizer is used when training the deep graph model. The learning rate is 1e-4, the number of training epochs is 200, the batch size (the number of samples input in each training session) is 16, and a cosine annealing strategy is used to decay the learning rate in order to improve the convergence stability in the later stages of training.
[0046] Step 104: Input the simulated depth map into the real-world depth map model to fuse real-world features and obtain a fused depth map. In this embodiment, the simulated depth map is input into the trained real-world depth map model. The real-world depth map model is used to supplement the obstacle feature points of the simulated depth map, correct the depth values corresponding to the pixels, and complete the display colors to obtain a fused depth map.
[0047] Specifically, the generated simulated depth map is input into the trained real-world depth map model. The obstacle feature recognition capability of the real-world depth map model is used to supplement missing or blurred obstacle feature points in the simulated depth map. Simultaneously, based on the learned patterns of actual depth values from the real-world depth map model, the depth values corresponding to each pixel in the simulated depth map are precisely corrected. Furthermore, the display colors of the simulated depth map are completed according to the mapping rules between the actual depth values of the real-world depth map and the display colors, thus fusing real-world features with the simulated depth map to obtain a fused depth map that highly matches the real environment (e.g., ...). Figure 3 The image shown is an example of one of the fused depth maps.
[0048] Furthermore, after the simulated depth map is completely input into the real-world depth map model pixel by pixel, the real-world depth map model first calls upon the learned multi-scene obstacle feature library to specifically supplement the simulated depth map for issues such as missing obstacle feature points, blurred outlines, and insufficient details caused by problems with simulation modeling accuracy and scene rendering. This restores the typical feature point distribution of obstacles in the real scene, ensuring that the feature information of the simulated depth map is consistent with the real environment. Subsequently, based on the correlation between actual depth values and scene features, the real-world depth map model verifies and corrects the depth values of each pixel in the simulated depth map point by point, eliminating depth value deviations caused by simulation calculations and scene simulations. This ensures that the corrected depth values accurately match the actual depth features of the corresponding scene and obstacles in the real scene. At the same time, the real-world depth map model retrieves the actual depth values and display color mapping rules configured during the training phase. Based on the corrected depth values of each pixel, it completes the corresponding display colors for the simulated depth map, ensuring that the fused depth map maintains consistency with the real-world depth map captured by the physical camera in terms of color representation. Through full-dimensional fusion processing including feature point supplementation, depth value correction, and display color completion, the final fused depth map retains the test diversity of the simulation scene while possessing the depth data characteristics and visual display effects of the real environment.
[0049] Step 105: The fused depth map is converted into a video stream and pushed to the streaming media server. The obstacle avoidance algorithm obtains the fused depth map by accessing the streaming media server for verification. In this embodiment, the fused depth map is encapsulated frame by frame into a standardized video stream; the standardized video stream is pushed to the streaming media server via video streaming, and an independent data channel is configured for each obstacle avoidance algorithm to achieve shared distribution of the fused depth map.
[0050] Specifically, the fused depth map generated according to the acquisition time sequence is serialized and encapsulated frame by frame, converted into a standardized video stream that conforms to the streaming media transmission standard, and then pushed to the streaming media server through a dedicated video push technology. Each obstacle avoidance algorithm to be verified is configured with an independent data acquisition channel to achieve secure sharing and efficient distribution of fused depth map video stream data, so that each obstacle avoidance algorithm can access the data to carry out verification tests.
[0051] Furthermore, the fused depth map is sorted according to its actual acquisition time sequence, and the frame data is encapsulated following the general encoding standards for streaming media transmission to generate a standardized video stream that can be transmitted and parsed in real time. Simultaneously, the frame rate and resolution of the video stream are kept consistent with the original fused depth map, meeting the real-time and integrity requirements of the obstacle avoidance algorithm. A low-latency, high-stability video streaming method is used to continuously push the encapsulated standardized video stream to the streaming media server. For each obstacle avoidance algorithm to be verified connected to the streaming media server, a separate, isolated dedicated data acquisition channel is configured. Each channel is independently allocated data transmission resources, with no data interaction or resource conflicts between channels, ensuring that the data acquisition operation of each obstacle avoidance algorithm does not affect other obstacle avoidance algorithms. After receiving the fused depth map video stream, the streaming media server continuously maintains real-time data updates and stable output. Each obstacle avoidance algorithm can pull and parse the fused depth map video stream data from the streaming media server as needed through its own dedicated independent data channel, and conduct full-process verification tests of the obstacle avoidance algorithm based on this data in diverse simulation scenarios.
[0052] This application's embodiments effectively compensate for the differences between purely simulated depth maps and real-world data by constructing a fused depth map that combines virtual and real elements. This provides obstacle avoidance algorithms with verification data that is both diverse in scenarios and closely resembles the real physical world. Through the flexible construction of simulated scenarios and the deep fusion of real-world features, this method not only reduces the cost and risk of testing obstacle avoidance algorithms in real-world environments but also covers test conditions that are difficult to reproduce in real-world scenarios, such as extreme weather and complex obstacle distributions. This significantly improves the comprehensiveness and robustness of obstacle avoidance algorithm verification. Simultaneously, standardized video stream delivery and the design of independent channels for multiple algorithms enable efficient sharing of verification resources and parallel testing, providing strong support for the rapid iteration and optimization of obstacle avoidance algorithms.
[0053] While this application provides the method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive labor. The order of steps listed in this embodiment is merely one possible execution order among many and does not represent the only execution order. In actual device or client product execution, the methods shown in this embodiment or the accompanying drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment).
[0054] like Figure 4 As shown in the figure, this application embodiment also provides an obstacle avoidance verification device 400. The device includes: a data acquisition module 401, a correction module 402, a training module 403, a fusion module 404, and a verification module 405, as detailed below.
[0055] The acquisition module 401 is used to perform simulation tests in diverse simulation scenarios by mounting a motion model of a camera component, in order to acquire multiple sets of simulated binocular images.
[0056] The correction module 402 is used to correct the simulated binocular image and determine its simulated depth map by the disparity of the simulated binocular image.
[0057] The training module 403 is used to construct a depth map model, which is trained with real-world depth maps collected from the real environment to obtain a real-world depth map model.
[0058] The fusion module 404 is used to input the simulated depth map into the physical depth map model to fuse real-world features and obtain a fused depth map.
[0059] The verification module 405 is used to convert the fused depth map into a video stream and push it to the streaming media server. The obstacle avoidance algorithm obtains the fused depth map by accessing the streaming media server to verify the obstacle avoidance algorithm.
[0060] Some modules in the apparatus described in this application can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0061] The apparatus or module described in the above embodiments can be implemented by a computer chip or physical entity, or by a product with a certain function. For ease of description, the above apparatus is described by dividing it into various modules according to their functions. When implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.
[0062] The methods, apparatus, or modules described in this application can be implemented in a computer-readable program code manner. The controller can be implemented in any suitable manner, such as a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of a memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code manner, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included within it for implementing various functions can also be considered as structures within the hardware component. Alternatively, the device used to implement various functions can be viewed as either a software module that implements the method or a structure within a hardware component.
[0063] This application also provides an apparatus, the apparatus comprising: a processor; a memory for storing processor-executable instructions; wherein, when the processor executes the executable instructions, it implements the method described in this application.
[0064] This application also provides a non-volatile computer-readable storage medium storing a computer program or instructions thereon, which, when executed, enables the method described in this application embodiment to be implemented.
[0065] Furthermore, in the various embodiments of the present invention, each functional module can be integrated into a processing module, or each module can exist independently, or two or more modules can be integrated into a single module.
[0066] The aforementioned storage media include, but are not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), Cache, Hard Disk Drive (HDD), or Memory Card. The memory can be used to store computer program instructions.
[0067] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, or it can be embodied in the process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.
[0068] The various embodiments described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. All or part of this application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, mobile communication terminals, multiprocessor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.
[0069] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of this application.
Claims
1. An obstacle avoidance verification method, characterized in that, include: In diverse simulation scenarios, simulation tests are conducted using motion models with mounted camera components to acquire multiple sets of simulated binocular images. The simulated binocular image is corrected, and its simulated depth map is determined by the disparity of the simulated binocular image; Construct a depth map model and train it with real-world depth maps collected from the real environment to obtain a real-world depth map model; The simulated depth map is input into the physical depth map model to fuse real-world features and obtain a fused depth map. The fused depth map is converted into a video stream and pushed to the streaming media server. The obstacle avoidance algorithm obtains the fused depth map by accessing the streaming media server to verify the obstacle avoidance algorithm.
2. The method according to claim 1, characterized in that, In the aforementioned diversified simulation scenario, simulation testing is conducted using a motion model with mounted camera components to acquire multiple sets of simulated binocular images, including: Based on testing requirements, a diverse simulation scenario is constructed using 3D simulation software, and multiple motion models are defined within it. Enable multiple camera components in the 3D simulation software, configure their camera intrinsics, and set the camera spacing baseline; The camera components are mounted onto the motion model, and the motion model is driven to move according to a preset trajectory to acquire multiple sets of simulated binocular images.
3. The method according to claim 1, characterized in that, The correction of the simulated binocular image includes: Distortion correction is performed on the simulated binocular images using distortion coefficients to eliminate image distortion; and / or, the left and right camera components are corrected to the same imaging plane using a rotation matrix so that the x-axis of the corrected camera component imaging plane remains horizontal with the camera spacing baseline.
4. The method according to claim 1, characterized in that, The step of determining the simulated depth map using the disparity of the simulated binocular images includes: The disparity value is calculated based on the projection coordinates of the simulated binocular image; The simulated depth map corresponding to the simulated binocular image is determined by using camera intrinsic parameters, camera spacing baseline, and disparity value.
5. The method according to claim 1, characterized in that, The process of training the model with real-world depth maps collected from the actual environment to obtain a real-world depth map model includes: Add obstacle feature points to the real-world depth map captured by the physical camera in the real environment; The actual depth values between the real camera and the obstacle, along with the labeled real-scene depth map, are input into the depth map model for training. After multiple rounds of optimization, the real-scene depth map model is obtained.
6. The method according to claim 1, characterized in that, The step of inputting the simulated depth map into the physical depth map model to fuse real-world features and obtain a fused depth map includes: The simulated depth map is input into the trained physical depth map model. The physical depth map model is used to supplement the obstacle feature points of the simulated depth map, and the depth values corresponding to the pixels are corrected and the display colors are completed to obtain the fused depth map.
7. The method according to claim 1, characterized in that, The step of converting the fused depth map into a video stream and pushing it to the streaming media server, and the obstacle avoidance algorithm obtaining the fused depth map by accessing the streaming media server for obstacle avoidance algorithm verification, includes: The fused depth map is encapsulated frame by frame into a standardized video stream; The standardized video stream is pushed to the streaming media server via video push, and an independent data channel is configured for each obstacle avoidance algorithm to achieve the sharing and distribution of the fused depth map.
8. An obstacle avoidance verification device for implementing the method according to any one of claims 1-7, characterized in that, include: The acquisition module is used to conduct simulation tests in diverse simulation scenarios by using motion models with mounted camera components to acquire multiple sets of simulated binocular images. The correction module is used to correct the simulated binocular image and determine its simulated depth map based on the disparity of the simulated binocular image. The training module is used to build a depth map model. It is trained with real-world depth maps collected from the real environment to obtain a real-world depth map model. The fusion module is used to input the simulated depth map into the physical depth map model to fuse real-world features and obtain a fused depth map. The verification module is used to convert the fused depth map into a video stream and push it to the streaming media server. The obstacle avoidance algorithm obtains the fused depth map by accessing the streaming media server to verify the obstacle avoidance algorithm.
9. An apparatus for performing an obstacle avoidance verification method, characterized in that, include: processor; Memory used to store processor-executable instructions; When the processor executes the executable instructions, it implements the method as described in any one of claims 1 to 7.
10. A non-volatile computer-readable storage medium, characterized in that, Includes storage of computer programs or instructions that, when executed, cause the method as described in any one of claims 1 to 7 to be implemented.