A machine vision-based autonomous driving method and system

Abstract

This invention proposes an autonomous driving method and system based on machine vision. The machine vision-based autonomous driving method includes: acquiring video stream data of the vehicle's surroundings using an onboard monocular camera; performing semantic segmentation on the video stream data using the vehicle's speed characteristics and the rate of change in pedestrian and vehicle traffic within the area captured by the onboard monocular camera to obtain a semantically segmented 3D scene; and using the semantically segmented 3D scene and a high-precision map for integrated positioning and detour path generation. The system includes modules corresponding to the steps of the method.

Description

Technical Field

[0001] This invention proposes an autonomous driving method and system based on machine vision, belonging to the field of autonomous driving technology. Background Technology

[0002] With the rapid development of artificial intelligence technology, autonomous driving has become a crucial direction for the future automotive industry. The realization of autonomous driving relies on the comprehensive application of various advanced technologies. Among them, machine vision technology, due to its ability to simulate the functions of the human visual system and achieve accurate perception and understanding of the surrounding environment, has become one of the core technologies in the field of autonomous driving. In autonomous driving systems, onboard cameras, as the main sensors of machine vision, undertake the important task of acquiring information about the vehicle's surrounding environment. By performing semantic segmentation and recognition on the collected video stream data, they can determine the drivable area and obtain detour routes for autonomous driving.

[0003] However, existing technologies, in order to improve the accuracy of semantic segmentation, only consider the perfection of the semantic segmentation method itself. This leads to increasingly redundant and cumbersome semantic segmentation methods, increasing the computational load on the vehicle system without significantly improving the accuracy of semantic recognition. On the other hand, with the continuous development of the performance of new energy vehicles, the mobility and flexibility of vehicles during operation are high. In this case, relying solely on video streams of the vehicle environment for semantic segmentation reduces the flexibility and sensitivity of semantic segmentation. Summary of the Invention

[0004] This invention provides a machine vision-based autonomous driving method and system to solve the technical problems existing in the prior art. The technical solution adopted is as follows:

[0005] A machine vision-based autonomous driving method, the machine vision-based autonomous driving method comprising:

[0006] Use an in-vehicle monocular camera to acquire video stream data of the area surrounding the vehicle;

[0007] The video stream data is semantically segmented by utilizing the vehicle speed characteristics and the rate of change of pedestrian and vehicle flow within the shooting range of the vehicle-mounted monocular camera, and the semantically segmented 3D scene is obtained.

[0008] The semantically segmented 3D scene and high-precision map are used for comprehensive positioning and detour path generation.

[0009] Furthermore, the method of acquiring video stream data around the vehicle using an in-vehicle monocular camera includes:

[0010] The shooting parameters of the vehicle-mounted monocular camera are adjusted in real time according to the environmental conditions of the vehicle.

[0011] After the shooting parameters of the vehicle-mounted monocular camera are adjusted, the vehicle-mounted monocular camera is controlled in real time to shoot and acquire video stream data around the vehicle.

[0012] Furthermore, semantic segmentation is performed on the video stream data using the vehicle speed characteristics and the rate of change in pedestrian and vehicle traffic within the shooting range of the vehicle-mounted monocular camera, to obtain the semantically segmented 3D scene, including:

[0013] The video stream data is processed using the vehicle speed characteristics and the rate of change of pedestrian and vehicle traffic within the shooting range of the vehicle-mounted monocular camera to obtain the depth map, material property map and 3D scene stream corresponding to the video stream data;

[0014] Perform semantic segmentation on the 3D scene stream to obtain the initial semantic segmentation result corresponding to the 3D scene stream;

[0015] Feature extraction and fusion are performed on continuous RGB video frames, depth maps, and material property maps corresponding to video stream data to obtain a multimodal feature pyramid;

[0016] The initial semantic segmentation result is repaired using a multimodal feature pyramid and a 3D scene flow to obtain the repaired semantic segmentation result;

[0017] The 3D scene flow is labeled according to the semantic segmentation results to obtain the semantically segmented 3D scene.

[0018] Furthermore, the video stream data is processed using the vehicle speed characteristics and the rate of change of pedestrian and vehicle traffic within the shooting range of the vehicle-mounted monocular camera to obtain the depth map, material property map, and 3D scene stream corresponding to the video stream data, including:

[0019] Perform frame processing on the video stream data to obtain consecutive RGB video frames corresponding to the video stream data;

[0020] The number of consecutive frames input to the dynamic neural radiation field NeRF model is set based on the maximum and minimum speeds of the vehicle within a preset time range, combined with the rate of change of pedestrian and vehicle traffic within the current field of view of the vehicle-mounted monocular camera.

[0021] The continuous RGB video frames are divided according to the number of consecutive frames, and the divided continuous RGB video frames are input into the dynamic neural radiation field NeRF model to output the depth map, material property map and 3D scene flow corresponding to the video stream data.

[0022] Furthermore, feature extraction and fusion are performed on the continuous RGB video frames, depth maps, and material property maps corresponding to the video stream data to obtain a multimodal feature pyramid, including:

[0023] Convolutional neural networks were used to process the semantic feature maps corresponding to continuous RGB video frames, material property maps, and depth maps, respectively.

[0024] Using the scale of the continuous RGB video semantic feature map as a standard, the semantic feature maps corresponding to the material property map and the depth map are adjusted to be consistent with the scale of the RGB video semantic feature map. The semantic feature maps corresponding to the adjusted material property map and the depth map are then mapped to the semantic space that matches the RGB features and weighted and fused to obtain a multimodal feature pyramid.

[0025] Furthermore, the initial semantic segmentation results are repaired using a multimodal feature pyramid and a 3D scene flow to obtain the repaired semantic segmentation results, including:

[0026] Spatiotemporal features are obtained by extracting features from a 3D voxel grid generated from consecutive RGB video frames using 3D sparse convolution kernels.

[0027] By combining the vehicle's current speed, the rate of change of pedestrian traffic and the rate of change of vehicle traffic with the 3D scene flow and its corresponding density map, the occlusion mask of the current frame in the continuous RGB video frames is calculated to obtain the occlusion area in the current frame.

[0028] The occluded areas are repaired to generate the repaired semantic segmentation results.

[0029] Furthermore, the occlusion area mask is 1 or 0, wherein when the occlusion area mask is 1, it means that the pixel corresponding to the occlusion area mask is marked as an occlusion area; when the occlusion area mask is 0, it means that the pixel corresponding to the occlusion area mask is marked as an unoccluded area.

[0030] Furthermore, the occlusion region mask is obtained by combining the difference between the density value of pixel x in the previous consecutive RGB video frame and the density value of pixel x in the current consecutive RGB video frame with a dynamically adjusted occlusion threshold and a dynamically adjusted motion residual threshold.

[0031] A machine vision-based autonomous driving system, the machine vision-based autonomous driving system comprising:

[0032] The video stream data acquisition module is used to acquire video stream data around the vehicle using an in-vehicle monocular camera;

[0033] The semantic segmentation module is used to perform semantic segmentation on video stream data by utilizing the vehicle speed characteristics and the rate of change of pedestrian and vehicle flow within the shooting range of the vehicle-mounted monocular camera, and to obtain the semantically segmented 3D scene.

[0034] The path generation module is used to perform comprehensive positioning and detour path generation by combining the semantically segmented 3D scene with a high-precision map.

[0035] Beneficial effects of this invention:

[0036] This invention provides a machine vision-based autonomous driving method and system that, while performing semantic segmentation, not only relies on the video stream of the vehicle environment but also considers the vehicle's operating state during driving and the actual state of pedestrians and vehicles on the road. This effectively improves the accuracy of semantic segmentation while greatly enhancing the tracking of semantic segmentation with vehicle operation and the automatic adjustment of semantic segmentation, thereby maximizing the sensitivity and flexibility of semantic segmentation. Furthermore, this machine vision-based autonomous driving method and system effectively improves the matching between semantic segmentation and the vehicle's driving state and road state, thus improving the accuracy and processing efficiency of semantic segmentation when objects are temporarily occluded (such as pedestrians briefly occluded by vehicles and then reappearing), as well as the accuracy and processing efficiency of semantic segmentation under conditions of rare object materials (such as transparent glass, reflective pavement) or special lighting (such as strong specular reflection) on the road. Meanwhile, since the semantic segmentation process utilizes the vehicle's operating status and the actual status of pedestrians and vehicles on the road, it can adjust the mobility and sensitivity of semantic segmentation in handling the above-mentioned special cases according to the vehicle's operating status and the actual status of pedestrians and vehicles on the road. Thus, while ensuring that the vehicle's operating status is compatible with the semantic segmentation operation rate, it can maximize the release of the vehicle system's computing power and reduce the computing load on the vehicle system while improving the accuracy of intelligent recognition. Attached Figure Description

[0037] Figure 1 This is a flowchart of the method described in this invention;

[0038] Figure 2 This is a system block diagram of the system described in this invention. Detailed Implementation

[0039] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0040] This invention proposes a machine vision-based autonomous driving method, such as... Figure 1 As shown, the machine vision-based autonomous driving method includes:

[0041] S1. Acquire video stream data around the vehicle using an in-vehicle monocular camera; specifically, adjust the shooting parameters of the in-vehicle monocular camera in real time according to the environmental conditions of the vehicle, wherein the environmental conditions include but are not limited to light intensity and weather conditions; the shooting parameters include but are not limited to resolution, field of view angle and dynamic range; then, after the shooting parameters of the in-vehicle monocular camera are adjusted, control the in-vehicle monocular camera to shoot in real time to acquire video stream data around the vehicle.

[0042] S2. Use the vehicle speed characteristics and the rate of change of pedestrian and vehicle flow within the shooting range of the vehicle-mounted monocular camera to perform semantic segmentation on the video stream data and obtain the semantically segmented 3D scene.

[0043] S3. Utilize the semantically segmented 3D scene and high-precision map for comprehensive positioning and detour path generation. Specifically, match the semantically segmented 3D scene with the high-precision map to obtain the vehicle's real-time positioning information; then, obtain the vehicle's corresponding drivable area based on the semantically segmented 3D scene and the current vehicle's positioning location; finally, generate a detour path using a detour path algorithm combined with the vehicle's corresponding drivable area. The detour path algorithm includes, but is not limited to, the A* algorithm, Dijkstra's algorithm, and the RRT (Fast Random Search Tree) algorithm.

[0044] The working principle of the above technical solution is as follows: A vehicle-mounted monocular camera continuously captures images of the environment surrounding the vehicle, acquiring continuous video stream data. The monocular camera captures light through its lens and converts it into electrical signals. After a series of signal processing and conversions, digital video stream data is formed that can be used for subsequent analysis. This data contains various information about the vehicle's surroundings, such as roads, vehicles, pedestrians, traffic signs, and traffic lights. Then, semantic segmentation is performed on the video stream data using the vehicle's speed characteristics and the rate of change in pedestrian and vehicle traffic within the area captured by the vehicle-mounted monocular camera, obtaining a semantically segmented 3D scene. The semantically segmented 3D scene is then fused with a high-precision map. The high-precision map contains detailed road information, such as lane line positions, traffic sign positions, road slopes, and curvatures. By matching and comparing objects and features in the 3D scene with information on the high-precision map, the vehicle can determine its precise location on the map. For example, by identifying lane lines in the video and matching them with lane line information on the high-precision map, the vehicle can determine its current lane and direction of travel. After determining the vehicle's location, a detour route is generated using a path planning algorithm based on the surrounding traffic conditions (obtained from the semantically segmented 3D scene) and destination information. The path planning algorithm considers various factors, such as road capacity, traffic flow, and speed limits, to generate a safe and efficient route. If obstacles or traffic congestion are detected ahead in the 3D scene, the algorithm automatically finds other feasible routes to guide the vehicle around, ensuring it reaches its destination smoothly.

[0045] The above technical solution achieves the following results: By combining vehicle speed characteristics, pedestrian flow change rate, and vehicle flow change rate to perform semantic segmentation on video stream data, different objects and scene elements can be distinguished more accurately. For example, when vehicles are traveling at high speeds, distant traffic signs, pedestrians, and other targets can be identified more accurately; based on the pedestrian and vehicle flow change rates, dynamic objects, such as pedestrians crossing the road or vehicles changing lanes, can be more clearly distinguished, thereby improving the accuracy of semantic segmentation and providing a more reliable foundation for subsequent localization and path planning. Based on the accurate semantically segmented 3D scene and high-precision map, comprehensive localization and detour path generation can more accurately determine the vehicle's position on the map and reduce localization errors. Simultaneously, when generating detour paths, actual traffic conditions on the road, such as dynamic pedestrian and vehicle flow, can be fully considered to plan safer and more efficient driving routes, avoiding accidents or delays caused by inaccurate localization or unreasonable path planning.

[0046] Meanwhile, the aforementioned technical solution utilizes information such as vehicle speed characteristics, pedestrian flow change rates, and traffic flow change rates to process video stream data, enabling rapid filtering and analysis of key information and improving the speed of semantic segmentation. This real-time processing capability allows vehicles to respond promptly to changes in the road environment, generating accurate semantic segmentation results and 3D scenes in a short time, providing timely data support for subsequent localization and path planning. Because it can acquire the semantically segmented 3D scene in real time, vehicles can adjust their detour routes promptly based on real-time changes in the road environment, such as sudden obstacles or traffic congestion. This real-time adjustment capability enhances the vehicle's ability to cope with emergencies, ensuring the safety and efficiency of autonomous driving.

[0047] One embodiment of the present invention utilizes vehicle speed characteristics and the rate of change of pedestrian and vehicle traffic within the shooting range of an onboard monocular camera to perform semantic segmentation on video stream data, thereby obtaining a semantically segmented 3D scene, including:

[0048] S201. The video stream data is processed using the vehicle speed characteristics and the rate of change of pedestrian and vehicle traffic within the shooting range of the vehicle-mounted monocular camera to obtain the depth map, material property map, and 3D scene stream corresponding to the video stream data; wherein, the material property map maps the material feature vector m to 4 channels, and the index parameters corresponding to the 4 channels include diffuse reflectance, specular reflectance, transparency, and roughness.

[0049] S202. Perform semantic segmentation on the 3D scene stream to obtain the initial semantic segmentation result corresponding to the 3D scene stream;

[0050] S203. Perform feature extraction and fusion on the continuous RGB video frames, depth map and material property map corresponding to the video stream data to obtain a multimodal feature pyramid; wherein, the multimodal feature pyramid contains color, geometry and material information;

[0051] S204. The initial semantic segmentation result is repaired using a multimodal feature pyramid and a 3D scene flow to obtain the repaired semantic segmentation result; wherein, the semantic segmentation result includes, but is not limited to, pedestrians, objects, roads, obstacles, puddles, and traffic element lights, etc.

[0052] S205. Annotate the 3D scene flow according to the semantic segmentation results to obtain the semantically segmented 3D scene.

[0053] The working principle of the above technical solution is as follows: Utilizing the vehicle's speed characteristics and the rate of change in pedestrian and vehicle traffic within the shooting range of the vehicle-mounted monocular camera, the acquired video stream data is processed to generate a depth map, a material property map, and a 3D scene stream. The depth map reflects the distance information between objects in the scene and the camera; the material property map maps material feature vectors to four channels (diffuse reflectivity, specular reflectivity, transparency, and roughness) to describe the material properties of the object's surface; and the 3D scene stream reflects the motion information of objects in the scene. The generation of these data is based on a comprehensive analysis and calculation of pixel information, speed, and the rate of change in pedestrian and vehicle traffic in the video stream. Semantic segmentation is performed on the generated 3D scene stream to obtain the initial semantic segmentation results. The above technical solution uses a semantic segmentation algorithm to initially classify different parts of the scene into different semantic categories, such as pedestrians and vehicles, based on the motion characteristics and other information of objects in the 3D scene stream. Feature extraction is performed on continuous RGB video frames, depth maps, and material property maps corresponding to the video stream data. These features from different modalities (color, geometry, and material) are then fused to construct a multimodal feature pyramid. The feature extraction process may employ methods such as convolutional neural networks (CNNs) to extract representative features from different images and data. The multimodal feature pyramid integrates information from multiple aspects, including color, geometry, and material, providing richer information support for subsequent semantic segmentation result repair. The constructed multimodal feature pyramid and 3D scene stream are used to repair the initial semantic segmentation results. By comprehensively considering multimodal features and motion information in the scene stream, possible errors, missing parts, or inaccuracies in the initial results are corrected and improved, resulting in a more accurate semantic segmentation result covering multiple semantic categories such as pedestrians, objects, roads, obstacles, puddles, and traffic elements. Based on the repaired semantic segmentation results, the 3D scene stream is labeled, assigning each part to the corresponding semantic category, thereby obtaining the semantically segmented 3D scene. This 3D scene provides detailed and accurate environmental information for subsequent autonomous driving decisions.

[0054] The aforementioned technical solution achieves the following results: by combining information from multiple aspects such as vehicle speed, pedestrian and vehicle flow change rates, and comprehensively utilizing multimodal data (RGB video frames, depth maps, material property maps, and 3D scene streams) for semantic segmentation and result repair, it can more accurately identify and classify various elements in a scene. For example, analysis of material properties can help distinguish objects of different materials, and multimodal feature fusion provides richer information to determine the object category, thereby improving the accuracy of semantic segmentation and providing more reliable environmental perception for autonomous driving. The use of multimodal data makes the system more resistant to different environmental conditions and data noise. For example, under conditions of changing lighting and occlusion, information such as depth maps and material property maps can assist RGB video frames in accurate object recognition, and 3D scene streams can provide clues about object movement, helping the system maintain stable semantic segmentation performance in complex environments and improving the system's robustness. While processing multimodal data may increase computational load, with proper algorithm design and optimization, key data can be quickly filtered and processed using information such as speed, pedestrian flow, and vehicle flow change rates. Furthermore, the construction of a multimodal feature pyramid can improve the efficiency of feature extraction and fusion to some extent. This ensures timely and accurate acquisition of semantically segmented 3D scenes in real-time autonomous driving scenarios, supporting real-time vehicle decision-making. The semantic segmentation results cover a rich set of semantic categories, including pedestrians, objects, roads, obstacles, puddles, and traffic elements. Through analysis of material properties and fusion of multimodal information, the system gains a deeper and more comprehensive understanding of the scene. This helps vehicles better perceive their surroundings and make more rational decisions, such as avoiding obstacles and choosing appropriate driving paths, thus improving the safety and intelligence of autonomous driving.

[0055] In one embodiment of the present invention, the video stream data is processed using vehicle speed characteristics and the rate of change of pedestrian and vehicle traffic within the shooting range of an onboard monocular camera to obtain a depth map, material property map, and 3D scene stream corresponding to the video stream data, including:

[0056] S2011. Perform frame processing on the video stream data to obtain continuous RGB video frames corresponding to the video stream data;

[0057] S2012. Based on the maximum and minimum speed of the vehicle within a preset time range, combined with the rate of change of pedestrian flow and the rate of change of traffic flow within the current shooting range of the vehicle-mounted monocular camera, set the number of consecutive frames input to the dynamic neural radiation field NeRF model.

[0058] S2013. Divide the continuous RGB video frames according to the number of consecutive frames, and input the divided continuous RGB video frames into the dynamic neural radiation field NeRF model, and output the depth map, material property map and 3D scene flow corresponding to the video stream data.

[0059] The working principle of the above technical solution is as follows: The video stream data acquired by the vehicle-mounted monocular camera undergoes frame processing, dividing the continuous video stream into individual image frames, thus obtaining continuous RGB video frames corresponding to the video stream data. These RGB video frames contain visual information about the vehicle's surrounding environment and serve as the foundational data for subsequent processing. First, the maximum and minimum speeds of the vehicle within a preset time range are monitored and recorded in real time. The preset time range can be specifically set experimentally based on the vehicle's actual performance parameters (such as 0-100 km / h acceleration time) to ensure effective reflection of vehicle speed changes. Simultaneously, each continuous RGB video frame is analyzed, and the number of pedestrians and vehicles appearing in the frame is identified using algorithms such as object detection. This allows for the calculation of the rate of change in pedestrian and vehicle traffic within the current field of view of the vehicle-mounted monocular camera.

[0060] Then, based on the maximum and minimum speed values, as well as the rates of change in pedestrian and vehicle traffic, and combined with a preset base number of consecutive frames (not less than 5), the number of consecutive frames input to the Dynamic Neural Radiation Field (NeRF) model is calculated and set using a specific formula. This process takes into account vehicle speed status and dynamic changes in the surrounding environment, dynamically determining the appropriate number of input frames.

[0061] Data Input and Output: The previously acquired continuous RGB video frames are divided into segments according to a predetermined number of consecutive frames, forming a video sequence that meets the requirements. These segmented continuous RGB video frames are then input into the Dynamic Neural Radiation Field (NeRF) model. The NeRF model is a deep learning-based technique that processes the input video sequence. By learning and analyzing visual information, vehicle speed, and environmental changes in the video frames, it outputs a depth map, material property map, and 3D scene flow corresponding to the video stream data. The depth map reflects the distance information between objects in the scene and the camera; the material property map describes the material characteristics of the object's surface (such as diffuse reflectance, specular reflectance, transparency, and roughness); and the 3D scene flow reflects the motion information of objects in the scene. Specifically, the video processing procedure of the Dynamic Neural Radiation Field (NeRF) model includes:

[0062] Spatiotemporal modeling: The video frame sequence (including timestamps) is input into the NeRF network, and the geometry (volume density) and material properties (reflectivity, transparency) of the 3D scene are implicitly expressed through MLP.

[0063] Dynamic separation: An implicit deformation field is introduced to distinguish the motion trajectory of static backgrounds from that of dynamic objects.

[0064] Volume rendering: Generates a depth map, material property map, and 3D scene flow (3D motion vectors between adjacent frames) from the current viewpoint through raycasting.

[0065] Finally, the output consists of a depth map (object distance information), a material property map (physical properties), and a 3D scene flow (dynamic object motion information) aligned with the input video frames. This technical solution introduces a video frame sequence (including timestamps) to provide prior physical properties for subsequent segmentation, resolving the material misjudgment problem caused by traditional methods relying on pure RGB data.

[0066] The technical benefits of the above solution are as follows: Dynamically adjusting the number of consecutive frames input to the NeRF model based on vehicle speed and the rate of change in pedestrian and vehicle traffic allows the model to better adapt to different driving scenarios. In complex and dynamic scenarios (such as areas with high pedestrian and vehicle traffic and large changes in vehicle speed), increasing the number of input frames provides richer information, enabling the model to more accurately generate depth maps, material property maps, and 3D scene streams. This allows for more precise perception of the surrounding environment, including the position, material, and motion state of objects, providing a more reliable basis for autonomous driving decisions. This solution can dynamically adjust input data according to actual driving conditions, making the NeRF model more adaptable to different road conditions, traffic situations, and vehicle driving states. Whether in congested urban areas, highways, or rural roads, the model can obtain appropriate input through reasonable consecutive frame settings, accurately process video stream data, and output high-quality environmental representation information, improving the model's generalization ability in various scenarios. It avoids the waste or insufficiency of computational resources that may result from a fixed number of input frames. When the scene is relatively stable, vehicle speed changes are small, and pedestrian and vehicle traffic is low, reducing the number of input frames can reduce the computational load of the model, improve operating efficiency, and save computing resources. Conversely, when more information is needed to process complex scenes, increasing the number of frames ensures the model's accuracy. This dynamic adjustment mechanism optimizes the allocation of computing resources, improving the system's real-time performance and resource utilization efficiency without affecting model performance. By combining vehicle speed characteristics with pedestrian and vehicle traffic change rates to process video stream data and input it into the NeRF model, the model can better capture dynamic changes and detailed information in the scene. This helps generate more accurate and complete depth maps, material property maps, and 3D scene streams, improving the quality of 3D scene reconstruction and providing a more realistic and accurate virtual environment representation for autonomous driving systems. This supports more advanced autonomous driving functions, such as path planning, obstacle detection, and obstacle avoidance.

[0067] In one embodiment of the present invention, the number of consecutive frames input to the dynamic neural radiation field NeRF model is set based on the maximum and minimum speeds of the vehicle within a preset time range, combined with the rate of change of pedestrian and vehicle traffic within the current field of view of the vehicle-mounted monocular camera. This includes:

[0068] Step 1: Identify the number of pedestrians and vehicles appearing in each consecutive RGB video frame;

[0069] Step 2: Obtain the pedestrian flow change rate and vehicle flow change rate within the current vehicle-mounted monocular camera's shooting range area based on the number of pedestrians and vehicles appearing in each consecutive RGB video frame;

[0070] Step 3: Monitor the maximum and minimum speeds of the vehicle within a preset time range in real time;

[0071] Step 4: Based on the maximum and minimum speed values, combined with the rate of change of pedestrian and vehicle traffic within the current vehicle-mounted monocular camera's shooting range, set the number of consecutive frames (monocular video sequence) input to the dynamic neural radiation field NeRF model.

[0072] The number of consecutive frames (monocular video sequence) input to the dynamic neural radiation field NeRF model is obtained by the following formula:

[0073]

[0074] Where N represents the number of consecutive frames input to the dynamic neural radiation field NeRF model, and N is rounded up; n This represents the preset base number of consecutive frames, and the value of the number of consecutive frames must not be less than 5; B r This indicates the rate of change in pedestrian traffic around the vehicle. B c Indicates the rate of change in traffic flow around the vehicle; r The velocity state parameter is represented by the following formula:

[0075]

[0076] in, r Indicates velocity state parameters; v max and v min These represent the maximum and minimum speeds of the vehicle within a preset time range, respectively. tThis indicates the length of time corresponding to a preset time range, and the value of the preset time range is specifically set through experiments based on the actual performance parameters of the vehicle (e.g., 0-100 km / h acceleration time).

[0077] The working principle of the above technical solution is as follows: It identifies continuous RGB video frames acquired by an in-vehicle monocular camera and determines the number of pedestrians and vehicles appearing in each continuous RGB video frame using computer vision algorithms (such as object detection algorithms). This technical solution forms the basis for subsequent calculations of pedestrian and vehicle flow change rates. By detecting and counting targets in the video frames, it obtains the distribution information of people and vehicles in the scene. Based on the number of pedestrians and vehicles appearing in each continuous RGB video frame, it calculates the pedestrian and vehicle flow change rates within the current field of view of the in-vehicle monocular camera. It also monitors the maximum and minimum speeds of vehicles within a preset time range in real time. Vehicle speed information can be obtained through devices such as the vehicle's own speed sensors. The length of the preset time range is specifically set experimentally based on the vehicle's actual performance parameters (such as 0-100 km / h acceleration time) to ensure that the acquired speed data reflects the vehicle's speed change characteristics within that time period.

[0078] The above technical solution achieves the following effects: By dynamically setting the number of consecutive frames input to the NeRF model based on vehicle speed and the rate of change in surrounding pedestrian and vehicle traffic, the model can better adapt to different driving scenarios. When vehicle speed changes significantly or surrounding pedestrian and vehicle traffic changes frequently, increasing the number of input frames provides richer information, helping the model to more accurately reconstruct the scene and understand the environment. Conversely, when the scene is relatively stable, reducing the number of input frames reduces computational load and improves model efficiency. A reasonable setting of the number of consecutive frames helps the NeRF model capture dynamic information in the scene more comprehensively. For example, in areas with high pedestrian and vehicle traffic, a larger number of consecutive frames allows the model to better track the movement trajectories of pedestrians and vehicles, thereby more accurately modeling and analyzing the scene and enhancing the model's understanding and processing capabilities for complex scenes. It avoids the problem of wasted or insufficient computational resources that might result from a fixed number of input frames. By dynamically adjusting the number of consecutive frames according to actual conditions, computational resources are effectively utilized while ensuring model performance. When vehicle speed is stable and the surrounding environment changes little, reducing the number of input frames can lower computational costs and improve the system's real-time performance. Conversely, when more information is needed to handle complex scenarios, increasing the number of frames ensures model accuracy and optimizes computational resource allocation. Setting the number of consecutive frames based on vehicle speed and environmental change rate provides more suitable input data for the NeRF model, contributing to improved training and inference accuracy. More accurate scene modeling and environmental understanding provide more reliable environmental perception for autonomous driving systems, thereby supporting safer and smarter driving decisions and enhancing the overall performance of autonomous driving.

[0079] One embodiment of the present invention involves extracting and fusing features from consecutive RGB video frames, depth maps, and material property maps corresponding to video stream data to obtain a multimodal feature pyramid, including:

[0080] S2031. Use a convolutional neural network to process the semantic feature maps corresponding to continuous RGB video frames, material property maps, and depth maps respectively.

[0081] S2032. Using the scale of the continuous RGB video semantic feature map as the standard, adjust the semantic feature maps corresponding to the material property map and the depth map to be consistent with the scale of the RGB video semantic feature map, and map the semantic feature maps corresponding to the material property map and the depth map after adjustment to the semantic space that matches the RGB features for weighted fusion to obtain the multimodal feature pyramid.

[0082] Specifically: Convolutional neural networks (CNNs) are used to extract features from continuous RGB video frames to obtain multi-scale RGB semantic feature maps; CNNs are also used to extract features from material property maps to obtain corresponding semantic feature maps; CNNs are used to extract features from depth maps to obtain corresponding depth feature maps; bilinear interpolation is used to adjust the resolution of the semantic feature maps and depth feature maps corresponding to the material property maps to match the RGB semantic feature maps; and the semantic feature maps and depth feature maps corresponding to the material property maps are mapped to a semantic space matching the RGB features and then weighted and fused to obtain a multimodal feature pyramid.

[0083] The working principle of the above technical solution is to use a convolutional neural network (CNN) to process continuous RGB video frames. The CNN automatically extracts various semantic features from the video frames, such as edges, textures, and shapes, through structures such as convolutional layers and pooling layers, generating multi-scale RGB semantic feature maps. Feature maps at different scales contain different levels of semantic information; small-scale feature maps may capture more detailed information, while large-scale feature maps reflect a more macroscopic scene structure. Similarly, CNN is used to extract features from the material property map. The material property map contains information mapping the material feature vector to four channels (diffuse reflectance, specular reflectance, transparency, and roughness). The CNN extracts semantic features related to material properties from these channels, obtaining the semantic feature map corresponding to the material property map. CNN is then used to extract features from the depth map. The depth map represents the distance information between objects in the scene and the camera. The CNN can extract semantic features related to spatial structure and object position from the depth data, generating the depth feature map corresponding to the depth map. Using the scale of the continuous RGB video semantic feature map as a standard, bilinear interpolation is used to process the semantic feature map and depth feature map corresponding to the material property map. Bilinear interpolation is a commonly used image scaling technique that adjusts the resolution of the semantic feature maps of material property maps and depth maps to match that of RGB semantic feature maps by calculating a weighted average between adjacent pixels. Figure 1 To ensure scale comparability of feature maps from different modalities, the semantic and depth feature maps corresponding to the scaled material property maps are mapped to a semantic space matching the RGB features. This step aims to fuse features from different modalities at the same semantic level. Then, these mapped feature maps are weighted and fused, assigning appropriate weights based on the importance of different modal features, and combined to form a multimodal feature pyramid. The multimodal feature pyramid integrates information from RGB video frames, material property maps, and depth maps, encompassing semantic features related to color, geometry, and material properties, providing richer information for subsequent analysis and processing.

[0084] The effects of the above technical solution are as follows: By fusing multimodal features from continuous RGB video frames, material property maps, and depth maps, the multimodal feature pyramid can more comprehensively represent scene information. It not only includes visual color and texture information (from RGB video frames), but also covers spatial location information of objects (from depth maps) and material property information (from material property maps). This rich feature representation helps to more accurately identify and understand objects and elements in the scene, improving the system's perception ability in complex scenes. Different modalities of data exhibit varying stability in the face of various environmental factors (such as changes in lighting and occlusion). For example, depth maps can still provide good object location information even in low-light conditions, while RGB video frames have advantages in color and texture recognition. Through multimodal feature fusion, the system can comprehensively utilize the advantages of different modalities, reduce the limitations of single-modal data, enhance resistance to environmental changes, and improve the system's robustness and reliability. The multimodal feature pyramid provides richer and more accurate feature inputs for subsequent tasks (such as semantic segmentation and object detection). In semantic segmentation, combining multimodal features enables more precise classification of each pixel, distinguishing different objects and scene regions. In object detection, it helps to more accurately locate and identify target objects, reducing false positives and false negatives. Therefore, this technical solution can significantly improve the accuracy and performance of related tasks. Because multimodal feature fusion can capture more comprehensive and diverse information in a scene, models trained based on multimodal feature pyramids have stronger generalization capabilities when facing different scenes and datasets. The model can better adapt to different environmental conditions, object types, and scene layouts, and also demonstrates good performance in new and unseen scenes, improving the model's application scope and practicality.

[0085] One embodiment of the present invention utilizes a multimodal feature pyramid and a 3D scene flow to repair the initial semantic segmentation result, obtaining a repaired semantic segmentation result, including:

[0086] S2041. Use 3D sparse convolution kernels to extract features from the 3D voxel grid generated by consecutive RGB video frames to obtain spatiotemporal features; specifically: stack the 2D feature maps in consecutive RGB video frames corresponding to consecutive frame numbers into a 3D voxel grid; use 3D sparse convolution kernels to slide on the 3D voxel grid to extract spatiotemporal features.

[0087] S2042. Calculate the occlusion mask of the current frame in the continuous RGB video frames by combining the vehicle's current speed, the rate of change of pedestrian traffic around the vehicle, and the rate of change of vehicle traffic with the 3D scene flow and its corresponding density map, and obtain the occlusion area that appears in the current frame.

[0088] S2043. Repair the occluded area to generate a repaired semantic segmentation result. Specifically: use LSTM memory units to repair the occluded area of ​​the current frame using historical frame features from consecutive RGB video frames to obtain repaired spatiotemporal features; then, upsample the repaired spatiotemporal features to generate a repaired semantic segmentation result.

[0089] The working principle of the above technical solution is as follows: First, the 2D feature maps from consecutive RGB video frames corresponding to consecutive frames are stacked into a 3D voxel grid. This step combines the 2D information of multiple consecutive frames into a 3D structure with spatial and temporal dimensions for subsequent extraction of spatiotemporal features. Then, a sliding operation is performed on the 3D voxel grid using a 3D sparse convolution kernel. The 3D sparse convolution kernel moves on the 3D voxel grid, extracting spatiotemporal features through convolution operations. This operation can capture the motion information of objects in the video frame sequence and the spatial relationships between different frames, thereby obtaining a feature representation containing temporal and spatial dimensional information. Combining the vehicle's current speed, the rate of change of pedestrian traffic and vehicle traffic around the vehicle, and referencing the 3D scene flow and its corresponding density map, this information comprehensively reflects the vehicle's motion state and the dynamic changes of the surrounding environment. Through analysis and calculation of this information, the occlusion region mask of the current frame in the consecutive RGB video frames is determined. The occlusion region mask is an identifier indicating which areas in the current frame are occluded, allowing for accurate location of occluded areas in the current frame. LSTM (Long Short-Term Memory) memory units are used to repair occluded regions in the current frame by utilizing features from historical frames in consecutive RGB video frames. LSTM has a memory function, remembering relevant information from historical frames and applying it to the repair of occluded regions in the current frame. In this way, features from the unoccluded parts of historical frames can be used to infer and fill in the occluded regions of the current frame, thereby obtaining the repaired spatiotemporal features. Upsampling is then performed on the repaired spatiotemporal features. Upsampling is the process of converting low-resolution feature maps to high-resolution ones. By upsampling, the resolution of the repaired feature map can be matched with that of the original semantic segmentation result, ultimately generating the repaired semantic segmentation result.

[0090] The above technical solution achieves the following results: by extracting spatiotemporal features and repairing occluded areas, it can more accurately identify and segment objects and regions in a scene. Extracting spatiotemporal features captures the motion information of objects, helping to track and segment them more accurately in dynamic scenes; repairing occluded areas solves the problem of inaccurate segmentation caused by occlusion, making the semantic segmentation results more complete and accurate, thereby improving the overall semantic segmentation accuracy. This technical solution comprehensively considers multiple factors such as vehicle speed, pedestrian and vehicle traffic flow rates, as well as information such as 3D scene flow and density maps, making the system more adaptable to complex and changing environments. Under different traffic scenarios and environmental conditions, it can more accurately handle occlusion and object motion changes, reducing segmentation errors caused by environmental factors and improving the system's robustness and reliability. The extraction of spatiotemporal features and the use of LSTM to remember and apply historical frame features enable the system to better handle dynamic scenes. It can accurately track the motion trajectory of objects in consecutive frames, and even when objects are occluded or partially visible, it can make reasonable inferences and repairs based on historical information and spatiotemporal features, thereby improving the understanding and processing capabilities of dynamic scenes. Repairing occluded regions ensures the integrity of information in the semantic segmentation results. Occlusion is a common phenomenon in real-world scenarios. By repairing occluded regions, information loss due to occlusion is avoided, allowing the semantic segmentation results to more comprehensively reflect the real-world situation and providing richer and more accurate information support for subsequent tasks such as autonomous driving decisions.

[0091] In one embodiment of the present invention, the occlusion region mask is 1 or 0, wherein when the occlusion region mask is 1, it indicates that the pixel corresponding to the occlusion region mask is marked as an occluded region; when the occlusion region mask is 0, it indicates that the pixel corresponding to the occlusion region mask is marked as an unoccluded region. Furthermore, the occlusion region mask is obtained by combining the difference between the density value of pixel x in the previous consecutive RGB video frame and the density value of pixel x in the current consecutive RGB video frame with a dynamically adjusted occlusion threshold and a dynamically adjusted motion residual threshold.

[0092] The model for the occlusion region mask is as follows:

[0093]

[0094] in, R t ( p ) represents the mask for the occluded area; Δ δ ( p ) represents the difference between the density value of pixel x in the previous consecutive RGB video frame and the density value of pixel x in the current consecutive RGB video frame; wherein, the density value is the probability of object occupancy, Δ δ ( pA value greater than zero indicates that the object may disappear (i.e., become transparent), be occupied, or reflect light. In practice, the term "occupied" here represents the probability that the pixel is occupied, reflected, or transparent. μ 01 This indicates the dynamically adjusted occlusion threshold. W r ( x () represents the motion residual during vehicle movement, used to assess the difference between the actual observed density and the predicted motion density, and to eliminate misjudgments caused by rapid movement of objects; μ 02 This represents the dynamically adjusted motion residual threshold.

[0095] The expression for the motion residual during vehicle movement is as follows:

[0096]

[0097] in, w s ( x This represents the density value of pixel x in the current consecutive RGB video frames; w t ( x This indicates that the density map of the previous frame is distorted to the density value of the current frame based on the scene flow.

[0098] The dynamically adjusted occlusion threshold is obtained using the following formula:

[0099]

[0100] in, μ 01 This indicates the dynamically adjusted occlusion threshold. μ z0 This represents the preset baseline occlusion threshold, preferably 0.5; v Indicates the vehicle's current speed; v cy This indicates the preset speed reference value; B r This indicates the rate of change in pedestrian traffic around the vehicle. B c Indicates the rate of change in traffic flow around the vehicle;

[0101] Furthermore, the dynamically adjusted motion residual threshold is obtained using the following formula:

[0102]

[0103] in, μ 02 This represents the dynamically adjusted motion residual threshold. μ c0This represents the preset baseline motion residual threshold, preferably 0.3; v Indicates the vehicle's current speed; v cy This indicates the preset speed reference value.

[0104] The working principle and effects of the above technical solution are as follows: By comprehensively considering density difference, motion residual, and thresholds dynamically adjusted according to vehicle speed, pedestrian flow, and traffic flow, the above technical solution can more accurately detect occlusion areas. Dynamically adjusting the threshold allows the system to adaptively determine occlusion based on different driving scenarios (such as vehicle speed, pedestrian flow, and traffic flow), avoiding misjudgments or missed judgments that may occur with fixed thresholds, thus improving the accuracy of occlusion detection. The introduction of motion residual effectively eliminates misjudgments caused by rapid object movement. During vehicle movement, rapid object movement may cause changes in density values, easily leading to misjudgments as occlusion. By calculating the motion residual and comparing it with the dynamically adjusted motion residual threshold, it is possible to accurately distinguish between true occlusion and normal object movement, enhancing the system's resistance to interference factors and improving system stability. Dynamically adjusting the occlusion threshold and motion residual threshold according to the rate of change of vehicle speed, pedestrian flow, and traffic flow allows the system to better adapt to complex and changing traffic scenarios. In various scenarios, such as congested urban roads and highways, the system can adjust thresholds according to the actual situation to accurately detect occluded areas, improving its adaptability and reliability in complex scenarios. Accurate occlusion detection provides a more reliable foundation for subsequent semantic segmentation. During semantic segmentation, it can handle occluded areas more accurately, avoiding segmentation errors caused by occlusion, thereby improving the accuracy of semantic segmentation. This makes the segmentation results more reflective of real-world scene information, helping the autonomous driving system make more rational decisions.

[0105] Meanwhile, the above-mentioned technical solution not only relies on the video stream of the vehicle environment for semantic segmentation, but also considers the vehicle's operating status during driving and the actual state of pedestrians and vehicles on the road. This can effectively improve the accuracy of semantic segmentation, while greatly improving the tracking of semantic segmentation with vehicle operation and the automatic adjustment of semantic segmentation. In this way, the sensitivity and flexibility of semantic segmentation can be maximized. At the same time, the above-mentioned technical solution can effectively improve the matching between semantic segmentation and vehicle driving status and road status. This improves the accuracy and processing efficiency of semantic segmentation when objects are temporarily occluded (such as pedestrians being briefly occluded by vehicles and then reappearing), as well as the accuracy and processing efficiency of semantic segmentation in cases where rare object materials (such as transparent glass, reflective road surface) or special lighting (such as strong specular reflection) appear on the road. Meanwhile, since the semantic segmentation process utilizes the vehicle's operating status and the actual status of pedestrians and vehicles on the road, it can adjust the mobility and sensitivity of semantic segmentation in handling the above-mentioned special cases according to the vehicle's operating status and the actual status of pedestrians and vehicles on the road. Thus, while ensuring that the vehicle's operating status is compatible with the semantic segmentation operation rate, it can maximize the release of the vehicle system's computing power and reduce the computing load on the vehicle system while improving the accuracy of intelligent recognition.

[0106] This invention proposes an autonomous driving system based on machine vision, such as... Figure 2 As shown, the machine vision-based autonomous driving system includes:

[0107] The video stream data acquisition module is used to acquire video stream data around the vehicle using an in-vehicle monocular camera. Specifically, it adjusts the shooting parameters of the in-vehicle monocular camera in real time according to the environmental conditions of the vehicle, wherein the environmental conditions include, but are not limited to, light intensity and weather conditions; the shooting parameters include, but are not limited to, resolution, field of view angle, and dynamic range; then, after the shooting parameters of the in-vehicle monocular camera are adjusted, it controls the in-vehicle monocular camera to shoot in real time to acquire video stream data around the vehicle.

[0108] The semantic segmentation module is used to perform semantic segmentation on video stream data by utilizing the vehicle speed characteristics and the rate of change of pedestrian and vehicle flow within the shooting range of the vehicle-mounted monocular camera, and to obtain the semantically segmented 3D scene.

[0109] The path generation module is used to perform comprehensive positioning and detour path generation using the semantically segmented 3D scene and high-precision map. Specifically, the semantically segmented 3D scene is matched with the high-precision map to obtain the vehicle's real-time positioning information; then, the drivable area corresponding to the vehicle is obtained based on the semantically segmented 3D scene and the current vehicle's positioning location; finally, a detour path is generated using a detour path algorithm combined with the drivable area corresponding to the vehicle. The detour path algorithm includes, but is not limited to, the A* algorithm, Dijkstra's algorithm, and RRT (Fast Random Search Tree) algorithm.

[0110] The working principle of the above technical solution is as follows: A vehicle-mounted monocular camera continuously captures images of the environment surrounding the vehicle, acquiring continuous video stream data. The monocular camera captures light through its lens and converts it into electrical signals. After a series of signal processing and conversions, digital video stream data is formed that can be used for subsequent analysis. This data contains various information about the vehicle's surroundings, such as roads, vehicles, pedestrians, traffic signs, and traffic lights. Then, semantic segmentation is performed on the video stream data using the vehicle's speed characteristics and the rate of change in pedestrian and vehicle traffic within the area captured by the vehicle-mounted monocular camera, obtaining a semantically segmented 3D scene. The semantically segmented 3D scene is then fused with a high-precision map. The high-precision map contains detailed road information, such as lane line positions, traffic sign positions, road slopes, and curvatures. By matching and comparing objects and features in the 3D scene with information on the high-precision map, the vehicle can determine its precise location on the map. For example, by identifying lane lines in the video and matching them with lane line information on the high-precision map, the vehicle can determine its current lane and direction of travel. After determining the vehicle's location, a detour route is generated using a path planning algorithm based on the surrounding traffic conditions (obtained from the semantically segmented 3D scene) and destination information. The path planning algorithm considers various factors, such as road capacity, traffic flow, and speed limits, to generate a safe and efficient route. If obstacles or traffic congestion are detected ahead in the 3D scene, the algorithm automatically finds other feasible routes to guide the vehicle around, ensuring it reaches its destination smoothly.

[0111] The above technical solution achieves the following results: By combining vehicle speed characteristics, pedestrian flow change rate, and vehicle flow change rate to perform semantic segmentation on video stream data, different objects and scene elements can be distinguished more accurately. For example, when vehicles are traveling at high speeds, distant traffic signs, pedestrians, and other targets can be identified more accurately; based on the pedestrian and vehicle flow change rates, dynamic objects, such as pedestrians crossing the road or vehicles changing lanes, can be more clearly distinguished, thereby improving the accuracy of semantic segmentation and providing a more reliable foundation for subsequent localization and path planning. Based on the accurate semantically segmented 3D scene and high-precision map, comprehensive localization and detour path generation can more accurately determine the vehicle's position on the map and reduce localization errors. Simultaneously, when generating detour paths, actual traffic conditions on the road, such as dynamic pedestrian and vehicle flow, can be fully considered to plan safer and more efficient driving routes, avoiding accidents or delays caused by inaccurate localization or unreasonable path planning.

[0112] Meanwhile, the aforementioned technical solution utilizes information such as vehicle speed characteristics, pedestrian flow change rates, and traffic flow change rates to process video stream data, enabling rapid filtering and analysis of key information and improving the speed of semantic segmentation. This real-time processing capability allows vehicles to respond promptly to changes in the road environment, generating accurate semantic segmentation results and 3D scenes in a short time, providing timely data support for subsequent localization and path planning. Because it can acquire the semantically segmented 3D scene in real time, vehicles can adjust their detour routes promptly based on real-time changes in the road environment, such as sudden obstacles or traffic congestion. This real-time adjustment capability enhances the vehicle's ability to cope with emergencies, ensuring the safety and efficiency of autonomous driving.

[0113] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for automatic driving based on machine vision, characterized in that, The machine vision-based automatic driving method comprises: acquiring video stream data around the vehicle by using a vehicle-mounted monocular camera; performing semantic segmentation on the video stream data by using vehicle speed characteristics and a flow rate of people and a flow rate of vehicles in a range of the shooting range of the vehicle-mounted monocular camera, to obtain a 3D scene after semantic segmentation; performing comprehensive positioning and generating a detour path by using the 3D scene after semantic segmentation and a high-definition map; wherein the semantic segmentation on the video stream data by using the vehicle speed characteristics and the flow rate of people and the flow rate of vehicles in the range of the shooting range of the vehicle-mounted monocular camera comprises: processing the video stream data by using the vehicle speed characteristics and the flow rate of people and the flow rate of vehicles in the range of the shooting range of the vehicle-mounted monocular camera, to obtain a depth map, a material attribute map and a 3D scene stream corresponding to the video stream data; performing semantic segmentation on the 3D scene stream to obtain an initial semantic segmentation result corresponding to the 3D scene stream; extracting and fusing features of continuous RGB video frames, the depth map and the material attribute map corresponding to the video stream data to obtain a multi-modal feature pyramid; repairing the initial semantic segmentation result by using the multi-modal feature pyramid and the 3D scene stream, to obtain a repaired semantic segmentation result; annotating the 3D scene stream according to the semantic segmentation result, to obtain the 3D scene after semantic segmentation.

2. The method of claim 1, wherein, The acquiring of the video stream data around the vehicle by using the vehicle-mounted monocular camera comprises: controlling the shooting parameters of the vehicle-mounted monocular camera in real time according to the environmental conditions of the vehicle; controlling the vehicle-mounted monocular camera to shoot in real time after the shooting parameters of the vehicle-mounted monocular camera are controlled, to acquire the video stream data around the vehicle.

3. The method of claim 1, wherein, The processing of the video stream data by using the vehicle speed characteristics and the flow rate of people and the flow rate of vehicles in the range of the shooting range of the vehicle-mounted monocular camera comprises: processing frames of the video stream data to obtain continuous RGB video frames corresponding to the video stream data; setting the number of continuous frames input into a dynamic NeRF model according to the maximum and minimum values of the speed of the vehicle in a preset time range, in combination with the flow rate of people and the flow rate of vehicles in the range of the shooting range of the current vehicle-mounted monocular camera; dividing the continuous RGB video frames according to the number of continuous frames, and inputting the divided continuous RGB video frames into the dynamic NeRF model, to output a depth map, a material attribute map and a 3D scene stream corresponding to the video stream data.

4. The method of claim 1, wherein, The extracting and fusing of features of the continuous RGB video frames, the depth map and the material attribute map corresponding to the video stream data to obtain a multi-modal feature pyramid comprises: using a convolutional neural network to extract semantic feature maps corresponding to the continuous RGB video frames, the material attribute map and the depth map, respectively; The material attribute graph and the depth graph are adjusted to be consistent with the scale of the RGB video semantic feature map, and the adjusted semantic feature maps of the material attribute graph and the depth graph are mapped to a semantic space matched with the RGB feature for weighted fusion to obtain a multi-modal feature pyramid.

5. The method of claim 1, wherein, The initial semantic segmentation result is repaired by using the multi-modal feature pyramid and the 3D scene flow to obtain a repaired semantic segmentation result, including: The 3D voxel grid generated by the continuous RGB video frame is extracted by using the 3D sparse convolution kernel to obtain a spatio-temporal feature; The occlusion region mask of the current frame in the continuous RGB video frame is calculated by using the current driving speed of the vehicle, the crowd change rate around the vehicle, the traffic flow change rate, the 3D scene flow and the corresponding density graph to obtain an occlusion region appearing in the current frame; The occlusion region is repaired to generate a repaired semantic segmentation result.

6. The method of claim 5, wherein the machine vision-based autonomous driving method further comprises: The occlusion region mask is 1 or 0, wherein when the occlusion region mask is 1, the pixel point corresponding to the occlusion region mask is marked as an occlusion region; and when the occlusion region mask is 0, the pixel point corresponding to the occlusion region mask is marked as a non-occlusion region. 7.The method of claim 5 or 6, wherein, The occlusion region mask is obtained by using the difference between the density value of pixel x in the previous continuous RGB video frame and the density value of pixel x in the current continuous RGB video frame, combined with a dynamically adjusted occlusion threshold and a dynamically adjusted motion residual threshold.

8. A machine vision-based autonomous driving system, characterized by, The automatic driving system based on machine vision includes: A video stream data acquisition module for acquiring video stream data around the vehicle by using a vehicle-mounted monocular camera; A semantic segmentation module for performing semantic segmentation on the video stream data by using the vehicle driving speed characteristics and the crowd change rate and the traffic flow change rate in the vehicle-mounted monocular camera shooting range to obtain a 3D scene after semantic segmentation; A path generation module for comprehensive positioning and detour path generation by using the 3D scene after semantic segmentation and a high-precision map; The vehicle driving speed characteristics and the crowd change rate and the traffic flow change rate in the vehicle-mounted monocular camera shooting range are used to perform semantic segmentation on the video stream data to obtain a 3D scene after semantic segmentation, including: The video stream data is processed by using the vehicle driving speed characteristics and the crowd change rate and the traffic flow change rate in the vehicle-mounted monocular camera shooting range to obtain a depth graph, a material attribute graph and a 3D scene flow corresponding to the video stream data; The 3D scene flow is subjected to semantic segmentation to obtain an initial semantic segmentation result corresponding to the 3D scene flow; The continuous RGB video frame, the depth graph and the material attribute graph corresponding to the video stream data are subjected to feature extraction and fusion to obtain a multi-modal feature pyramid; The initial semantic segmentation result is repaired by using the multi-modal feature pyramid and the 3D scene flow to obtain a repaired semantic segmentation result; The 3D scene flow is labeled according to the semantic segmentation result to obtain a 3D scene after semantic segmentation.

Images