Environmental perception methods, systems, devices, and media based on vehicle sentry mode.
By using an environmental perception network model that combines infrared cameras and lidar, the reduced recognition capability of the sentry mode under low light conditions and the problem of multi-camera fusion were solved, achieving all-round target detection and tracking, and improving the accuracy and real-time performance of environmental perception.
Patent Information
- Application Number
- CN202511909936.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-17
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2045-12-17
AI Technical Summary
Existing sentinel-mode environmental perception systems suffer from reduced recognition capabilities under low-light conditions, and multi-camera fusion suffers from computational complexity and poor real-time performance. In particular, depth image errors are significant in complex environments, making it difficult to improve the accuracy and real-time performance of target detection.
Infrared cameras are used to acquire color and depth images. Combined with intrinsic and extrinsic parameters, a pre-trained environmental perception network model is used for feature extraction and fusion. Target detection and tracking are performed using lidar point cloud data to generate sentinel early warning decisions.
The robustness of environmental perception under various lighting conditions has been improved, enabling all-round target detection and tracking, enhancing the accuracy and real-time performance of the sentinel mode, and solving the difficulties of multi-camera fusion.
Smart Images

Figure CN121716646B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the technical field of environmental perception, and relates to an environmental perception method, system, device and medium based on vehicle sentry mode. Background Technology
[0002] In the field of intelligent vehicles, Sentinel Mode is an all-weather active safety protection system that uses a network of vehicle sensors and cameras to monitor the vehicle's surroundings in real time, providing continuous safety protection while the vehicle is parked. Sentinel Mode typically includes functions such as environmental monitoring and anomaly response. When the vehicle is accidentally damaged, it can promptly record the perpetrator's actions, providing the owner with strong evidence to protect their rights and enabling them to successfully seek justice and obtain appropriate compensation.
[0003] Currently, sentinel mode relies heavily on cameras for perception, utilizing neural network algorithms to detect image data and issue warnings when anomalies are detected. However, most cameras are currently fisheye cameras, which become blurry in low light conditions, significantly reducing the neural network's recognition capabilities and affecting the normal functioning of sentinel mode. Furthermore, sentinel perception typically uses four single-camera sensors, then fuses the results for target selection and tracking. Multi-camera fusion has always been a challenge and pain point in the industry, especially for truncated targets, where neither two adjacent cameras can fully capture the entire image, resulting in inaccurate detection information.
[0004] Therefore, existing multi-sensor fusion methods suffer from problems such as computational complexity and poor real-time performance. Furthermore, the depth images generated in complex environments have significant errors, making it difficult to fully utilize various types of information to improve the accuracy and real-time performance of target detection. Summary of the Invention
[0005] This application provides an environmental perception method, system, device, and medium based on vehicle sentry mode, which is used to avoid the influence of natural light and improve the robustness of sentry mode environmental perception.
[0006] In a first aspect, this application provides an environmental perception method based on vehicle sentry mode, comprising: acquiring color images and depth images from infrared cameras and acquiring intrinsic and extrinsic parameter information from each infrared camera, wherein the intrinsic and extrinsic parameter information includes intrinsic parameters of each infrared camera and extrinsic parameters from the infrared camera to the vehicle coordinate system; combining the intrinsic and extrinsic parameter information, using a pre-trained environmental perception network model to extract and fuse features from the color images and the depth images respectively to obtain fused features; the environmental perception network model using labels of lidar point cloud data and corresponding color and depth images as training data; decoding the fused features to determine obstacle information corresponding to the perceived target; dynamically tracking the perceived target based on the obstacle information to generate tracking information; and generating a decision result on whether the tracking information triggers a sentry warning based on preset alarm conditions.
[0007] In one implementation of the first aspect, the step of combining the intrinsic and extrinsic parameter information and using a pre-trained environment perception network model to extract and fuse features from the color image and the depth image to obtain fused features includes: extracting features from the color image using a 2D deep neural network; projecting the extracted features onto a vehicle coordinate system using the intrinsic and extrinsic parameter information to generate color image features; projecting the depth image onto a vehicle coordinate system using the intrinsic and extrinsic parameter information to obtain point cloud data of the depth image based on the vehicle coordinate system; extracting features from the point cloud data using a first 3D deep neural network to obtain depth image features; and fusing the color image features and the depth image features to obtain the fused features.
[0008] In one implementation of the first aspect, the color image includes a front-view color image, a rear-view color image, a left-view color image, and a right-view color image. The acquisition of the color image features includes: extracting features from the front-view color image, the rear-view color image, the left-view color image, and the right-view color image respectively using the deep neural network; and projecting the extracted front-view color image features, the rear-view color image features, the left-view color image features, and the right-view color image features into the vehicle coordinate system using an explicit projection method, in conjunction with the intrinsic and extrinsic parameter information, to generate the final color image features.
[0009] In one implementation of the first aspect, the depth image includes a forward-looking depth image, a rear-looking depth image, a left-looking depth image, and a right-looking depth image; the acquisition of the depth image features includes: projecting the forward-looking depth image, the rear-looking depth image, the left-looking depth image, and the right-looking depth image onto a vehicle coordinate system using the intrinsic and extrinsic parameter information, transforming them to obtain four depth images; fusing the four point cloud data based on the forward-looking point cloud data, the rear-looking point cloud data, the left-looking point cloud data, and the right-looking point cloud data in the vehicle coordinate system to obtain a fused point cloud; and extracting features from the fused point cloud based on a first 3D deep neural network to obtain the final depth image features.
[0010] In one implementation of the first aspect, fusing the color image features and the depth image features to obtain the fused features includes: adding, multiplying, or directly concatenating the color image features and the depth image features to obtain concatenated features; and further extracting features from the concatenated features based on a second 3D deep neural network to obtain the fused features.
[0011] In one implementation of the first aspect, training data for an environmental perception network model is obtained based on an environmental perception system. The environmental perception system includes multiple infrared cameras installed on a vehicle and at least one lidar. The pre-training steps of the environmental perception network model include: acquiring color images and depth images collected by the infrared cameras in different scenes as training data, and acquiring point cloud data collected by the lidar at the same time; labeling the training data based on the point cloud data; setting a training loss function before model training; and iteratively training the model using the labeled training data and the training loss function to obtain the environmental perception network model.
[0012] In one implementation of the first aspect, the step of annotating the training data based on point cloud data includes: transforming the point cloud data to a vehicle coordinate system using extrinsic parameters to obtain point cloud data in the vehicle coordinate system; annotating obstacle information on the point cloud data in the vehicle coordinate system; projecting the obstacle information onto a corresponding color image using the extrinsic and extrinsic parameter information; adjusting and updating the annotated obstacle information in the point cloud data according to the degree of matching between the obstacle information and the corresponding obstacle in the color image, until the projection of the obstacle information onto the corresponding color image coincides with the corresponding obstacle in the color image.
[0013] In one implementation of the first aspect, in response to the laser radar scanning across the center of the field of view of the infrared camera, the infrared camera is triggered to expose itself and begin acquiring the color image and the depth image.
[0014] Secondly, this application provides an environmental perception system based on vehicle sentry mode, comprising: an image acquisition module configured to acquire color images and depth images from an infrared camera; an environmental perception network model including an RGB module, a depth module, a feature fusion module, and a detection module; wherein, the RGB module is configured to extract features from the color images using a deep neural network to generate color image features; the depth module is configured to project the depth images onto a vehicle coordinate system, acquire point cloud data of the depth images based on the vehicle coordinate system, and extract features from the point cloud data to obtain depth image features; the feature fusion module is configured to fuse the color image features and the depth image features to obtain fused features; the detection module is configured to decode the fused features to determine obstacle information corresponding to the perceived target; a tracking module is configured to dynamically track the perceived target based on the obstacle information and generate tracking information; and a decision module is configured to combine preset alarm conditions to generate a decision result on whether the tracking information triggers a sentry warning.
[0015] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the method described thereon.
[0016] As described above, the environmental perception method, system, device, and medium based on vehicle sentinel mode described in this application have the following beneficial effects:
[0017] This application combines infrared cameras to acquire color and depth images, and then analyzes and detects them using a trained environment perception network model. In specific implementation, this application uses a fisheye camera with a built-in infrared emitter for sentinel perception, which is unaffected by natural light. By unifying perception to BEV (Bird's Eye View), it can solve the problem of multi-camera fusion, enabling comprehensive perception of the target's visual information and making the obtained detection information more accurate. Attached Figure Description
[0018] Figure 1 The diagram shows an application scenario of the environmental perception method based on vehicle sentinel mode described in this application embodiment.
[0019] Figure 2 The diagram shown is a flowchart illustrating the principle of the environmental perception method based on vehicle sentinel mode as described in this application embodiment.
[0020] Figure 3 The diagram shown is a data acquisition schematic of the environmental perception method based on vehicle sentinel mode described in an embodiment of this application.
[0021] Figure 4The diagram shown is a feature fusion flowchart of the environmental perception method based on vehicle sentinel mode described in this application embodiment.
[0022] Figure 5 The diagram shows a sensor installation schematic of the environmental perception method based on vehicle sentinel mode described in an embodiment of this application.
[0023] Figure 6 The diagram shown is a model training flowchart of the environmental perception method based on vehicle sentinel mode described in this application embodiment.
[0024] Figure 7 The diagram shown is a schematic diagram of the environmental perception system based on vehicle sentinel mode as described in the embodiments of this application.
[0025] Figure 8 The diagram shown is a functional principle diagram of the environmental perception system based on vehicle sentinel mode as described in the embodiments of this application.
[0026] Component designation explanation
[0027] 7 Environmental perception system based on vehicle sentinel mode 71 Image acquisition module 72 Environmental perception network model 721 RGB module 722 Depth module 723 Feature fusion module 724 Detection module 73 Tracking module 74 Decision module S21~S25 step S221~S223 step S61~S63 step Detailed Implementation
[0028] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.
[0029] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.
[0030] The following embodiments of this application provide an environmental perception method, system, device, and medium based on vehicle sentry mode, including but not limited to applications in a vehicle. The following description will use this hardware application scenario as an example.
[0031] Please see Figure 1 The image shown is a schematic diagram illustrating an application scenario of the environmental perception method based on vehicle sentry mode described in this application embodiment. Figure 1As shown, this embodiment provides a vehicle as a hardware application scenario, specifically including: sensors installed on the vehicle. The sensors include infrared cameras. In practical applications, the infrared cameras can be surround-view camera modules, with four surround-view camera modules installed at four gray dots on the front, rear, left, and right sides of the vehicle. Each surround-view camera module consists of a camera and infrared equipment, and can simultaneously acquire RGB images and depth images. The RGB image is a color image synthesized based on the three basic color channels: red, green, and blue.
[0032] In a specific embodiment, the infrared camera includes a fisheye camera with a built-in infrared emitter. The infrared light emitted by the fisheye camera is reflected when it encounters objects in the scene (such as faces, walls, cars, etc.). The reflected light patterns and intensities vary depending on the distance, shape, and material of the object. The camera is equipped with an infrared filter that blocks most visible light (to prevent ambient light interference), allowing only infrared light of the same wavelength as the emitted light source to pass through. Behind the filter, a dedicated infrared image sensor captures this reflected infrared light and converts it into an electrical signal. The raw electrical signal obtained by the sensor is processed to ultimately generate a color image and a depth image. The active infrared camera does not rely on ambient light; instead, it emits a beam of infrared light invisible to the human eye, so the camera is largely unaffected by lighting conditions.
[0033] The technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0034] Please see Figure 2 The diagram shows the principle flowchart of the environmental perception method based on vehicle sentry mode described in the embodiments of this application. Figure 2 As shown, this embodiment provides an environmental perception method based on vehicle sentry mode, applied to an environmental perception system detected by infrared cameras; the infrared cameras include four infrared cameras installed at the front, rear, left, and right of the vehicle; the environmental perception method based on vehicle sentry mode specifically includes the following steps:
[0035] S21, acquire the color image and depth image of the infrared camera, and acquire the intrinsic and extrinsic parameter information of each infrared camera, such as... Figure 3 As shown, the intrinsic and extrinsic parameter information includes the intrinsic parameters of each infrared camera and the extrinsic parameters of the infrared camera to the vehicle coordinate system.
[0036] S22, combining the aforementioned intrinsic and extrinsic parameter information, a pre-trained environment perception network model is used to extract and fuse features from the color image and the depth image respectively, obtaining fused features. The environment perception network model uses the labels of the LiDAR point cloud data and the corresponding color and depth images as training data.
[0037] Please see Figure 4 The diagram shows a feature fusion flowchart of the environmental perception method based on vehicle sentry mode described in this application embodiment. Figure 4 As shown, in one embodiment, the step of combining the intrinsic and extrinsic parameter information and using a pre-trained environment perception network model to extract and fuse features from the color image and the depth image respectively to obtain fused features includes:
[0038] S221, the color image is feature extracted using a 2D deep neural network, and the extracted features are projected onto the vehicle coordinate system in combination with the intrinsic and extrinsic parameter information to generate color image features.
[0039] S222, combine the intrinsic and extrinsic parameter information to project the depth image into the vehicle coordinate system, obtain the point cloud data of the depth image based on the vehicle coordinate system, and extract the features of the depth image by performing feature extraction on the point cloud data through a first 3D deep neural network.
[0040] S223, the color image features and the depth image features are fused to obtain the fused features.
[0041] Please see Figure 5 The image shown is a schematic diagram of the image distribution of the environmental perception method based on vehicle sentry mode described in the embodiments of this application. Figure 5 As shown, in one embodiment, the color image includes a front-view color image, a rear-view color image, a left-view color image, and a right-view color image; the depth image includes a front-view depth image, a rear-view depth image, a left-view depth image, and a right-view depth image; the step of combining the intrinsic and extrinsic parameter information and using a pre-trained environment perception network model to extract and fuse features from the color image and the depth image to obtain fused features includes:
[0042] (1) Steps for obtaining the color image features: The deep neural network is used to extract features from the front-view color image, the rear-view color image, the left-view color image and the right-view color image respectively. Combined with the intrinsic and extrinsic parameter information, the extracted front-view color image features, the rear-view color image features, the left-view color image features and the right-view color image features are projected into the vehicle coordinate system using an explicit projection method to generate the final color image features.
[0043] Specifically, in the RGB module, a DNN network is used to extract image features from the RGB images of the four cameras. Combined with the intrinsic and extrinsic parameters of the cameras, an explicit projection method is used to generate RGB BEV features (BF), with LSS being a representative of the explicit projection method.
[0044] (2) Steps for obtaining the depth image features: Project the front-view depth image, the rear-view depth image, the left-view depth image and the right-view depth image into the vehicle coordinate system by combining the intrinsic and extrinsic parameter information, and transform to obtain four depth images. Based on the front-view point cloud data, rear-view point cloud data, left-view point cloud data and right-view point cloud data of the vehicle coordinate system, fuse the four point cloud data to obtain a fused point cloud, and extract features from the fused point cloud based on the first 3D deep neural network to obtain the final depth image features.
[0045] Specifically, in the depth module, formula transformation is used to project the four depth maps onto the Vehicle Coordinate System (VCS) to obtain four point cloud data based on the VCS coordinate system. The four point cloud data are then directly stitched together to form BEV fused point cloud data. The point cloud feature extraction module extracts features from the BEV fused point cloud data to obtain depth BEV features (PF). The point cloud feature extraction module can refer to mainstream methods such as SECOND, PointPillars, and VoxelNet.
[0046] In this embodiment, the four depth maps are transformed into coordinate maps using the following formula:
[0047] (X, Y, Z) T = T ce * F -1 (x*d, y*d, d)
[0048] x, y, and d represent the x-coordinate, y-coordinate, and depth value of the depth map image, respectively. X, Y, and Z are the coordinates in the vehicle coordinate system. T ce It is the coordinate transformation matrix from the camera coordinate system to the vehicle coordinate system, and F is the camera intrinsic parameter.
[0049] (3) The color image features and the depth image features are fused to obtain the fused features. The fusion method is divided into two steps. The first step is to add, multiply or directly splice the color image features and the depth image features together to obtain spliced features. The second step is to extract the spliced features based on the second 3D deep neural network to obtain the fused features.
[0050] Specifically, the feature fusion module fuses RGB BEV features with deep BEV features to obtain high-order fused features (AFS). The fusion method generally consists of two steps. The first step is to add, multiply, or directly concatenate them together. The second step uses a DNN network to further process the output of the first step to obtain high-order BEV features. RGB BEV features are more inclined to provide rich texture, edge, gradient, and contrast information, while deep BEV features are more inclined to provide the shape and distance of objects. The fusion of the two can significantly improve the subsequent detection results, and deep BEV features can also enhance the model's resistance to ambient light interference.
[0051] Therefore, a fusion method is adopted to fuse the depth map and RGB image features of the infrared camera. After changing the internal and external parameters, the features of the depth map and RGB image of the infrared camera are unified under the BEV perspective. The target detector is used to decode the BEV features to obtain the corresponding target information. The target is tracked by the tracking algorithm. When the target distance or behavior may cause damage to the vehicle, the sentry alarm function is activated.
[0052] Color images (intensity images) provide rich information on texture, edges, gradients, and contrast. These are appearance features that pre-trained deep learning models (such as CNNs trained on ImageNet) excel at extracting. They are easy to acquire and have mature processing techniques, but are sensitive to changes in lighting and easily fail in situations with missing textures, shadows, or camouflage. They also lack information on the actual position and size of objects in 3D space. Depth maps provide information on the distance of each pixel to the camera (Z-axis information), directly reflecting the geometry of the scene. They are invariant to lighting and color: the shape and distance relationships of objects remain unchanged regardless of day or night. They simplify background separation, easily separating foreground objects from the background based on distance; they provide scale information, allowing inference of the true physical size of objects based on their actual distance, helping to distinguish objects that look similar but are different in size (e.g., a distant truck vs. a nearby car), but are typically low resolution; data may be missing or noisy in areas such as reflective surfaces (glass, water) or infinity (sky).
[0053] The information from these two modalities is highly complementary. Appearance information (grayscale) tells you "what it is," while geometric information (depth) tells you "where it is and how big it is." By combining them, the environmental perception network model in this application can make more reliable and accurate judgments.
[0054] S23, decode the fused features to determine the obstacle information corresponding to the perceived target; the obstacle information includes the target's position information, category information, size information, and orientation angle information around the Z-axis.
[0055] Specifically, the fused features are decoded to obtain the target's position information (x, y), category information (type), size information (w, h, l), and orientation angle information (yaw) around the Z-axis.
[0056] In practical applications, both color image features and depth image features involve fusing data from four cameras. This fusing operation transforms all data into the VCS coordinate system, and the fused data contains obstacle information from 360° around the vehicle. This unified processing of the data avoids the problems caused by multi-camera fusion.
[0057] In this embodiment, the dimensions of the obtained RGB BEV feature, deep BEV feature and high-order BEV feature are all 128*128*9, each voxel represents 0.5m, and the vehicle is located at the center of the bottom surface of the voxel feature. That is to say, the voxel feature represents the spatial features of the vehicle in front and behind 32m and with a height of 4.5m.
[0058] S24, dynamically track the perceived target based on the obstacle information and generate tracking information; the tracking information includes the target's speed, appearance time, and acceleration.
[0059] S25, based on preset alarm conditions, generate a decision result on whether the tracking information triggers a sentry warning.
[0060] Specifically, the preset alarm conditions may include the vehicle's movement status, the target object being close to the vehicle, or vehicle parts being in an abnormal state or damaged.
[0061] The warning conditions for vehicle sentry mode mainly include the following situations:
[0062] (1) Threat detected by the sensor: After the vehicle is locked and put into parking gear, if the sensor detects that the vehicle is being towed, shaken or violently bumped (such as being dragged), an alarm will be triggered. When someone leans against the vehicle or approaches the vehicle, the sentry mode will switch to alert mode, which will be manifested by the headlights flashing and the touch screen displaying a message.
[0063] (2) Alarm triggered by dangerous behavior: If dangerous behavior such as smashing car windows is detected, Sentry Mode will enter alarm state, activate audio alarm, display the message at high brightness on the touch screen, and send a notification to the mobile application. When the alarm is triggered, the system will record a video clip and save it to the USB drive.
[0064] Please see Figure 6The diagram shows a model training flowchart for the environmental perception method based on vehicle sentry mode described in this application embodiment. In one embodiment, training data for the environmental perception network model is acquired based on an environmental perception system. The environmental perception system includes multiple infrared cameras installed on the vehicle and at least one lidar. In practical applications, the infrared cameras can be surround-view camera modules, and the lidar is installed at the black dot position at the center of the vehicle, i.e., as shown below. Figure 1 The vehicle's coordinate center point should be noted. It should be noted that the lidar is only set during the training phase of the environmental perception network model and does not exist when the environmental perception network model is used for real-time environmental perception applications.
[0065] The pre-training steps of the environment-aware network model include:
[0066] S61, acquire color images and depth images collected by infrared cameras in different scenarios as training data, and acquire point cloud data collected by lidar at the same time, and label the training data based on the point cloud data.
[0067] Specifically, the data collection vehicle will travel to various scenarios, record a large amount of data, and filter out the better data. In practical applications, the four infrared cameras will be calibrated; in response to the lidar scanning the center of the field of view of the infrared cameras, the infrared cameras will be triggered to expose and begin acquiring the color image and the depth image.
[0068] Specifically, the four surround-view camera modules are calibrated to obtain the intrinsic parameters F of each camera (a total of four), and the extrinsic parameters T of each camera module and the LiDAR to the vehicle coordinate system (a total of four). ce and 1 T Le Vehicle coordinate system as follows Figure 1 As shown, X is aligned with the direction the car is facing, Y is to the left, and Z is upward.
[0069] Specifically, data synchronization is performed during the acquisition of color and depth images. To achieve good cross-modal data alignment between the LiDAR and camera modules, the camera exposure is triggered only when the top LiDAR scans the center of the camera's field of view (FOV), and camera data acquisition begins. After the LiDAR rotates once, it acquires four channels of camera module data, and these four channels of camera module data, along with the LiDAR data at that moment, are saved as a single frame.
[0070] In one embodiment, the step of labeling the training data based on point cloud data includes:
[0071] (1) Transform the point cloud data into the vehicle coordinate system using extrinsic parameters.
[0072] Specifically, since it is difficult to obtain 3D information about obstacles from image data, it is necessary to annotate the LiDAR data. However, the final output of the environmental perception network model is in the vehicle coordinate system, so the LiDAR data needs to be transformed to the vehicle coordinate system through extrinsic parameters.
[0073] (2) Multiply the transformation matrix of the radar to the vehicle with the matrix of the laser radar data to obtain point cloud data in the vehicle coordinate system.
[0074] Specifically, assuming the lidar data is P L The transformation formula is:
[0075] P e =T Le *P L
[0076] Among them, P e For point cloud data in the vehicle coordinate system, T Le It is the transformation matrix from radar to vehicle.
[0077] (3) Mark the obstacle information on the point cloud data in the vehicle coordinate system.
[0078] Specifically, obstacle information (D) is labeled on the data Pe using relevant software. L The information about obstacles generally includes location information (gx, gy), category information (gtype), size information (gw, gh, gl), orientation angle information (gyaw), etc. Since it is assumed that the obstacles are all on the ground plane, the obstacle height z is generally not output, but can be calculated later from the obstacle height h.
[0079] Then, the obstacle information is projected onto the corresponding color image based on the intrinsic and extrinsic parameter information. According to the degree of matching between the obstacle information and the corresponding obstacle in the color image, the obstacle information marked in the point cloud data is adjusted and updated until the projection of the obstacle information on the corresponding color image coincides with the corresponding obstacle in the color image.
[0080] S62 sets the training loss function before model training.
[0081] In one embodiment, the step of setting the training loss function before model training includes:
[0082] (1) During training, the location labels in the labeled obstacle information are converted into anchor point offsets; the anchor point offsets are used to generate the labels of the prediction boxes, and then participate in the calculation process under different loss calculation methods.
[0083] Specifically, different loss categories are set according to the output data of the detection module. The location information is obtained based on the anchor point method during decoding, that is, the raw output of the location information is based on the offset (△x, △y) of the anchor point. Therefore, it is necessary to add the labeled D during training. L The location labels in the information are converted into anchor-based offsets (△gx, △gy).
[0084] The relationship between anchor point offset and loss calculation is mainly reflected in object detection. Anchor point offset is used to generate the labels of the predicted bounding boxes, which then participate in the calculation of the loss function. The specific relationship is as follows:
[0085] Label generation: For positive samples, the Anchor needs to be calculated to offset from the Ground Truth (gt) as the label for the regression loss, while negative samples are ignored.
[0086] (2) For the location problem, orientation angle information and size information, the regression task loss method is used to calculate the loss.
[0087] Specifically, location, orientation angle, and size information fall under the category of regression problems, so smoothL1 loss is used to calculate the loss. In the regression loss calculation, the difference in offset between the predicted bounding box of a positive sample and the ground truth (such as smoothL1 loss) directly affects the bounding box regression calculation.
[0088] (3) For category information, the cross-entropy loss method is used to calculate the difference between category information and detection results.
[0089] Specifically, since category information falls under the category of classification problems, cross-entropy loss is used to calculate D. L The difference between the category information gtype in the information and the detection result type. In the classification loss calculation, the objectness score (whether the target is contained) and the classification score (target category) of the positive sample are both involved in the loss calculation.
[0090] (4) Different loss weights are set according to the different impacts of each task on the environmental perception network model. The final loss function is the weighted sum of all losses.
[0091] S63, the environment-aware network model is obtained by iteratively training the labeled training data and the training loss function multiple times.
[0092] The scope of protection of the environmental perception method based on vehicle sentinel mode described in this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.
[0093] This application also provides an environmental perception system based on vehicle sentry mode. The environmental perception system based on vehicle sentry mode can implement the environmental perception method based on vehicle sentry mode described in this application. However, the implementation device of the environmental perception method based on vehicle sentry mode described in this application includes, but is not limited to, the structure of the environmental perception system based on vehicle sentry mode listed in this embodiment. All structural modifications and substitutions of the prior art made in accordance with the principles of this application are included within the protection scope of this application.
[0094] Please see Figure 7 The diagram shows the structural principle of the environmental perception system based on vehicle sentry mode described in this application embodiment. Figure 7 As shown, this embodiment provides an environment perception system 7 based on vehicle sentry mode, including: an image acquisition module 71, an environment perception network model 72, a tracking module 73, and a decision module 74.
[0095] The image acquisition module 71 is configured to acquire color images and depth images from the infrared camera.
[0096] The environmental perception network model 72 includes an RGB module 721, a depth module 722, a feature fusion module 723, and a detection module 724. The RGB module 721 is configured to extract features from the color image using a deep neural network to generate color image features. The depth module 722 is configured to project the depth image onto a vehicle coordinate system, obtain point cloud data of the depth image based on the vehicle coordinate system, and extract features from the point cloud data to obtain depth image features. The feature fusion module 723 is configured to fuse the color image features and the depth image features to obtain the fused features. The detection module 724 is configured to decode the fused features to determine the obstacle information corresponding to the perceived target.
[0097] The tracking module 73 is configured to dynamically track the perceived target based on the obstacle information and generate tracking information.
[0098] The decision module 74 is configured to generate a decision result on whether the tracking information triggers a sentry warning, based on preset alarm conditions.
[0099] Please see Figure 8 The diagram shows the functional principle of the environmental perception system based on vehicle sentry mode described in this application embodiment. Figure 8As shown, the principle of the environmental perception system based on vehicle sentry mode is as follows: RGB and depth images from four infrared cameras are acquired, and the intrinsic and extrinsic parameters of the four cameras are combined. An environmental perception network model 72 processes the data from the four surround-view camera modules. Specifically, color image features are acquired through the RGB module 721, depth image features are acquired through the depth module 722, fused features are obtained through the feature fusion module 723, and obstacle information is obtained by decoding the fused features through the detection module 724. The obstacle information is input to the tracking module 73 to track obstacles. The tracking module 73 can acquire tracking information such as the target's speed, appearance time, and acceleration. The tracking information is input to the decision module 74, which selects whether to trigger sentry alarms or other functions based on set conditions.
[0100] In one embodiment, the RGB module is further configured to extract features from the color image using a 2D deep neural network, and project the extracted features into the vehicle coordinate system in combination with the intrinsic and extrinsic parameter information to generate color image features.
[0101] In one embodiment, the depth module is further configured to project the depth image onto a vehicle coordinate system by combining the intrinsic and extrinsic parameter information, obtain point cloud data of the depth image based on the vehicle coordinate system, and extract depth image features from the point cloud data by a first 3D deep neural network.
[0102] In one embodiment, the color image includes a front-view color image, a rear-view color image, a left-view color image, and a right-view color image; the depth image includes a front-view depth image, a rear-view depth image, a left-view depth image, and a right-view depth image.
[0103] Specifically, the RGB module is further configured to extract features from the front-view color image, the rear-view color image, the left-view color image, and the right-view color image respectively through the deep neural network, and to project the extracted front-view color image features, the rear-view color image features, the left-view color image features, and the right-view color image features into the vehicle coordinate system using an explicit projection method, in combination with the intrinsic and extrinsic parameter information, to generate the final color image features.
[0104] Specifically, the depth module is further configured to project the forward-looking depth image, the rear-looking depth image, the left-looking depth image, and the right-looking depth image onto the vehicle coordinate system by combining the intrinsic and extrinsic parameter information, and transform them to obtain four depth images. Based on the forward-looking point cloud data, rear-looking point cloud data, left-looking point cloud data, and right-looking point cloud data of the vehicle coordinate system, the four point cloud data are fused to obtain a fused point cloud. Based on the first 3D deep neural network, features are extracted from the fused point cloud to obtain the final depth image features.
[0105] Specifically, the feature fusion module is further configured to fuse the color image features and the depth image features to obtain the fused features; the fusion method consists of two steps: the first step is to add, multiply, or directly concatenate the color image features and the depth image features together to obtain the concatenated features; the second step is to further extract features from the concatenated features based on a second 3D deep neural network to obtain the fused features.
[0106] In one embodiment, the environment perception system includes multiple infrared cameras installed on the vehicle and at least one lidar; the pre-training of the environment perception network model includes: acquiring color images and depth images collected by the infrared cameras in different scenes as training data, and acquiring point cloud data collected by the lidar at the same time; labeling the training data based on the point cloud data; setting a training loss function before model training; and obtaining the environment perception network model through multiple iterations of training with the labeled training data and the training loss function.
[0107] In the embodiments provided in this application, it should be understood that the disclosed systems or methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or modules or units, and may be electrical, mechanical, or other forms.
[0108] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this application, depending on actual needs. For example, the functional modules / units in the various embodiments of this application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.
[0109] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0110] In practical applications, the environmental perception method based on vehicle sentry mode is executed by an electronic device. This electronic device can be the main control unit within the vehicle. The infrared camera on the vehicle transmits the acquired color and depth images to the electronic device, which then executes the environmental perception method based on vehicle sentry mode. Finally, combined with preset alarm conditions, it generates a decision result regarding whether the tracking information triggers a sentry warning.
[0111] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed, implements the environmental perception method based on vehicle sentry mode.
[0112] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. This available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive (SSD)).
[0113] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.
[0114] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.
Claims
1. An environmental perception method based on vehicle sentry mode, characterized in that, include: Acquire color and depth images from infrared cameras and acquire intrinsic and extrinsic parameter information for each infrared camera, including intrinsic parameters of each infrared camera and extrinsic parameters of the infrared camera to the vehicle coordinate system; Combining the intrinsic and extrinsic parameter information, a pre-trained environment perception network model is used to extract and fuse features from the color image and the depth image respectively to obtain fused features. This includes: extracting features from the color image using a 2D deep neural network; projecting the extracted features onto a vehicle coordinate system using the intrinsic and extrinsic parameter information to generate color image features; projecting the depth image onto a vehicle coordinate system using the intrinsic and extrinsic parameter information to obtain point cloud data of the depth image based on the vehicle coordinate system; extracting features from the point cloud data using a first 3D deep neural network to obtain depth image features; and fusing the color image features and the depth image features to obtain the fused features. The environment perception network model uses labels from LiDAR point cloud data and corresponding color and depth images as training data. The LiDAR is only set during the training phase of the environment perception network model and is not set up during subsequent real-time environment perception applications using the environment perception network model. The fused features are decoded to determine the obstacle information corresponding to the perceived target; Based on the obstacle information, the perceived target is dynamically tracked to generate tracking information; Based on preset alarm conditions, a decision result is generated as to whether the tracking information triggers a sentinel warning.
2. The method according to claim 1, characterized in that, The color image includes a front-view color image, a rear-view color image, a left-view color image, and a right-view color image. The acquisition of the color image features includes: The 2D deep neural network extracts features from the front-view color image, the rear-view color image, the left-view color image, and the right-view color image, respectively. The extracted features are then projected onto the vehicle coordinate system using an explicit projection method, based on the intrinsic and extrinsic parameters, to generate the final color image features.
3. The method according to claim 1, characterized in that, The depth image includes a forward-looking depth image, a backward-looking depth image, a left-looking depth image, and a right-looking depth image; the acquisition of the depth image features includes: The forward-looking depth image, the rear-looking depth image, the left-looking depth image, and the right-looking depth image are projected onto the vehicle coordinate system using the aforementioned intrinsic and extrinsic parameter information, and transformed to obtain four depth images. Based on the forward-looking point cloud data, rear-looking point cloud data, left-looking point cloud data, and right-looking point cloud data of the vehicle coordinate system, the four point cloud data are fused to obtain a fused point cloud. Based on the first 3D deep neural network, feature extraction is performed on the fused point cloud to obtain the final depth image features.
4. The method according to claim 1, characterized in that, The fused features obtained by fusing the color image features and the depth image features include: The color image features and the depth image features are added, multiplied, or directly concatenated together to obtain the concatenated features; The spliced features are further extracted based on the second 3D deep neural network to obtain the fused features.
5. The method according to claim 1, characterized in that, Training data for an environmental perception network model is acquired based on an environmental perception system, which includes multiple infrared cameras and at least one lidar mounted on the vehicle. The pre-training steps for the environmental perception network model include: Color and depth images captured by infrared cameras in different scenarios are acquired as training data, and point cloud data collected by lidar at the same time are acquired. The training data is then labeled based on the point cloud data. Set the training loss function before model training; The environment-aware network model is obtained by iteratively training the labeled training data and the training loss function.
6. The method according to claim 5, characterized in that, The step of labeling the training data based on the point cloud data includes: The point cloud data is transformed to the vehicle coordinate system using extrinsic parameters to obtain point cloud data in the vehicle coordinate system. Obstacle information is marked on the point cloud data in the vehicle coordinate system; The obstacle information is projected onto the corresponding color image by combining the intrinsic and extrinsic parameters. The obstacle information marked in the point cloud data is adjusted and updated according to the degree of matching between the obstacle information and the corresponding obstacle in the color image, until the projection of the obstacle information on the corresponding color image coincides with the corresponding obstacle in the color image.
7. The method according to claim 5, characterized in that, When the lidar sweeps across the center of the infrared camera's field of view, the infrared camera is triggered to expose itself and begin acquiring the color image and the depth image.
8. An environmental perception system based on vehicle sentry mode, characterized in that, The method described in any one of claims 1-7 comprises: The image acquisition module is configured to acquire color and depth images from the infrared camera; The processing module based on the environmental perception network model includes an RGB module, a depth module, a feature fusion module, and a detection module. The RGB module is configured to extract features from the color image using a 2D deep neural network to generate color image features. The depth module is configured to project the depth image onto a vehicle coordinate system, obtain point cloud data of the depth image based on the vehicle coordinate system, and extract features from the point cloud data to obtain depth image features. The feature fusion module is configured to fuse the color image features and the depth image features to obtain the fused features. The detection module is configured to decode the fused features to determine the obstacle information corresponding to the perceived target. The tracking module is configured to dynamically track the perceived target based on the obstacle information and generate tracking information. The decision module is configured to generate a decision result on whether the tracking information triggers a sentinel warning, based on preset alarm conditions.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the method described in any one of claims 1 to 7.
Citation Information
Patent Citations
Indoor scene surface reconstruction method and device, equipment and storage medium
CN117058334A
Target object position determination method and device, electronic equipment and storage medium
CN117765063A
Vehicle sentry mode environment sensing method and system and electronic equipment
CN121106088A
