Storage location identification method and device, equipment and storage medium
The method enhances parking space recognition by using fisheye images and neural networks to generate a bird's-eye view, addressing environmental challenges and improving accuracy for automatic parking systems.
Patent Information
- Application Number
- CN202411038049.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-12-06
AI Technical Summary
Existing vision-based parking technologies face challenges in complex environments due to light changes, obstructions, and shadows, leading to reduced image quality and recognition accuracy.
A method and device that utilize a vehicle-mounted camera to acquire fisheye images at low speeds, generate a target bird's-eye view BEV image, and employ a neural network model to identify parking spaces, incorporating image processing and fusion techniques to enhance accuracy.
Provides real-time, accurate parking space information by supplementing the vehicle's chassis image, ensuring quick and efficient automatic parking decisions.
Smart Images

Figure CN119091415A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to parking technology, and relate to but are not limited to a parking location identification method and device, equipment, and storage medium. Background Art
[0002] Among the existing automatic parking technologies, vision-based parking space detection technology has the advantages of high precision and high real-time performance. It can accurately identify and locate parking spaces in a short time, providing reliable navigation information for automatic parking of vehicles. However, in complex environments, such as light changes, obstructions, shadows, etc., image quality and recognition accuracy will be interfered with.
[0003] Therefore, how to improve image quality and recognition accuracy is an urgent problem to be solved. Summary of the invention
[0004] In view of this, the embodiments of the present application provide a method and device for identifying parking spaces, equipment, and storage medium, which can instantly acquire and process image data when the vehicle is moving at a low speed, supplement the chassis image of the vehicle, and provide real-time parking space information. The embodiments of the present application provide a method and device for identifying parking spaces, equipment, and storage medium, which are implemented as follows:
[0005] A storage location identification method provided in an embodiment of the present application includes:
[0006] When the speed of the vehicle is lower than a preset speed threshold, a plurality of fisheye images are acquired through the vehicle-mounted camera;
[0007] Obtaining a target bird's-eye view BEV image according to the plurality of fisheye images, wherein a target area for representing a chassis of the vehicle in the target bird's-eye view BEV image is smaller than an actual area of the chassis of the vehicle;
[0008] According to the target bird's-eye view BEV image and a preset neural network model, an identification result of the storage location in the target bird's-eye view BEV image is obtained. The preset neural network model is obtained by training an initial neural network model based on sample data collected in a storage location identification scenario. The sample data includes multiple historical images collected in the storage location identification scenario and multiple historical storage location information in the historical storage location identification scenario.
[0009] In some embodiments, obtaining a bird's-eye view BEV image of the target according to the plurality of fisheye images includes:
[0010] According to the plurality of fisheye images, a plurality of surround-view bird's-eye view BEV images are obtained;
[0011] Acquire driving data corresponding to the plurality of fisheye images, and obtain a motion trajectory of the vehicle in a world coordinate system according to the driving data;
[0012] The target bird's-eye view BEV image is obtained according to the multiple surround bird's-eye view BEV images and the motion trajectory.
[0013] In some embodiments, the plurality of surround bird's-eye view BEV images are images that do not include a vehicle chassis area, the plurality of surround bird's-eye view BEV images include a plurality of historical surround bird's-eye view BEV images and a current surround bird's-eye view BEV image, and the target bird's-eye view BEV image is obtained according to the plurality of surround bird's-eye view BEV images and the motion trajectory, including:
[0014] Performing fusion and superposition processing on the multiple historical surround bird's-eye view BEV images according to the motion trajectory to obtain an image of the vehicle chassis area of the current surround bird's-eye view BEV image;
[0015] The image of the vehicle chassis area is filled into the vehicle chassis area of the current surround bird's-eye view BEV image to obtain the target bird's-eye view BEV image.
[0016] In some embodiments, the plurality of surround bird's-eye view BEV images are images that do not include a vehicle chassis area, the plurality of surround bird's-eye view BEV images include historical surround bird's-eye view BEV images and current surround bird's-eye view BEV images, and the target bird's-eye view BEV image is obtained according to the plurality of surround bird's-eye view BEV images and the motion trajectory, further comprising:
[0017] The historical surround bird's-eye view BEV image is superimposed and fused onto the current surround bird's-eye view BEV image according to the motion trajectory to obtain the target bird's-eye view BEV image.
[0018] In some embodiments, the historical bird's-eye view BEV image is superimposed and fused onto the current bird's-eye view BEV image according to the motion trajectory to obtain a target bird's-eye view BEV image, including:
[0019] Acquire a current visible area and a current invisible area in the current surround bird's-eye view BEV image, and a historical visible area and a historical invisible area in each of the historical surround bird's-eye view BEV images, wherein the invisible area corresponds to the position of the vehicle chassis in the corresponding surround bird's-eye view BEV image;
[0020] According to the motion trajectory and the preset superposition rule, the historical surround bird's-eye view BEV image is sequentially superimposed and fused with the current surround bird's-eye view BEV image to obtain a target bird's-eye view BEV image;
[0021] The preset superposition rules include:
[0022] In the case where the current invisible area overlaps with the historical visible area, the historical visible area is retained; and in the case where the current visible area overlaps with the historical visible area, the current visible area is retained.
[0023] In some embodiments, obtaining a plurality of surround bird's-eye view BEV images according to the plurality of fisheye images includes:
[0024] Acquire calibration parameters of the vehicle-mounted camera, and obtain multiple fisheye images in a world coordinate system according to the calibration parameters;
[0025] Performing spatial alignment on the multiple fisheye images in the world coordinate system to obtain multiple aligned fisheye images, wherein the multiple fisheye images include overlapping image areas and non-overlapping image areas;
[0026] Pixel filling is performed on non-overlapping image areas of the multiple fisheye images, and weighted filling is performed on overlapping image areas of the multiple fisheye images to obtain multiple surround bird's-eye view BEV images.
[0027] In some embodiments, obtaining the recognition result of the storage location in the target bird's-eye view BEV image according to the target bird's-eye view BEV image and a preset neural network model includes:
[0028] Performing multi-layer feature sampling on the target bird's-eye view BEV image to obtain an effective feature map;
[0029] The effective feature map is subjected to target detection to obtain a recognition result of the storage location in the target bird's-eye view BEV image, wherein the recognition result includes the position of the storage location, the ground mark position of the storage location, and the confidence level of each position.
[0030] An embodiment of the present application provides a storage location identification device, comprising:
[0031] An acquisition module, used for acquiring a plurality of fisheye images through a vehicle-mounted camera when the speed of the vehicle is lower than a preset speed threshold;
[0032] The acquisition module is further used to obtain a target bird's-eye view BEV image according to the multiple fisheye images, wherein a target area for representing the chassis of the vehicle in the target bird's-eye view BEV image is smaller than an actual area of the chassis of the vehicle;
[0033] A generation module is used to obtain the recognition result of the storage location in the target bird's-eye view BEV image based on the target bird's-eye view BEV image and a preset neural network model, wherein the preset neural network model is obtained by training an initial neural network model based on sample data collected in a storage location recognition scenario, and the sample data includes multiple historical images collected in the storage location recognition scenario and multiple historical storage location information in the historical storage location recognition scenario.
[0034] The computer device provided in the embodiment of the present application includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the method described in the embodiment of the present application is implemented.
[0035] The computer-readable storage medium provided in the embodiment of the present application stores a computer program thereon, and when the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.
[0036] The parking space identification method, device, computer equipment and computer-readable storage medium provided in the embodiments of the present application can provide real-time parking space information by acquiring and processing image data in real time when the vehicle is moving at a low speed, supplementing the chassis image of the vehicle, and thereby ensuring that the automatic parking system can respond quickly and make accurate decisions, making the entire parking process more efficient and solving the technical problems raised in the background technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.
[0038] Figure 1 This is an application scenario diagram of a storage location identification method disclosed in an embodiment of the present application;
[0039] Figure 2 It is a schematic diagram of the implementation process of a storage location identification method disclosed in an embodiment of the present application;
[0040] Figure 3 It is a schematic diagram of the implementation process of another storage location identification method disclosed in the embodiment of the present application;
[0041] Figure 4 It is a schematic diagram of the implementation process of another storage location identification method disclosed in the embodiment of the present application;
[0042] Figure 5 It is a schematic diagram of the implementation process of another storage location identification method disclosed in the embodiment of the present application;
[0043] Figure 6 is a schematic diagram of generating a surround-view BEV image disclosed in an embodiment of the present application;
[0044] Figure 7 is a schematic diagram of generating a target BEV image disclosed in an embodiment of the present application;
[0045] Figure 8 It is a workflow diagram of a storage location identification method disclosed in an embodiment of the present application;
[0046] Fig. 9 is a schematic diagram of generating a surround-view BEV image disclosed in an embodiment of the present application;
[0047] Fig.10 It is a coordinate conversion diagram of a storage location identification method disclosed in an embodiment of the present application;
[0048] Fig.11 is a schematic diagram of generating a target BEV image disclosed in an embodiment of the present application;
[0049] Fig.12 It is a structural schematic diagram of a storage location identification device disclosed in an embodiment of the present application;
[0050] Fig.13 It is a computer device disclosed in an embodiment of the present application. DETAILED DESCRIPTION
[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.
[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0053] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0054] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are used to distinguish similar or different objects, and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.
[0055] In view of this, an embodiment of the present application provides a storage location identification method, which is applied to a vehicle-mounted system. Figure 1 FIG. 1 is an application scenario diagram of a storage location identification method in an embodiment. Figure 1 As shown, the user drives the vehicle 10. The functions implemented by the method can be implemented by the processor in the vehicle 10 calling the program code. Of course, the program code can be stored in a computer storage medium. It can be seen that the vehicle 10 at least includes a processor and a storage medium.
[0056] Figure 2 The following is a schematic diagram of the implementation process of a storage location identification method provided in an embodiment of the present application. Figure 2 As shown, the method may include the following steps 201 to 203:
[0057] Step 201 , when the speed of the vehicle is lower than a preset speed threshold, a plurality of fisheye images are acquired through a vehicle-mounted camera.
[0058] In an embodiment of the present application, the system monitors the real-time speed of the vehicle, and when the speed is lower than a preset speed threshold, the storage location identification process is triggered. This speed threshold can be set to the typical speed of the vehicle when it is moving slowly or preparing to park, and can be pre-set based on empirical data. The on-board camera captures multiple fisheye images, each covering a different perspective and environmental information around the vehicle. These images first undergo preprocessing steps, such as lens distortion correction and image enhancement, to optimize subsequent processing and analysis processes.
[0059] Step 202 : obtaining a target bird's-eye view BEV image based on the multiple fisheye images, wherein a target area for representing the chassis of the vehicle in the target bird's-eye view BEV image is smaller than an actual area of the chassis of the vehicle.
[0060] In the embodiment of the present application, multiple fisheye images can be used to generate a target bird's-eye view BEV image through geometric transformation and image fusion technology. The fisheye image is converted into a bird's-eye view to more clearly show the parking scene around the vehicle and ensure accurate representation of the chassis area.
[0061] Step 203 , obtaining a recognition result of a storage location in the target bird's-eye view BEV image according to the target bird's-eye view BEV image and a preset neural network model.
[0062] In an embodiment of the present application, the initial neural network model is trained using the collected sample data. These sample data include multiple historical images taken in different parking scenarios, and the known storage location information in these images. During the training process, the model optimizes parameters through the back propagation algorithm to maximize the accuracy of predicting the location and status of the storage location. During real-time storage location identification, the generated target bird's-eye view BEV image is input into the trained neural network model. The model analyzes the feature points and patterns in the image through forward propagation calculations, and determines the location and availability of each storage location. The neural network model outputs the recognition results for each storage location. These results can be used directly to display on the driver information interface, or integrated with the vehicle's autonomous driving system to support autonomous parking operations. The recognition results usually include the location coordinates of the storage location, whether it is idle, and other relevant information to help users quickly find a suitable parking space.
[0063] The embodiment of the present application can improve the recognition accuracy of the storage location by triggering image acquisition at low speed and using multiple fisheye images to generate a target bird's-eye view BEV image. The bird's-eye view helps to reduce distortion and occlusion in the image, making the representation of the chassis area clearer and more accurate. Using a neural network model to analyze the target bird's-eye view BEV image can quickly and accurately determine the available parking spaces, thereby improving the efficiency of the parking process.
[0064] Furthermore, the recognition result of the storage location in the target bird's-eye view BEV image is obtained based on the target bird's-eye view BEV image and a preset neural network model, including: performing multi-layer feature sampling on the target bird's-eye view BEV image to obtain a valid feature map.
[0065] Specifically, the acquired bird's-eye view BEV image of the target needs to be preprocessed to ensure the quality and accuracy of the image. This includes image denoising, brightness adjustment, and any necessary color space conversion so that the image can be effectively used in subsequent processing steps. Use multi-scale feature extraction techniques, such as processing methods based on convolutional neural networks (CNNs), to extract multi-layer features from the bird's-eye view BEV image of the target. In CNN, through a series of convolutional layers and pooling layers, the spatial size of the image is gradually reduced, while the depth and abstractness of the features are increased. It is determined according to the needs of the specific application and the complexity of the image. In CNN, features are extracted at different levels (after multiple convolutional layers or pooling layers). The feature map of each level can be understood as a different abstraction and expression of the original image. The deeper feature map usually has more advanced semantic information, while the shallower feature map is closer to the visual features of the original image. After the multi-layer feature extraction, these features are merged and processed so that the subsequent target detection and recognition tasks can be carried out effectively.
[0066] The feature maps at different levels are fused according to certain rules, such as simple superposition, weighted superposition or more complex feature fusion methods (such as attention mechanism). The fused feature maps are further processed, such as normalization, scale adjustment or other necessary preprocessing steps, to ensure that the data finally input to the target detection model is consistent and valid. Ultimately, the whole process will produce a set of valid feature maps, which not only retain the information of the original target bird's-eye view BEV image, but also add higher-level abstract features. These feature maps will be used as input to the target detection model in subsequent steps to identify and locate objects in the environment, such as storage locations or other obstacles.
[0067] Furthermore, the effective feature map is subjected to target detection to obtain a recognition result of the storage location in the target bird's-eye view BEV image, wherein the recognition result includes the position of the storage location, the ground mark position of the storage location and the confidence of each position.
[0068] Specifically, select an object detection model suitable for processing the target bird's-eye view BEV image. Common choices include: YOLO (You Only Live Once) is a real-time object detection algorithm that can directly output the location and category of the target in a single forward propagation. Faster R-CNN (Fast Region-based Convolutional Neural Networks) is a classic object detection framework that first generates candidate boxes and then classifies them through a two-stage detection method. SSD (Single Shot MultiBox Detector) is also a single-stage detector that combines multi-scale feature maps for detection and positioning. The valid feature maps obtained from the preprocessing and multi-layer feature extraction steps are used as input to the object detection model. These feature maps should be appropriately resized and normalized to meet the input requirements of the object detection model. Based on the input feature maps, the model predicts the location information of each storage location in the target bird's-eye view BEV image, usually given in the form of a rectangular bounding box. For each detected bounding box, the model can also predict its category, mainly the storage location here. Each detection result is accompanied by a confidence score, which indicates the model's confidence in the detection result. For each detected storage location, its bounding box information can be further processed, for example, based on the size and position of the storage location, its ground marker position in the target bird's-eye view BEV image is calculated. This may involve mapping image coordinates to actual geographic coordinates, and calculating the position information of each detected storage location based on the relative position of the vehicle coordinate system and the ground marker, usually given in the form of a rectangular bounding box. The ground marker position of each storage location, that is, the actual position of the storage location on the ground. The confidence score of each detection result indicates the confidence of the model in the result.
[0069] The embodiment of the present application can effectively extract key features in the image by performing multi-layer feature sampling on the target bird's-eye view BEV image. This helps to capture features of different scales and complexities and improve the accuracy and robustness of warehouse location recognition. By using effective feature maps for target detection, the warehouse location in the target bird's-eye view BEV image can be accurately identified.
[0070] In the above Figure 2 In the case shown, the embodiment of the present application also provides a schematic diagram of the implementation process of the storage location identification method. Figure 3 As shown, the method may include the following steps 301 to 305:
[0071] Step 301, when the speed of the vehicle is lower than a preset speed threshold, a plurality of fisheye images are acquired through the vehicle-mounted camera.
[0072] The implementation process of step 301 is consistent with the above-mentioned step 201, and this application will not repeat it here.
[0073] Step 302: obtaining a plurality of surround-view bird's-eye view BEV images according to the plurality of fisheye images.
[0074] In an embodiment of the present application, the fisheye image is first corrected. The image captured by the fisheye lens will be distorted, and geometric correction is required to restore the true scene geometry. For example, the fisheye image can be converted into a corrected perspective image through a geometric model or a computer vision algorithm (such as a distortion correction algorithm). The corrected multiple fisheye images are stitched together to form an image of a large surrounding area. This can be achieved through an image stitching algorithm, such as a panoramic image stitching algorithm or a multi-view image stitching technology. By projecting the stitched fisheye image onto a plane, multiple bird's-eye view (BEV) images are generated. The BEV image takes the vehicle as the center and presents a panoramic view around the vehicle.
[0075] Step 303: Acquire driving data corresponding to the plurality of fisheye images, and obtain a motion trajectory of the vehicle in a world coordinate system according to the driving data.
[0076] In the embodiment of the present application, driving data is obtained from a vehicle sensor (such as an inertial measurement unit (IMU) or GPS). This data includes the speed, acceleration and direction of the vehicle, as well as information such as a timestamp. The acquired driving data, especially the speed and direction information of the vehicle, is used to calculate the motion trajectory of the vehicle in the world coordinate system. The kinematic and dynamic models are used to predict the position and posture of the vehicle in the future time period.
[0077] Step 304 : obtaining the target bird's-eye view BEV image according to the multiple surround bird's-eye view BEV images and the motion trajectory.
[0078] In the embodiment of the present application, a plurality of surround bird's-eye view BEV images and the calculated motion trajectory information are integrated to generate a target bird's-eye view BEV image. Specifically, the plurality of surround bird's-eye view BEV images are stitched into a larger image to completely cover the environment around the vehicle. According to the actual motion trajectory of the vehicle, the viewing angle and perspective effect of each bird's-eye view image are adjusted to ensure that the generated target bird's-eye view BEV image is consistent with the actual scene.
[0079] Step 305 , obtaining a recognition result of a storage location in the target bird's-eye view BEV image according to the target bird's-eye view BEV image and a preset neural network model.
[0080] The implementation process of step 305 is consistent with the above-mentioned step 203, and this application will not repeat it here.
[0081] By using multiple surround bird's-eye view BEV images, the embodiment of the present application can more comprehensively cover the environment around the vehicle, thereby providing more accurate and comprehensive environmental perception. Combined with the actual motion trajectory of the vehicle, the generated target bird's-eye view BEV image can be dynamically adjusted and corrected according to the position and direction of the vehicle. This can reduce the recognition error caused by changes in viewing angle or perspective effects, thereby improving the accuracy of storage location recognition.
[0082] In the above Figure 3 In the case shown, the embodiment of the present application also provides a schematic diagram of the implementation process of the storage location identification method. Figure 4 As shown, the method may include the following steps 401 to 406:
[0083] Step 401, when the speed of the vehicle is lower than a preset speed threshold, a plurality of fisheye images are acquired through the vehicle-mounted camera.
[0084] The implementation process of step 401 is consistent with the above-mentioned step 301, and this application will not repeat it here.
[0085] Step 402: obtaining a plurality of surround bird's-eye view BEV images according to the plurality of fisheye images.
[0086] The implementation process of step 402 is consistent with the above-mentioned step 302, and this application will not repeat it here.
[0087] Step 403: Acquire driving data corresponding to the plurality of fisheye images, and obtain a motion trajectory of the vehicle in a world coordinate system according to the driving data.
[0088] The implementation process of step 403 is consistent with the above-mentioned step 303, and this application will not repeat it here.
[0089] Step 404 , fusing and superimposing the plurality of historical surround bird's-eye view BEV images according to the motion trajectory to obtain an image of the vehicle chassis area of the current surround bird's-eye view BEV image.
[0090] In an embodiment of the present application, multiple surround bird's-eye view (BEV) images in historical records are obtained. They may be images of past scenes that are pre-recorded or stored in the system. The motion trajectory information of the current vehicle is obtained. This can be obtained by an on-board navigation system, an inertial measurement unit (IMU), or other position tracking sensors to determine the current position and direction of movement of the vehicle. The acquired historical surround bird's-eye view (BEV) images are geometrically calibrated and time synchronized. Ensure that these images are consistent with the environment corresponding to the current vehicle position in space and time. Align the motion trajectory of the current vehicle with the historical surround bird's-eye view (BEV) images. Map the current position and posture information of the vehicle to the position in the historical image to ensure the correct viewing angle and coverage. Use image processing algorithms, such as image fusion technology or multi-view image synthesis methods, to fuse multiple historical surround bird's-eye view (BEV) images into the image of the current vehicle position. This can be achieved by pixel-level fusion or mixing to ensure complete coverage of the vehicle chassis area and high-quality image synthesis. In the fused image, the vehicle chassis area of the current surround bird's-eye view BEV image is extracted and displayed. This can be achieved through image segmentation or region labeling, ensuring visual attention and display of the vehicle chassis area.
[0091] Step 405 , fill the image of the vehicle chassis area into the vehicle chassis area of the current surround bird's-eye view BEV image to obtain the target bird's-eye view BEV image.
[0092] In the embodiment of the present application, the vehicle chassis area is calibrated and extracted on the historical surround bird's-eye view BEV image according to the current surround bird's-eye view BEV image and the vehicle motion trajectory. This is achieved by image processing technology based on motion estimation. The vehicle chassis area in the current surround bird's-eye view BEV image is filled into the corresponding position in the fused historical surround bird's-eye view BEV image. It is ensured that the filled image maintains consistency and accuracy with the historical data to generate the final target bird's-eye view BEV image.
[0093] Step 406 , obtaining a recognition result of the storage location in the target bird's-eye view BEV image according to the target bird's-eye view BEV image and a preset neural network model.
[0094] The implementation process of step 406 is consistent with that of the above-mentioned step 305, and will not be repeated herein.
[0095] The embodiment of the present application can effectively enhance the information richness and clarity of the image by fusing and superimposing multiple historical surround bird's-eye view BEV images with the current surround bird's-eye view BEV image. After obtaining the vehicle chassis area image of the current surround bird's-eye view BEV image, it is synthesized with the historical surround bird's-eye view BEV image using a filling technique to ensure the integrity and accuracy of the chassis area in the target bird's-eye view BEV image.
[0096] In the above Figure 3 In the case shown, the embodiment of the present application also provides a schematic diagram of the implementation process of the storage location identification method. Figure 5 As shown, the method may include the following steps 501 to 505:
[0097] Step 501, when the speed of the vehicle is lower than a preset speed threshold, a plurality of fisheye images are acquired through the vehicle-mounted camera.
[0098] The implementation process of step 501 is consistent with the above-mentioned step 301, and this application will not repeat it here.
[0099] Step 502: obtaining a plurality of surround-view bird's-eye view BEV images according to the plurality of fisheye images.
[0100] The implementation process of step 502 is consistent with the above-mentioned step 302, and this application will not repeat it here.
[0101] Step 503: Acquire driving data corresponding to the plurality of fisheye images, and obtain a motion trajectory of the vehicle in a world coordinate system according to the driving data.
[0102] The implementation process of step 503 is consistent with the above-mentioned step 303, and this application will not repeat it here.
[0103] Step 504 : According to the motion trajectory, the historical bird's-eye view BEV image is superimposed and fused with the current bird's-eye view BEV image to obtain the target bird's-eye view BEV image.
[0104] In an embodiment of the present application, for each historical bird's-eye view BEV image, its position and orientation transformation parameters in the world coordinate system are calculated according to the time point or condition of its recording. These parameters can be obtained by visual odometer or other positioning technology. Using the calculated transformation parameters, the perspective and perspective of each historical bird's-eye view BEV image are converted to the perspective and perspective of the current bird's-eye view BEV image. Including transformations such as rotation, translation and scaling to ensure that the historical image can be accurately integrated into the current environment. The converted historical bird's-eye view BEV image is superimposed on the appropriate position in the current bird's-eye view BEV image. Ensure that the non-vehicle chassis area (i.e., the surrounding environment) of the historical image is reasonably overlapped and integrated with the current environment. According to the chassis area information in the current bird's-eye view BEV image of the vehicle, the chassis area of the synthesized target bird's-eye view BEV image is filled and integrated. Ensure that the chassis area accurately reflects the position and shape of the vehicle.
[0105] Step 505 , obtaining a recognition result of a storage location in the target bird's-eye view BEV image according to the target bird's-eye view BEV image and a preset neural network model.
[0106] The implementation process of step 505 is consistent with the above-mentioned step 305, and this application will not repeat it here.
[0107] The embodiment of the present application can more accurately determine the position and shape of the vehicle chassis area by superimposing and fusing the historical surround bird's-eye view BEV image with the current surround bird's-eye view BEV image. Dynamically fusing historical images with current data processing can effectively improve the accuracy and stability of the storage location recognition system, especially in complex actual scenes. This method can better adapt to different visual environments and motion states, thereby optimizing the generation of the target bird's-eye view BEV image and the results of storage location recognition.
[0108] Furthermore, the historical surround bird's-eye view BEV image is superimposed and fused into the current surround bird's-eye view BEV image according to the motion trajectory to obtain a target bird's-eye view BEV image, including: obtaining a current visible area and a current invisible area in the current surround bird's-eye view BEV image, and a historical visible area and a historical invisible area in each of the historical surround bird's-eye view BEV images, wherein the invisible area corresponds to the position of the vehicle chassis in the corresponding target bird's-eye view BEV image.
[0109] Specifically, the visible area and invisible area of the current surround bird's-eye view BEV image are obtained, and the current surround bird's-eye view BEV image is obtained using multiple cameras or sensors installed around the vehicle. These images provide visual information around the vehicle chassis at the current moment, including visible areas and invisible areas. Through image processing algorithms, the vehicle chassis area that can be directly observed in the current surround bird's-eye view BEV image is identified and extracted. By using technologies such as obstacle detection and occluded object segmentation, the area where the vehicle chassis is not directly observed in the current environment is inferred. The system saves surround bird's-eye view BEV images at multiple historical moments, each of which records the visual information around the previous vehicle chassis. For each historical surround bird's-eye view BEV image, the same image processing algorithm is used to extract the vehicle chassis area that can be directly observed at the historical moment. Similarly, by analyzing the area that is not directly observed in the historical surround bird's-eye view BEV image, the invisible area of the vehicle chassis at the historical moment is inferred. According to the current dynamic motion trajectory of the vehicle, the historical visible area in the historical surround bird's-eye view BEV image is superimposed with the current surround bird's-eye view BEV image. Through the preset overlay rules, the vehicle chassis area that is not visible in the current environment is processed to ensure that the comprehensive image contains accurate chassis information.
[0110] Furthermore, according to the motion trajectory and the preset superposition rules, the historical bird's-eye view BEV image is sequentially superimposed and fused with the current bird's-eye view BEV image to obtain the target bird's-eye view BEV image. The preset superposition rules include: if the current invisible area overlaps with the historical visible area, the historical visible area is retained; and if the current visible area overlaps with the historical visible area, the current visible area is retained.
[0111] Specifically, check whether the visible area of the historical image overlaps with the invisible area of the current surround bird's-eye view BEV image, and whether the invisible area of the historical image overlaps with the visible area of the current surround bird's-eye view BEV image. If the visible area of the historical image overlaps with the invisible area of the current surround bird's-eye view BEV image, the visible area of the historical image is retained. If the invisible area of the historical image overlaps with the visible area of the current surround bird's-eye view BEV image, the visible area of the current surround bird's-eye view BEV image is maintained. After the superposition is completed, the historical surround bird's-eye view BEV image sequence that has been superimposed and fused, and the processing results of the current surround bird's-eye view BEV image are synthesized into the target bird's-eye view BEV image.
[0112] The embodiment of the present application combines current data and historical data, retains important visual information through preset superposition rules, and also effectively handles invisible areas. This balance ensures that the generated target bird's-eye view BEV image is both real-time and practical, and can serve as an important input for the driving assistance system.
[0113] Furthermore, obtaining a plurality of surround bird's-eye view BEV images based on the plurality of fisheye images includes: acquiring calibration parameters of the vehicle-mounted camera, and obtaining a plurality of fisheye images in a world coordinate system based on the calibration parameters.
[0114] Specifically, the camera is calibrated to determine the focal length, principal point position, distortion parameters, and the position and orientation of the camera relative to the vehicle coordinate system. Based on the calibration parameters, the original fisheye image captured by the camera is converted to the vehicle's world coordinate system. This includes using the camera's distortion correction model to map the pixels in the image to positions in the actual world coordinate system.
[0115] Furthermore, the multiple fisheye images in the world coordinate system are spatially aligned to obtain multiple aligned fisheye images, wherein the multiple fisheye images include overlapping image areas and non-overlapping image areas.
[0116] Specifically, if there are multiple cameras or images captured by the same camera at different times, it is necessary to ensure that they are spatially aligned, that is, projected into the same coordinate system. This can be achieved by transforming each image into a common reference frame or base frame. For each fisheye image, there are boundary areas or areas that are not covered by other camera images. These areas are filled with pixels to ensure that the entire surround view image has complete coverage.
[0117] Furthermore, pixel filling is performed on non-overlapping image areas of the multiple fisheye images, and weighted filling is performed on overlapping image areas of the multiple fisheye images to obtain multiple surround bird's-eye view BEV images.
[0118] Specifically, between different fisheye images, the overlapping area may have different perspectives or distortion problems. Through the weighted filling method, the information of the overlapping area can be integrated to ensure the continuity and smoothness of the image.
[0119] The embodiment of the present application can effectively eliminate distortion and align images by accurately calibrating and spatially aligning the fisheye images of the vehicle-mounted camera, thereby obtaining a high-quality bird's-eye view BEV image. By spatially aligning and fusing multiple fisheye images, the best part of each image is utilized, and the display of the chassis area is further optimized through weighted filling, thereby improving the recognition system's understanding and accuracy of the storage location or environment.
[0120] The present application embodiment provides a specific implementation of a storage location identification method, which is applied to the above Figure 2 , Figure 3 , Figure 4 as well as Figure 5 The schematic diagram of the implementation process of a storage location identification method shown in FIG. 1 includes:
[0121] like Figure 6 As shown, Figure 6 A schematic diagram of generating a surround-view BEV image is provided for an embodiment of the present application, wherein the FOV (Field of View) of the on-board fisheye image is generally around 200°, and there is typical fisheye distortion. Through the camera internal and external parameter calibration technology, the distortion can be calibrated, and the camera imaging perspective is converted to the BEV (Bird's Eye View) perspective to construct a new imaging plane, which is the ideal ground in the world coordinate system. On the calibrated BEV image, the deformation of the storage location becomes smaller. The BEV image generated by four fisheye images can be spliced into a 360° surround-view BEV image, so that the layout of the storage locations around the vehicle body can be observed without blind spots, thereby reducing the difficulty of detecting the storage locations.
[0122] like Figure 7 As shown, Figure 7 Schematic diagram of generating a target BEV image provided by an embodiment of the present application. Although the surround view BEV image can observe the surroundings of the vehicle body without blind spots, the vehicle body area cannot be observed due to occlusion. The following will appear on all surround view BEV images: Figure 6 The black block area shown in the figure will cover part or almost all of the parking space lines when the vehicle is parking. This phenomenon will seriously affect the detection accuracy of the parking space and even cause false alarms. The chassis splicing module can perform historical imaging in-situ filling of the black block area, which can effectively avoid the problem of corner pressure points causing errors in storage space detection.
[0123] On the basis of completing the image filling of the chassis area, the key point-based storage location detection algorithm can ensure that the key points of the storage location will not be blocked by the vehicle body at any time, which can effectively improve the detection accuracy.
[0124] like Figure 8 As shown, Figure 8 A workflow diagram of a storage location identification method provided in an embodiment of the present application includes: a pre-processing stage, the pre-processing includes obtaining fisheye image information and DR information from a message end, obtaining camera calibration information from a specified file, and then generating a single-view BEV map after time synchronization processing, generating a surround-view BEV map after field of view stitching, obtaining a target BEV map after chassis filling, and then converting it into input data matching a downstream perception module;
[0125] In the model inference stage, the output data of the pre-processing stage is sent to the perception model after pruning and quantization. The perception model completes the detection of the target BEV image and outputs the parking space detection result to the downstream fusion decision module.
[0126] When the vehicle enters the parking state or is within the limited low speed range, the parking reminder function is automatically turned on. The pre-processing stage completes the production of data target data. The perception model in the model reasoning stage identifies the parking space in the scene. The perception algorithm obtains the parking space detection result of each frame image and inputs it into the subsequent processing module in the form of category, location, and confidence for fusion calculation. The specific steps are as follows:
[0127] Step 1: Data parsing. This step has two main purposes: obtaining the input data required by the system and synchronizing timestamps. The data types mainly include four-way fisheye image data, DR wheel pulse data, and camera calibration data. The camera calibration data is static data and does not change with time, while the other two types of data are dynamic data and will change with time during the movement of the vehicle. In order to reduce the negative impact of this change, it is necessary to synchronize the above dynamic information in the time dimension. The main steps are as follows:
[0128] 1. Determine the timestamps that need to be synchronized, and use the timestamps (denoted as ts) of the four fisheye images currently acquired as the synchronization reference.
[0129] 2. Generally, from an engineering perspective, the DR information closest to the ts time interval is used to approximate the synchronous DR, or the bilinear interpolation method is used to determine the DR value at the target timestamp. The accuracy of the two algorithms is affected differently by vehicle speed. The higher the vehicle speed, the greater the error of the approximate algorithm. At this time, it is more suitable to use the interpolation algorithm. At low speeds, because the DR changes very little, both algorithms can be used.
[0130] Step 2: Format conversion, converting the original fisheye image data format into a data format that meets the input requirements of the inference model. The perception model training phase used in the present invention uses BRG format data, so the original image NV12 format needs to be converted into BGR format.
[0131] Step 3: Single-view BEV image generation. This step is used to convert the pixel coordinates of the fisheye image to the world coordinate system and convert the imaging perspective to the bird's-eye view perspective. The specific steps are as follows:
[0132] (1) Conversion formula between world coordinates and pixel coordinates
[0133]
[0134] Among them, (Xw, Yw, Zw) are the physical coordinates of a point in the world coordinate system, and (Xc, Yc, Zc) are the position of the point in the camera coordinate system.
[0135] (2) Conversion from camera coordinate system to image physical coordinate system
[0136]
[0137] Among them, (u, v) is the pixel coordinate of the point in the pixel coordinate system, and Z is the scale factor.
[0138]
[0139] The matrix above is the camera's intrinsic parameter matrix, which depends on the camera's internal parameters. Among them, f is the focal length, dX and dY are the physical lengths of a pixel on the camera's photosensitive plate in the X and Y directions (i.e., how many millimeters a pixel is on the photosensitive plate), and u0 and v0 are the coordinates of the camera's photosensitive plate center in the pixel coordinate system.
[0140] This application will matrix It is called the camera's extrinsic matrix, which depends on the relative position of the camera coordinate system and the world coordinate system. R represents the rotation matrix and T represents the translation vector. That is, without considering distortion, the camera's imaging model is as follows:
[0141]
[0142] (3) Introducing distortion correction
[0143] The incident angle θ of the light is calculated based on (Xc, Yc, Zc), and then the phase height corresponding to this incident angle is found in the normalized phase height table based on the incident angle θ. Then, based on the proportional relationship between the true phase height and the current phase height, the coordinates (u, v) corresponding to (Xc, Yc, Zc) in the pixel coordinate system are calculated.
[0144] In addition, considering that image distortion may lead to recognition failure, BEV is also an ideal detection output. Because the BEV projection plane uses the ground as a reference, the distortion of the plane target on the ground can be basically eliminated, and the above-mentioned ground target layout fully meets this assumption. Therefore, it can effectively suppress the recognition inaccuracy caused by lens distortion and significantly improve the recognition accuracy of such targets.
[0145] Step 4: Surround view BEV image generation. This step mainly solves the problem of overlapping fields of view between four single-view BEVs. The four single-view BEV images are fused into a surround view BEV to solve the problem of field of view coupling. In order to reduce the computing power requirements, this solution adopts a weighted fusion strategy. The specific operations are as follows.
[0146] (1) Fig. 9 As shown, Fig. 9 A schematic diagram of generating a surround view BEV image is provided in an embodiment of the present application. Four single-view BEVs are spatially aligned in a world coordinate system. After the alignment, each field of view will have overlapping areas and non-overlapping areas.
[0147] (2) Pixel weighting: In non-overlapping areas, single field pixels are directly used to fill, and in overlapping areas, weighted filling is used. Assuming there are two fields of view θ1 and θ2, the overlapping area is denoted as ROI, and the coordinates of the overlapping area are denoted as (x i ,y j ), where i and j represent the accurate position information in world coordinates. The complete fusion formula is as follows:
[0148]
[0149] Among them, P1 ij 、P2 ij That is, the weighted value of the two fields of view in the current ij coordinate system. After the above processing, the surround view BEV image result can be obtained.
[0150] Step 5: Target BEV image generation. This module uses historical surround BEV data and DR information to calculate the chassis area ( Figure 1 The image in the black block area of the surround view BEV) is filled into the current surround view BEV image to make the image truly 360° unobstructed, that is, the target BEV image. The specific operations are as follows.
[0151] (1) Carpose (ego vehicle coordinates) superposition: The relationship between the world coordinate system, the ego vehicle coordinate system, and the image coordinate system is established through the camera calibration results. The movement changes of the ego vehicle can be calculated through DR, so that the positions of the three coordinate systems can be continuously updated, such as Fig.10 As shown, Fig.10 A coordinate conversion diagram of a storage location identification method provided in an embodiment of the present application is provided, whereby all carposes can have an independent coordinate in the world coordinate system, and the motion trajectory of the vehicle in the world coordinate system can be formed by superimposing historical carpose information.
[0152] (2) Single-view BEV superposition, the core technology of the target BEV image is superposition fusion, such as Fig.11 As shown, Fig.11 A schematic diagram of generating a target BEV image provided in an embodiment of the present application is provided, and the current surround BEV image, historical BEV image and corresponding carpose information are obtained. The blue area in the figure is the visible area, and the black area is the invisible area, that is, the chassis position. After the information is superimposed based on carpose, the unnecessary information is eliminated using a fusion strategy.
[0153] (3) Target fusion, such as Fig.11As shown in the rightmost overlay image in , in order to simplify the description, only two images are selected here to be superimposed, focusing on the black block area. The overlapping area of the two black blocks indicates that neither image provides useful information, so it is still represented by black. It can be seen that the black overlapping area becomes smaller. The next step is to determine the calculation rule for the overlapping part of the black block area and the blue area. There are two possibilities here: the first is that both frames overlap the blue area, then the blue area result of the current frame is retained, and the second is that the black area overlaps the blue area, then regardless of whether the blue area is the current frame, the blue area is retained. In this way, a fused BEV image with a smaller black area can be obtained. As the number of historical frames increases, similarly, the black area will continue to decrease until the target BEV image is generated.
[0154] Step 6: Storage location perception module. Based on the target BEV image, the present invention designs a storage location key point detection network, which can quickly extract storage location information, such as storage location category, location and confidence information.
[0155] The algorithm VPLD-Yolox used in this application is improved based on YOLOX, and draws on YOLOX's anchor free idea and decoupled detection head to build the VPLD model. Darknet53 based on YOLOX is used as the backbone of the network, and PAFPN is used for effective feature extraction. The effective feature layer obtained by 32 times downsampling is input into the decoupled detection head for prediction output.
[0156] YOLOX's PAFPN outputs three effective feature layers, namely P3, P4, and P5, corresponding to 8x downsampling, 16x downsampling, and 32x downsampling. VPLD-YOLOX samples a single feature map design and directly applies the 32x downsampled feature map to the detection head for prediction. For example, for a 320x320 input image, the feature map output after 32x downsampling is Fig.10 x10, a maximum of 100 parking spaces can be predicted for one picture. In actual scenes, there will not be so many parking spaces in each frame. Therefore, 32x downsampling basically meets the requirements, and the output of a single feature map has almost no effect on the accuracy after testing.
[0157] VPLD-YOLOX finally predicts and outputs 8 channels: confidence, Ax, Ay, Bx, By, Dx, Dy, where confidence represents the confidence of the detected real warehouse location; Ax, Ay are the coordinates of marking_point relative to feature_map, with a value range of (0, 1); Bx, By are the offsets of entrance_line relative to marking_point in the x, y direction, with a value between [-1, 1]. Dx, Dy are the offsets of separate_line relative to marking_point in the x, y direction, with a value between [-1, 1].
[0158] This application implements data alignment, single-view BEV, surround-view BEV, target BEV and storage location perception, which can effectively identify various storage location targets. It can provide a 360-degree image without blind spots from the BEV perspective, effectively solve the problem of accuracy loss caused by occlusion scenes, and improve the parking success rate.
[0159] It should be understood that, although the steps in the above-mentioned flowcharts are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flowcharts may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0160] Based on the foregoing embodiments, an embodiment of the present application provides a storage location identification device, which includes the modules included and the units included in the modules, which can be implemented by a processor; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.
[0161] Fig.12 A schematic diagram of the structure of a storage location identification device provided in an embodiment of the present application is shown in FIG. Fig.12 As shown, the device 1200 includes an acquisition module 1201 and a generation module 1202, wherein:
[0162] The acquisition module 1201 is used to acquire multiple fisheye images through the vehicle camera when the speed of the vehicle is lower than a preset speed threshold;
[0163] The acquisition module 1201 is further used to obtain a target bird's-eye view BEV image according to the multiple fisheye images, wherein a target area for representing the chassis of the vehicle in the target bird's-eye view BEV image is smaller than an actual area of the chassis of the vehicle;
[0164] The generating module 1202 is used to obtain the recognition result of the storage location in the target bird's-eye view BEV image according to the target bird's-eye view BEV image and a preset neural network model.
[0165] In some embodiments, the acquisition module 1201 is further used to obtain a plurality of surround bird's-eye view BEV images according to the plurality of fisheye images;
[0166] The acquisition module 1201 is further used to acquire the driving data corresponding to the plurality of fisheye images, and obtain the motion trajectory of the vehicle in the world coordinate system according to the driving data;
[0167] The generating module 1202 is further configured to obtain the target bird's-eye view BEV image according to the multiple surround bird's-eye view BEV images and the motion trajectory.
[0168] In some embodiments, the acquisition module 1201 is further used to perform fusion and superposition processing on the multiple historical surround bird's-eye view BEV images according to the motion trajectory to obtain an image of the vehicle chassis area of the current surround bird's-eye view BEV image;
[0169] The generating module 1202 is further configured to fill the image of the vehicle chassis area into the vehicle chassis area of the current surround bird's-eye view BEV image to obtain the target bird's-eye view BEV image.
[0170] In some embodiments, the acquisition module 1201 is further used to superimpose and fuse the historical surround bird's-eye view BEV image into the current surround bird's-eye view BEV image according to the motion trajectory to obtain the target bird's-eye view BEV image.
[0171] In some embodiments, the acquisition module 1201 is further used to acquire the current visible area and the current invisible area in the current surround bird's-eye view BEV image, and the historical visible area and the historical invisible area in each of the historical surround bird's-eye view BEV images, wherein the invisible area corresponds to the position of the vehicle chassis in the corresponding bird's-eye view BEV image;
[0172] The generation module 1202 is further used to sequentially superimpose and fuse the historical surround bird's-eye view BEV image with the current surround bird's-eye view BEV image according to the motion trajectory and a preset superposition rule to obtain a target bird's-eye view BEV image.
[0173] In some embodiments, the acquisition module 1201 is further used to acquire calibration parameters of the vehicle-mounted camera, and obtain multiple fisheye images in the world coordinate system according to the calibration parameters;
[0174] The acquisition module 1201 is further used to spatially align the multiple fisheye images in the world coordinate system to obtain multiple aligned fisheye images, wherein the multiple fisheye images include overlapping image areas and non-overlapping image areas;
[0175] The acquisition module 1201 is further used to perform pixel filling on non-overlapping image areas of the multiple fisheye images and perform weighted filling on overlapping image areas of the multiple fisheye images to obtain multiple surround bird's-eye view BEV images.
[0176] In some embodiments, the acquisition module 1201 is further used to perform multi-layer feature sampling on the target bird's-eye view BEV image to obtain a valid feature map;
[0177] The generation module 1202 is also used to perform target detection on the effective feature map to obtain the recognition result of the storage location in the target bird's-eye view BEV image, and the recognition result includes the position of the storage location, the ground mark position of the storage location and the confidence of each position.
[0178] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.
[0179] It should be noted that in the embodiments of this application Fig.12 The division of modules in the storage location identification device shown is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. It may also be implemented in the form of a combination of software and hardware.
[0180] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.
[0181] The embodiment of the present application provides a computer device, which may be a server, and its internal structure diagram may be as follows: Fig.13 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.
[0182] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the method provided in the above embodiment are implemented.
[0183] An embodiment of the present application provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps of the method provided in the above method embodiment.
[0184] Those skilled in the art will understand that Fig.13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0185] In one embodiment, a storage location identification device provided by the present application can be implemented in the form of a computer program. The computer program can be Fig.13The computer device shown in the figure is run. The memory of the computer device can store various program modules constituting the above-mentioned device. The computer program composed of various program modules enables the processor to execute the steps in the method of each embodiment of the present application described in this specification.
[0186] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium, storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.
[0187] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above description of each embodiment tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. For the sake of brevity, this article will not repeat them.
[0188] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist at the same time, and object B exists alone.
[0189] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0190] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.
[0191] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.
[0192] In addition, all functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may be a separate unit, or two or more modules may be integrated into one unit; the above-mentioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0193] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.
[0194] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.
[0195] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.
[0196] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.
[0197] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.
[0198] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for identifying a storage location. It is characterized in that The method comprises: When the speed of the vehicle is lower than a preset speed threshold, a plurality of fisheye images are acquired through the vehicle-mounted camera; Obtaining a target bird's-eye view BEV image according to the plurality of fisheye images, wherein a target area for representing a chassis of the vehicle in the target bird's-eye view BEV image is smaller than an actual area of the chassis of the vehicle; According to the target bird's-eye view BEV image and a preset neural network model, an identification result of the storage location in the target bird's-eye view BEV image is obtained. The preset neural network model is obtained by training an initial neural network model based on sample data collected in a storage location identification scenario. The sample data includes multiple historical images collected in the storage location identification scenario and multiple historical storage location information in the historical storage location identification scenario.
2. The method according to claim 1, It is characterized in that The step of obtaining a bird's-eye view BEV image of the target according to the plurality of fisheye images comprises: According to the plurality of fisheye images, a plurality of surround-view bird's-eye view BEV images are obtained; Acquire driving data corresponding to the plurality of fisheye images, and obtain a motion trajectory of the vehicle in a world coordinate system according to the driving data; The target bird's-eye view BEV image is obtained according to the multiple surround bird's-eye view BEV images and the motion trajectory.
3. The method according to claim 2, It is characterized in that The multiple surround bird's-eye view BEV images are images that do not include the vehicle chassis area, and the multiple surround bird's-eye view BEV images include multiple historical surround bird's-eye view BEV images and a current surround bird's-eye view BEV image. The target bird's-eye view BEV image is obtained according to the multiple surround bird's-eye view BEV images and the motion trajectory, including: Performing fusion and superposition processing on the multiple historical surround bird's-eye view BEV images according to the motion trajectory to obtain an image of the vehicle chassis area of the current surround bird's-eye view BEV image; The image of the vehicle chassis area is filled into the vehicle chassis area of the current surround bird's-eye view BEV image to obtain the target bird's-eye view BEV image.
4. The method according to claim 2, It is characterized in that The multiple surround bird's-eye view BEV images are images that do not include the vehicle chassis area, and the multiple surround bird's-eye view BEV images include historical surround bird's-eye view BEV images and current surround bird's-eye view BEV images. The target bird's-eye view BEV image is obtained according to the multiple surround bird's-eye view BEV images and the motion trajectory, and further includes: The historical surround bird's-eye view BEV image is superimposed and fused onto the current surround bird's-eye view BEV image according to the motion trajectory to obtain the target bird's-eye view BEV image.
5. The method according to claim 4, It is characterized in that The historical bird's-eye view BEV image is superimposed and fused with the current bird's-eye view BEV image according to the motion trajectory to obtain a target bird's-eye view BEV image, including: Acquire a current visible area and a current invisible area in the current surround bird's-eye view BEV image, and a historical visible area and a historical invisible area in each of the historical surround bird's-eye view BEV images, wherein the invisible area corresponds to the position of the vehicle chassis in the corresponding surround bird's-eye view BEV image; According to the motion trajectory and the preset superposition rule, the historical surround bird's-eye view BEV image is sequentially superimposed and fused with the current surround bird's-eye view BEV image to obtain a target bird's-eye view BEV image; The preset superposition rules include: In the case where the current invisible area overlaps with the historical visible area, the historical visible area is retained; and in the case where the current visible area overlaps with the historical visible area, the current visible area is retained.
6. The method according to any one of claims 3 to 5, It is characterized in that The method of obtaining a plurality of surround bird's-eye view BEV images according to the plurality of fisheye images includes: Acquire calibration parameters of the vehicle-mounted camera, and obtain multiple fisheye images in a world coordinate system according to the calibration parameters; Performing spatial alignment on the multiple fisheye images in the world coordinate system to obtain multiple aligned fisheye images, wherein the multiple fisheye images include overlapping image areas and non-overlapping image areas; Pixel filling is performed on non-overlapping image areas of the multiple fisheye images, and weighted filling is performed on overlapping image areas of the multiple fisheye images to obtain multiple surround bird's-eye view BEV images.
7. The method according to claim 1, It is characterized in that The step of obtaining the recognition result of the storage location in the target bird's-eye view BEV image according to the target bird's-eye view BEV image and the preset neural network model includes: Performing multi-layer feature sampling on the target bird's-eye view BEV image to obtain an effective feature map; The effective feature map is subjected to target detection to obtain a recognition result of the storage location in the target bird's-eye view BEV image, wherein the recognition result includes the position of the storage location, the ground mark position of the storage location, and the confidence level of each position.
8. A storage location identification device, It is characterized in that include: An acquisition module, used for acquiring a plurality of fisheye images through a vehicle-mounted camera when the speed of the vehicle is lower than a preset speed threshold; The acquisition module is further used to obtain a target bird's-eye view BEV image according to the multiple fisheye images, wherein a target area for representing the chassis of the vehicle in the target bird's-eye view BEV image is smaller than an actual area of the chassis of the vehicle; A generation module is used to obtain the recognition result of the storage location in the target bird's-eye view BEV image based on the target bird's-eye view BEV image and a preset neural network model, wherein the preset neural network model is obtained by training an initial neural network model based on sample data collected in a storage location recognition scenario, and the sample data includes multiple historical images collected in the storage location recognition scenario and multiple historical storage location information in the historical storage location recognition scenario.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, It is characterized in that When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Cited By
Method, system, device and medium for adaptively generating vehicle look-around image
CN121708122A
