Vehicle panoramic image generation method and system based on perceptual fusion

Abstract

The invention provides a vehicle panoramic image generation method and system based on perceptual fusion, and relates to the technical field of automobiles, and the method comprises the steps: obtaining image data around a vehicle, and recognizing an obstacle in an image; constructing a panoramic image splicing model, and placing obstacle blocks in the panoramic image splicing model according to obstacles in the image; obtaining vehicle body radar data, and adjusting the shape and position of an obstacle block in the panoramic image splicing model based on the vehicle body radar data; and based on the adjusted panoramic image splicing model, splicing the acquired image data around the vehicle to obtain a spliced panoramic image. According to the invention, distortion of obstacles in the panoramic image can be avoided.

Description

Technical Field

[0001] The present disclosure relates to the field of automobile technology, and in particular to a method and system for generating a vehicle panoramic image based on perception fusion. Background Art

[0002] Panoramic imaging can display images around the vehicle in real time, eliminating blind spots, and has always been one of the most commonly used hot functions in the car. Panoramic imaging usually uses a fixed model stitching solution, but when the vehicle is close to the surrounding obstacles, the obstacles in the stitched panoramic image will be distorted. Summary of the invention

[0003] The embodiment of the present disclosure provides a method and system for generating a panoramic image of a vehicle based on perception fusion, which can avoid the distortion of obstacles in the panoramic image. The technical solution is as follows:

[0004] In a first aspect, a method for generating a vehicle panoramic image based on perception fusion is provided, comprising:

[0005] Obtain image data around the vehicle and identify obstacles in the image;

[0006] Construct a panoramic image stitching model, and place obstacle blocks in the panoramic image stitching model according to obstacles in the image;

[0007] Obtain vehicle body radar data, and adjust the shape and position of obstacle blocks in the panoramic image stitching model based on the vehicle body radar data;

[0008] Based on the adjusted panoramic image stitching model, the acquired image data around the vehicle are stitched to obtain a stitched panoramic image.

[0009] In a second aspect, a vehicle panoramic image generation system based on perception fusion is provided, comprising:

[0010] A preliminary recognition module is configured to obtain image data around the vehicle and identify obstacles in the image;

[0011] The model building module is configured to build a panoramic image stitching model and place obstacle blocks in the panoramic image stitching model according to obstacles in the image;

[0012] An adjustment module is configured to obtain vehicle body radar data and adjust the shape and position of the obstacle block in the panoramic image stitching model based on the vehicle body radar data;

[0013] The image generation module is configured to stitch the acquired image data around the vehicle based on the adjusted panoramic image stitching model to obtain a stitched panoramic image.

[0014] In a third aspect, a vehicle panoramic image generation system based on perception fusion is provided, comprising:

[0015] The vehicle-mounted camera module is arranged around the vehicle and is used to obtain image data around the vehicle;

[0016] A radar module is provided on the vehicle and is used to obtain vehicle body radar data;

[0017] The on-board central main controller is arranged on the vehicle and is used to identify obstacles in the image based on the image data around the vehicle; construct a panoramic image stitching model, and place obstacle blocks in the panoramic image stitching model according to the obstacles in the image; adjust the shape and position of the obstacle blocks in the panoramic image stitching model based on the vehicle body radar data; and stitch the acquired image data around the vehicle based on the adjusted panoramic image stitching model to obtain a stitched panoramic image.

[0018] In a fourth aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to complete the steps of the above-mentioned method for generating a vehicle panoramic image based on perception fusion.

[0019] In a fifth aspect, a computer-readable storage medium is provided for storing computer instructions. When the computer instructions are executed by a processor, the steps of the above-mentioned method for generating a vehicle panoramic image based on perception fusion are completed.

[0020] In a sixth aspect, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned method for generating a vehicle panoramic image based on perception fusion.

[0021] The technical solution provided by the embodiments of the present disclosure has the following beneficial effects:

[0022] In the disclosed embodiment, the collected images and radar information are fully utilized. First, the position of the obstacle block is preliminarily determined based on the image information. Then, the position and shape of the obstacle block are corrected based on the radar data to obtain the precise shape of the obstacle block. Finally, image stitching is performed based on the obstacle block to obtain a panoramic image. When the obstacle is close to the vehicle, the obstacle can still be correctly displayed without distortion or deformation, and clear and easy-to-identify obstacle information around the vehicle is displayed to the driver, which is beneficial for the driver to identify the information of the obstacles around the vehicle and improves the driver's driving and parking safety.

[0023] Advantages of additional aspects of the present disclosure will be given in part in the following description, and in part will become apparent from the following description, or will be learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 is a flow chart of a vehicle panoramic image generation method based on perception fusion provided by an embodiment of the present disclosure;

[0026] Figure 2 is a structural block diagram of a vehicle panoramic image generation system based on perception fusion provided by an embodiment of the present disclosure;

[0027] Figure 3 is a structural block diagram of another vehicle panoramic image generation system based on perception fusion provided by an embodiment of the present disclosure;

[0028] Figure 4 It is a structural block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0030] Figure 1 This is a flow chart of a vehicle panoramic image generation method based on perception fusion provided by an embodiment of the present disclosure. The method can be executed by a vehicle-mounted central controller to generate a panoramic image in real time during vehicle operation. Figure 1 , the method comprising:

[0031] Step 101: Acquire image data around the vehicle and identify obstacles in the image;

[0032] Step 102: construct a panoramic image stitching model, and place obstacle blocks in the panoramic image stitching model according to obstacles in the image;

[0033] Step 103: Acquire vehicle body radar data, and adjust the shape and position of the obstacle block in the panoramic image stitching model based on the vehicle body radar data;

[0034] Step 104 : stitching the acquired image data around the vehicle based on the adjusted panoramic image stitching model to obtain a stitched panoramic image.

[0035] In step 101, firstly, images transmitted from the cameras around the vehicle are received, and the obstacles around the vehicle are identified using an algorithm. In the disclosed embodiment, the execution of each step can be implemented in the on-board central main controller, which integrates functions such as image recognition conversion, high-speed storage calculation, and real-time data transmission, and can realize fast real-time data processing. Cameras are installed around the vehicle to obtain stable and clear image data around the vehicle in real time. Obstacles can be identified using a target detection algorithm based on deep learning, preferably a more efficient One-Stage target detection algorithm, such as YOLOv5, SSD, etc.

[0036] In step 102, a panoramic image stitching model is constructed, and corresponding obstacle blocks are placed at corresponding distances in the panoramic image stitching model. At this time, the size and distance of the obstacle blocks placed are somewhat different from the actual ones, and they need to be adjusted later in combination with the radar data. The panoramic image stitching model is a three-dimensional model, and the size is determined based on the range of the panoramic image to be generated. First, the vehicle model is set in the panoramic image stitching model. Taking the generation of a panoramic image from a bird's-eye view as an example, since the range of the panoramic image and the parameters of the vehicle are fixed, and the position of the vehicle in the panoramic image is fixed (located in the center of the panoramic image), the vehicle model with accurate size can be pre-placed in the panoramic image based on the vehicle parameters, and the panoramic image stitching model containing the vehicle model is used as the reference model. The target detection algorithm in step 101 can identify the preliminary shape, size and distance of the obstacle block from the vehicle. According to the obstacle recognition result of step 101, the obstacle block is placed at the corresponding position in the reference model.

[0037] In step 103, the on-board central main controller receives the data sent back by the vehicle body radar, corrects the size and position of the obstacle block according to the detection distance, adjusts the obstacle block to the accurate position in the panoramic image stitching model, and adjusts the obstacle block to the correct size. The vehicle is equipped with a vehicle body radar, which can scan the surrounding of the vehicle body and detect the size of the obstacles around the vehicle body and the distance information from the workshop during this period. The cameras set around the vehicle are fisheye lenses, which have serious barrel distortion. Even through algorithm correction, it is still difficult to accurately restore the shape, size, and position of the obstacle itself. When the obstacle is too close to the lens, its relative size in the image is greatly increased, and the edge of the obstacle is more likely to fall on the imaging edge with a greater degree of lens distortion, resulting in a panoramic image obtained by simply stitching images. The degree of restoration of closer obstacles is poor and there is a problem of distortion and deformation. Compared with the obstacle information obtained based on image recognition, the size and distance information of the obstacle obtained based on the vehicle body radar (usually a laser radar) eliminates the influence of the camera's own distortion. Based on the point cloud data obtained by the lidar, the precise shape, size, and position information of each obstacle within the range of the panoramic image to be generated are extracted, and the obtained information is used to correct the obstacle blocks in the benchmark model.

[0038] In order to meet the requirements of panoramic image generation and display, the panoramic image stitching model should be a bowl shape rather than a plane when there are no obstacle blocks, so the obstacle blocks should also be adjusted according to the panoramic image stitching model. The obstacle blocks identified by the target detection algorithm and the precise information of the obstacle blocks obtained based on the lidar point cloud data are first converted to a rectangular coordinate system through coordinate transformation, and then further transformed based on the shape of the panoramic image stitching model. At this time, the obstacle blocks are adjusted to the correct size if they fit the actual perception of the human eye.

[0039] Considering that the camera is more likely to be affected by obstruction, if there is no corresponding obstacle block in the panoramic image stitching model for the obstacle identified based on the lidar point cloud data, first generate a virtual obstacle in the panoramic image stitching model based on the recognition result of the lidar point cloud data (the shape, size, and location information of the obstacle), and then determine whether there is an obstacle corresponding to the virtual obstacle in the image data recognized by the camera at the next moment. If yes, then generate the obstacle block corresponding to the obstacle normally according to the above steps; if no, then judge the distance between the virtual obstacle and the vehicle, set a risk threshold, such as 1 meter, and if the distance between the virtual obstacle and the vehicle is less than or equal to the risk threshold, in order to ensure driving safety, retain the virtual obstacle generated based on the radar data, use a dark block to represent it when generating the panoramic image, and do not cover the virtual obstacle; if the distance between the virtual obstacle and the vehicle is greater than the risk threshold, then the possible virtual obstacle is unlikely to affect the normal driving of the vehicle and the driver's judgment, and the virtual obstacle is not retained.

[0040] In step 104, the on-board central main controller will scale and crop the images transmitted by the cameras around the vehicle in real time according to the panoramic image stitching model with obstacle blocks placed on it, so that it can be used for image stitching and the image corresponds to the model. Subsequently, the cropped images are overlaid on the surface of the panoramic image stitching model and the obstacle blocks therein in real time, and stitched to obtain a stitched panoramic image. The images collected by the camera are weightedly fused with the radar data to obtain detailed information about the surrounding obstacles when the vehicle is driving in different environmental scenes, and the accurate placement of obstacle blocks and the stitching coverage of images are achieved in real time.

[0041] The stitched panoramic image can be displayed to the driver through the display on the vehicle, and the driver can operate the panoramic image, including but not limited to adjusting the viewing angle and the display pitch angle. The stitched panoramic image combines image and radar data and can correctly display obstacles around the vehicle. Even if the surrounding obstacles are within 2 meters from the vehicle, the panoramic image can be displayed correctly without distortion.

[0042] Figure 2 is a structural block diagram of a vehicle panoramic image generation system 200 based on perception fusion provided by an embodiment of the present disclosure, such as Figure 2 As shown, the system includes: a preliminary recognition module 201, a model building module 202, an adjustment module 203, and an image generation module 204.

[0043] The preliminary recognition module 201 is configured to obtain image data around the vehicle and recognize obstacles in the image.

[0044] The model building module 202 is configured to build a panoramic image stitching model and place obstacle blocks in the panoramic image stitching model according to obstacles in the image.

[0045] The adjustment module 203 is configured to obtain vehicle body radar data, and adjust the shape and position of the obstacle block in the panoramic image stitching model based on the vehicle body radar data.

[0046] The image generation module 204 is configured to stitch the acquired image data around the vehicle based on the adjusted panoramic image stitching model to obtain a stitched panoramic image.

[0047] It should be noted that: the vehicle panoramic image generation system 200 based on perception fusion provided in the above embodiment only uses the division of the above functional modules as an example when generating vehicle panoramic images. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the vehicle panoramic image generation system 200 based on perception fusion provided in the above embodiment and the vehicle panoramic image generation method embodiment based on perception fusion belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0048] Figure 3 is a structural block diagram of another vehicle panoramic image generation system 300 based on perception fusion provided by an embodiment of the present disclosure, such as Figure 3 As shown, the system includes: a vehicle-mounted camera module 301, a radar module 302, and a vehicle-mounted central main controller 303.

[0049] The vehicle-mounted camera module 301 is arranged around the vehicle and is used to obtain image data around the vehicle.

[0050] The radar module 302 is arranged on the vehicle and is used to obtain vehicle body radar data.

[0051] The vehicle-mounted central main controller 303 is set on the vehicle and is used to identify obstacles in the image based on the image data around the vehicle; build a panoramic image stitching model, and place obstacle blocks in the panoramic image stitching model according to the obstacles in the image; adjust the shape and position of the obstacle blocks in the panoramic image stitching model based on the vehicle body radar data; based on the adjusted panoramic image stitching model, stitch the acquired image data around the vehicle to obtain a stitched panoramic image.

[0052] In some embodiments, the system also includes a display, which is arranged in the vehicle cab and can display the stitched panoramic image to the driver. The display can also have interactive methods such as touch or buttons, so that the driver can operate the panoramic image according to actual needs, including but not limited to adjusting the viewing angle, adjusting the display pitch angle, etc.

[0053] The above embodiment provides a vehicle panoramic image generation system 300 based on perception fusion. When generating vehicle panoramic images, the division of the above hardware modules is used as an example. In actual applications, the above hardware modules can be replaced with other hardware modules with similar or identical functions as needed, or the functions of a certain hardware module can be assigned to different hardware modules.

[0054] Figure 4 is a structural block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 4 As shown, the electronic device 400 may be a vehicle-mounted computer, etc. The electronic device 400 includes: a processor 401 and a memory 402 .

[0055] The processor 401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 401 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and LA (Programmable Logic Array). The processor 401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 401 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 401 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0056] The memory 402 may include one or more computer-readable media, which may be non-transitory. The memory 402 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable medium in the memory 402 is used to store at least one computer program, which is used to be executed by the processor 401 to implement a vehicle panoramic image generation method based on perception fusion provided in an embodiment of the present disclosure.

[0057] Those skilled in the art will understand that Figure 4 The structure shown in the figure does not constitute a limitation on the electronic device 400, and may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0058] The embodiment of the present disclosure also provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of a vehicle panoramic image generation method based on perception fusion provided in the embodiment of the present disclosure can be completed.

[0059] The embodiment of the present disclosure also provides a computer program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the steps of a vehicle panoramic image generation method based on perception fusion provided in the embodiment of the present disclosure are implemented.

[0060] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A vehicle panoramic image generation method based on perception fusion, characterized in that: include: Obtain image data around the vehicle and identify obstacles in the image; Construct a panoramic image stitching model, and place obstacle blocks in the panoramic image stitching model according to obstacles in the image; Obtain vehicle body radar data, and adjust the shape and position of obstacle blocks in the panoramic image stitching model based on the vehicle body radar data; Based on the adjusted panoramic image stitching model, the acquired image data around the vehicle are stitched to obtain a stitched panoramic image.

2. The vehicle panoramic image generation method based on perception fusion according to claim 1, characterized in that: The target detection algorithm is used to identify obstacles in the image and obtain the initial shape, size and distance between the obstacle and the vehicle.

3. The vehicle panoramic image generation method based on perception fusion according to claim 1, characterized in that: Based on the vehicle body radar data, the accurate shape, size and distance of the obstacle to the vehicle are obtained, and the obstacle blocks in the panoramic image stitching model are adjusted.

4. The vehicle panoramic image generation method based on perception fusion according to claim 1, characterized in that: If the obstacle identified by the vehicle body radar does not have a corresponding obstacle block in the panoramic image stitching model, a virtual obstacle is generated in the panoramic image stitching model to determine whether there is an obstacle corresponding to the virtual obstacle in the image data at the next moment.

5. The method for generating a vehicle panoramic image based on perception fusion according to claim 4, characterized in that: If there is no obstacle corresponding to the virtual obstacle in the image data at the next moment, the distance between the virtual obstacle and the vehicle is obtained. If the distance between the virtual obstacle and the vehicle is greater than the risk threshold, the virtual obstacle is not retained. If the distance between the virtual obstacle and the vehicle is less than or equal to the risk threshold, the virtual obstacle is retained, and a dark block is used to represent the virtual obstacle when generating a panoramic image.

6. A vehicle panoramic image generation system based on perception fusion, characterized in that: include: A preliminary recognition module is configured to obtain image data around the vehicle and identify obstacles in the image; The model building module is configured to build a panoramic image stitching model and place obstacle blocks in the panoramic image stitching model according to obstacles in the image; An adjustment module is configured to obtain vehicle body radar data and adjust the shape and position of the obstacle block in the panoramic image stitching model based on the vehicle body radar data; The image generation module is configured to stitch the acquired image data around the vehicle based on the adjusted panoramic image stitching model to obtain a stitched panoramic image.

7. A vehicle panoramic image generation system based on perception fusion, characterized in that: include: The vehicle-mounted camera module is arranged around the vehicle and is used to obtain image data around the vehicle; A radar module is provided on the vehicle and is used to obtain vehicle body radar data; The on-board central main controller is arranged on the vehicle and is used to identify obstacles in the image based on the image data around the vehicle; construct a panoramic image stitching model, and place obstacle blocks in the panoramic image stitching model according to the obstacles in the image; adjust the shape and position of the obstacle blocks in the panoramic image stitching model based on the vehicle body radar data; and stitch the acquired image data around the vehicle based on the adjusted panoramic image stitching model to obtain a stitched panoramic image.

8. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to complete the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps of the method according to any one of claims 1 to 5.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images