Bird-eye view panorama generation method, device, equipment and program product

Abstract

The invention provides a bird's-eye view panorama generation method and device, equipment and a program product, and relates to the technical field of intelligent driving. The method comprises the steps that collected images of cameras of all paths in a vehicle are obtained, and the collected images of the cameras of all paths are combined to completely cover ground scenes around the vehicle; determining a target pixel value corresponding to an original pixel value in the acquired image through parallel processing, and filling the target pixel value into a bird's-eye view panorama template according to a pixel position mapping relation to obtain a bird's-eye view panorama of the vehicle; wherein the pixel position mapping relation is a mapping relation of pixel positions between the acquired image and a bird's-eye view panorama template. The aerial view panorama generation method and device are used for solving the problems of complex aerial view panorama generation process and low generation speed in the prior art.

Description

Technical Field

[0001] The present disclosure relates to the field of intelligent driving technology, and in particular to a method, device, equipment and program product for generating a bird's-eye view panoramic image. Background Art

[0002] Bird's-eye view panoramic images play an important role in various intelligent driving scenarios of vehicles. At present, the process of generating bird's-eye view panoramic images is relatively complex and the generation speed is slow, which needs further optimization. Summary of the invention

[0003] In view of this, the embodiments of the present disclosure provide a method, device, equipment and program product for generating a bird's-eye view panoramic image, so as to solve the problems of complex generation process and generation speed of the bird's-eye view panoramic image in the prior art.

[0004] In a first aspect, the present disclosure provides a method for generating a bird's-eye view panoramic image, comprising:

[0005] Acquire images collected by various cameras in the vehicle, wherein the images collected by various cameras are combined to completely cover the ground scene around the vehicle;

[0006] Determine the target pixel value corresponding to the original pixel value in the collected image through parallel processing, and fill the target pixel value into the bird's-eye view panoramic image template according to the pixel position mapping relationship to obtain the bird's-eye view panoramic image of the vehicle;

[0007] The pixel position mapping relationship is a pixel position mapping relationship between the captured image and the bird's-eye view panoramic image template.

[0008] In a second aspect, the present disclosure provides a bird's-eye view panoramic image generation device, comprising:

[0009] An acquisition module is used to acquire images collected by various cameras in the vehicle, wherein the images collected by various cameras are combined to completely cover the ground scene around the vehicle;

[0010] A processing module, configured to determine target pixel values ​​corresponding to original pixel values ​​in the collected image through parallel processing, and fill the target pixel values ​​into a bird's-eye view panoramic image template according to a pixel position mapping relationship, so as to obtain a bird's-eye view panoramic image of the vehicle;

[0011] The pixel position mapping relationship is a pixel position mapping relationship between the captured image and the bird's-eye view panoramic image template.

[0012] In a third aspect, the present disclosure provides an electronic device, including:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores at least one computer program executable by the at least one processor, and the at least one computer program is executed by the at least one processor so that the at least one processor can execute the bird's-eye view panoramic image generation method as described in the first aspect.

[0016] In a fourth aspect, the present disclosure provides a computer program product, which includes a computer program. When the computer program is executed in a processor, the method for generating a bird's-eye view panoramic image described in the first aspect is implemented.

[0017] The embodiment provided by the present disclosure predetermines the pixel position mapping relationship between the captured image and the bird's-eye view panoramic view template. In the process of generating the bird's-eye view panoramic view, the ground scene around the vehicle is captured by each camera in the vehicle, so that the captured images of each camera are combined to completely cover the ground scene around the vehicle. The target pixel value corresponding to the original pixel value in the captured image is determined by parallel processing, and the target pixel value is filled into the bird's-eye view panoramic view template according to the pixel position mapping relationship to obtain the bird's-eye view panoramic view of the vehicle. The bird's-eye view panoramic view can be generated by using the captured images of multiple cameras and the predetermined pixel position mapping relationship. The generation process is simple, and the generation efficiency of the bird's-eye view panoramic view is improved by parallel processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0019] Figure 1 The figure is a schematic diagram of the process of the method for generating a bird's-eye view panoramic image in an embodiment of the present disclosure;

[0020] Figure 2 The figure shows a schematic diagram of the visual range of a four-way fisheye camera in an embodiment of the present disclosure;

[0021] Figure 3 Shown is a schematic diagram of the placement of targets and vehicles in an embodiment of the present disclosure;

[0022] Figure 4 FIG. 1 is a schematic diagram of a process of calibrating the external parameters of a fisheye camera in an embodiment of the present disclosure;

[0023] Figure 5The figure shows a schematic diagram of the pixel position mapping relationship from a fisheye image to a bird's-eye view panoramic image template in an embodiment of the present disclosure;

[0024] Figure 6 The figure shows a schematic diagram of the effect of stitching together a bird's-eye view panoramic view in an embodiment of the present disclosure;

[0025] Figure 7 The figure shows a schematic diagram of the correspondence between various areas and various cameras in the bird's-eye view panoramic view in the embodiment of the present disclosure;

[0026] Figure 8 The figure is a block diagram of a device for generating a bird's-eye view panoramic image in an embodiment of the present disclosure;

[0027] Fig. 9 FIG. 1 is a schematic diagram of the structure of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0029] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0030] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0031] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0032] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0033] Overview

[0034] In the process of generating the existing bird's-eye view panoramic image of a vehicle, it is necessary to select a calibration area, and mark lines with a line laser in the calibration area to form a rectangular laser area; install cameras at eight positions of the left front, right front, left, right, front, rear, left and right of the car, and the total field of view of the eight cameras covers 360 degrees around the car body, and the field of view of two adjacent cameras overlaps; move the car to the center of the laser area and establish a vehicle coordinate system; place a checkerboard within the field of view of the camera, collect the checkerboard image within the field of view of the camera, and calculate the coordinates of the corner points on the checkerboard in the vehicle coordinate system; calculate the coordinates of the corner points in the image coordinate system; calculate the projection transformation matrix corresponding to the checkerboard image, and convert the checkerboard image into a bird's-eye view image; splice all the bird's-eye view images to obtain a bird's-eye view panoramic image of the car.

[0035] The process of generating a bird's-eye view panoramic image requires the use of lasers and 8 cameras, which is complex and slow to stitch. Based on this, a method for generating a bird's-eye view panoramic image is provided in the embodiment of the present disclosure, which does not require the use of lasers, can also reduce the number of cameras and improve the generation efficiency.

[0036] Exemplary Methods

[0037] The present disclosure provides a method for generating a bird's-eye view panoramic image, which can be used in a vehicle or in an electronic device that establishes communication with the vehicle, such as Figure 1 As shown, the method mainly includes the following steps:

[0038] Step 101, acquiring images collected by various cameras in the vehicle, wherein the images collected by various cameras are combined to completely cover the ground scene around the vehicle.

[0039] In some embodiments, each of the cameras includes four fisheye cameras, namely a front fisheye camera, a left fisheye camera, a right fisheye camera and a rear fisheye camera. Acquiring the captured images of each camera in the vehicle includes: acquiring the front fisheye image captured by the front fisheye camera in the vehicle, the left fisheye image captured by the left fisheye camera, the right fisheye image captured by the right fisheye camera and the rear fisheye image captured by the rear fisheye camera.

[0040] The combination of the front fisheye image, the left fisheye image, the right fisheye image and the rear fisheye image completely covers the ground scene around the vehicle, that is, 360° full coverage of the ground scene around the vehicle is achieved. Figure 2 The figure shows a schematic diagram of the visual range of the four-way fisheye camera. The four-way fisheye images are spliced ​​according to the overlapping parts of the fisheye images collected by the four-way fisheye camera to obtain a collected image that can fully cover the surroundings of the vehicle.

[0041] Four fisheye cameras can achieve 360° full coverage of the scene around the vehicle, reducing the number of cameras, reducing implementation complexity and saving costs.

[0042] Step 102, determining the target pixel value corresponding to the original pixel value in the acquired image through parallel processing, and filling the target pixel value into the bird's-eye view template according to the pixel position mapping relationship, to obtain the bird's-eye view of the vehicle; wherein the pixel position mapping relationship is the mapping relationship of the pixel position between the acquired image and the bird's-eye view template.

[0043] The specific processing process and processing method of parallel processing are not limited in the disclosed embodiments. The original pixel values ​​in the captured image are converted into corresponding pixel values ​​in the bird's-eye view panoramic image template through the parallel processing process, which can significantly improve the processing efficiency compared to the serial processing method.

[0044] Exemplarily, the parallel processing process is as follows:

[0045] The pixel position mapping relationship is configured in a parallel computing kernel function in a graphics processing unit (GPU) using an open computing language (OpenCL);

[0046] The front-view fisheye image, the left-view fisheye image, the right-view fisheye image, the rear-view fisheye image and the bird's-eye view panoramic image template in NV12 format are passed into the parallel computing kernel function;

[0047] The parallel computing kernel function traverses each pixel of each fisheye image through parallel processing, calculates the R, G, B values ​​of the pixel, and the offset in the bird's-eye view panoramic template, and assigns values ​​to the pixel points corresponding to the bird's-eye view panoramic template;

[0048] If the fisheye image format is other formats, the corresponding pixel conversion formula is performed.

[0049] Among them, the subsequent processing of the bird's-eye view panoramic image, such as quantization and normalization, can also be integrated into the parallel computing kernel function.

[0050] In some embodiments, the process of obtaining the pixel position mapping relationship includes: determining the external parameters of each camera through a calibration scene; wherein the calibration scene is a target placed around the vehicle; and determining the pixel position mapping relationship between the captured image and the bird's-eye view panoramic template based on the external parameters and internal parameters of each camera.

[0051] The positions of the targets in this calibration scene are preset around the vehicle, corresponding to the positions of the fisheye cameras of the vehicle. The origin of the world coordinate system is set in advance, and the center of the rear axle of the vehicle is aligned with the origin of the world coordinate system. Figure 3 The target here is a checkerboard calibration plate, which is only an example. It is not necessary to use a calibration plate with a checkerboard pattern, and calibration plates with other patterns can also be used.

[0052] In some embodiments, determining the external parameters of each camera through a calibration scene includes: obtaining calibration scene images captured by each camera, wherein the calibration scene images of each camera can be combined to completely cover the targets placed around the vehicle; for each camera, determining the external parameters of the camera based on the intrinsic parameters of the camera, the relative position of the corner points of the target and the vehicle, and the pixel coordinate values ​​of the corner points in the calibration scene image. This method can improve the calibration accuracy of the camera's external parameters, provide a basis for ensuring the accuracy of the bird's-eye view panorama, and the calibration process is simple and fast.

[0053] Specifically, the relative position of the corner point of the target and the vehicle is: the relative position of the corner point of the target to the coordinate origin of the world coordinate system.

[0054] In some embodiments, the external parameters of the camera are determined according to the intrinsic parameters of the camera, the relative position of the corner points of the target and the vehicle, and the pixel coordinate values ​​of the corner points in the calibration scene image, including: determining the world coordinate values ​​of the corner points in a world coordinate system according to the relative position of the corner points of the target and the vehicle, wherein the origin of the world coordinate system is the center point of the rear axle of the vehicle, the X-axis is parallel to the ground and points to the front of the vehicle, the Y-axis is parallel to the ground and points to the left side of the vehicle, and the Z-axis is perpendicular to the ground and faces upward; the external parameters of the camera are determined according to the intrinsic parameters of the camera, the pixel coordinate values ​​corresponding to the corner points, and the world coordinate values.

[0055] In some embodiments, determining the external parameters of the camera based on the intrinsic parameters of the camera, the pixel coordinate values ​​corresponding to the corner points, and the world coordinate values ​​includes: determining the initial external parameters of the camera based on the intrinsic parameters of the camera, the pixel coordinate values ​​corresponding to the corner points, and the world coordinate values; determining the world coordinate verification value corresponding to the corner points based on the initial external parameters, the intrinsic parameters of the camera, and the pixel coordinate values ​​corresponding to the corner points; and optimizing the initial external parameters of the camera based on the difference between the world coordinate value and the world coordinate verification value.

[0056] In some embodiments, acquiring the calibration scene images captured by each of the cameras includes: acquiring the front-view calibration scene image captured by the front-view fisheye camera, the left-view calibration scene image captured by the left-view fisheye camera, the right-view calibration scene image captured by the right-view fisheye camera, and the rear-view calibration scene image captured by the rear-view fisheye camera.

[0057] In an exemplary embodiment, Figure 4 The figure shows a schematic diagram of the process of calibrating the external parameters of the fisheye camera. The specific process is as follows:

[0058] Step 401, arrange targets on a flat ground according to designed dimensions and relative positions.

[0059] Step 402: Park a vehicle with four fisheye cameras in front, behind, left and right directions in the middle of the targets arranged on the ground.

[0060] Specifically, the target is divided into four parts: front, back, left and right. Figure 3 The relative positions shown are laid on a flat ground, and the pattern elements of each target are a black and white checkerboard pattern of the same size, and the size is a preset value. The relative positions between the targets are fixed values. A vehicle equipped with four fisheye cameras, front, rear, left and right, is parked in the middle blank area surrounded by the four targets.

[0061] Step 403: Acquire the captured images of the four fisheye cameras, and calculate the pixel coordinate value of each corner point of the target in the captured images.

[0062] Specifically, after the calibration site is arranged, the images containing the target pattern are collected from the four fisheye cameras at the current moment, namely, front, rear, left, and right.

[0063] In image processing, the upper left corner of the image is defined as the origin of the pixel coordinate system, the upper left corner to the upper right corner of the image is defined as the positive direction of the pixel coordinate system u axis, and the upper left corner to the lower left corner of the image is defined as the positive direction of the pixel coordinate system v axis. Calculating the pixel coordinates of the corner points is to calculate the pixel coordinate values ​​of the corner points of the target pattern in the pixel coordinate system through the image processing algorithm.

[0064] Step 404, measuring the world coordinate value of the target in the world coordinate system.

[0065] The origin of the world coordinate system is the center point of the rear axle of the vehicle, and the positive directions of the X and Y axes are as follows: Figure 3 As shown, the positive direction of the Z axis is defined as being perpendicular to the ground and upward. Calculating the world coordinate value of the corner point is to calculate the coordinate value of each corner point in the target pattern in the world coordinate system through the known target pattern size.

[0066] Step 405, obtaining the intrinsic parameters of the camera, and calculating the undistorted pixel coordinate values ​​corresponding to the corner points of the target.

[0067] The pixel coordinates of the corresponding corner points in the dedistorted image can be calculated through the intrinsic parameters of the camera and the pixel coordinates of the corner points in the target pattern.

[0068] Step 406 , the initial extrinsic parameters of the camera are calculated from the intrinsic parameters of the camera, the pixel coordinates of the corner points in the target pattern (ie, the undistorted pixel coordinate values) and the world coordinate values.

[0069] Step 407, optimizing the initial extrinsic parameters by minimizing the back-projection error to obtain high-precision extrinsic parameters.

[0070] Specifically, the camera extrinsic parameters are expressed as the position and posture of the camera in the world coordinate system, which can be represented by a rotation matrix and a translation vector. The principle of projecting a point in the world coordinate system to the image plane can be expressed by the following formula. The pixel coordinates of the corresponding corner point in the dedistorted image can be calculated through the camera intrinsic parameters and the pixel coordinates of the corner points of the target pattern mentioned above. Given the dedistorted pixel coordinates, world coordinates and camera intrinsic parameters of the target corner points, the camera extrinsic parameters can be calculated. Then, the extrinsic parameters with higher accuracy are optimized by minimizing the back-projection errors of all corner points.

[0071]

[0072] The coordinates in the pixel coordinate system are expressed as (u, v), and the coordinates in the world coordinate system are expressed as (X w , Y w , Z w ), R is the rotation matrix, T is the translation matrix, f x and f y are the focal lengths in the x and y directions, usually the same value, c x , c y are the coordinates of the optical center in the pixel coordinate system, Z c Indicates the depth value from the camera imaging plane.

[0073] Among them, the initial external parameters are optimized by minimizing the back-projection error to obtain high-precision external parameters. The specific process is as follows: the pixel coordinates of the target corner points in the image have been obtained. The calculated values ​​of the world coordinates corresponding to the target corner points can be obtained through the internal parameters, pixel coordinates and the initially calculated external parameters. There is a certain difference between the calculated value and the theoretical value. The coordinate difference of all target corner points is used to construct the objective function, where the parameters of the objective function are the external parameters. The corresponding external parameters at this time are obtained by minimizing the objective function, and then the external parameters at this time are the external parameter values ​​with higher precision.

[0074] In some embodiments, determining the pixel position mapping relationship between the acquired image and the bird's-eye view panoramic view template based on the external parameters and internal parameters of each camera includes: determining that the pixel points in the bird's-eye view panoramic view template are mapped to the world coordinate points of the world coordinate system; determining that the world coordinate points are mapped to the camera coordinate points of the camera coordinate system of the camera based on the internal parameters of the camera and the external parameters of the camera; determining that the camera coordinate points are mapped to the pixel coordinate points of the acquired image; and determining the pixel position mapping relationship between the acquired image and the bird's-eye view panoramic view template based on the pixel points in the bird's-eye view panoramic view template corresponding to the pixel points of the acquired image.

[0075] For example, taking the captured image as a fisheye image as an example, Figure 5 The figure shows the pixel position mapping relationship between a fisheye image and a bird's eye view (BEV) template. The mapping process is as follows:

[0076] a. Pixel coordinate system on BEV (i, j) -> b. World coordinate system (x, y) -> c. Camera coordinate system (x, y, z) -> d. Fisheye image coordinate system (x, y); In this process, BEV can be regarded as a 640*640 blank table;

[0077] Among them, a->b is to calculate the real-world coordinates corresponding to each grid in the BEV template (i.e., the position of the pixel to be filled in the BEV template). The origin of the real-world coordinates is the center of the rear axle of the vehicle. The field of view of the entire BEV is 16m*16m. The size of the final BEV template is also known, so each pixel on the BEV corresponds to a position (x, y) in the world coordinate system.

[0078] b->c maps the point in the world coordinate system to the fisheye camera according to the internal and external parameters of the camera. z=1 is due to the characteristics of the fisheye camera.

[0079] c->d obtains the corresponding position on the fisheye image from the point in the camera coordinate system according to the de-distortion parameters, that is, the pixel position on the fisheye image. The offset is used because the pixel value is determined by the offset in storage.

[0080] At this point, the pixel position lookup table (i.e., LUT table) from the BEV template to the fisheye image is completed, and the corresponding pixel position mapping relationship is obtained.

[0081] like Figure 6 The figure shows the effect of stitching the fisheye images taken by four fisheye cameras in the calibration scene into a bird's-eye view panoramic image according to the pixel position mapping relationship. The corresponding relationship between each area and each camera in the stitched bird's-eye view panoramic image can be found in Figure 7 shown.

[0082] The embodiment provided by the present disclosure predetermines the pixel position mapping relationship between the captured image and the bird's-eye view panoramic view template. In the process of generating the bird's-eye view panoramic view, the ground scene around the vehicle is captured by each camera in the vehicle, so that the captured images of each camera are combined to completely cover the ground scene around the vehicle. The target pixel value corresponding to the original pixel value in the captured image is determined by parallel processing, and the target pixel value is filled into the bird's-eye view panoramic view template according to the pixel position mapping relationship to obtain the bird's-eye view panoramic view of the vehicle. The bird's-eye view panoramic view can be generated by using the captured images of multiple cameras and the predetermined pixel position mapping relationship. The generation process is simple, and the generation efficiency of the bird's-eye view panoramic view is improved by parallel processing.

[0083] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation, the specific execution order of each step should be determined by its function and possible internal logic, and the execution order between the steps is not limited to being implemented according to the step number.

[0084] Exemplary Devices

[0085] In addition, the present disclosure also provides a bird's-eye view panoramic image generation device, an electronic device, and a computer program product, all of which can be used to implement any bird's-eye view panoramic image generation method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0086] Figure 8 A block diagram of a bird's-eye view panoramic image generation device provided in an embodiment of the present disclosure, the device mainly includes:

[0087] The acquisition module 801 is used to acquire images collected by various cameras in the vehicle, wherein the images collected by various cameras are combined to completely cover the ground scene around the vehicle;

[0088] The processing module 802 is used to determine the target pixel value corresponding to the original pixel value in the collected image through parallel processing, and fill the target pixel value into the bird's-eye view panoramic image template according to the pixel position mapping relationship to obtain the bird's-eye view panoramic image of the vehicle;

[0089] The pixel position mapping relationship is a pixel position mapping relationship between the captured image and the bird's-eye view panoramic image template.

[0090] Each module in the above device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0091] Exemplary Devices

[0092] Fig. 9 A block diagram of an electronic device provided in an embodiment of the present disclosure.

[0093] Reference Fig. 9 An embodiment of the present disclosure provides an electronic device, which includes: at least one processor 901; at least one memory 902, and one or more I / O interfaces 903, connected between the processor 901 and the memory 902; wherein the memory 902 stores one or more computer programs that can be executed by the at least one processor 901, and the one or more computer programs are executed by the at least one processor 901 so that the at least one processor 901 can execute the above-mentioned bird's-eye view panoramic image generation method.

[0094] Each module in the above electronic device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0095] Exemplary computer program products and storage media

[0096] The embodiments of the present disclosure also provide a computer program product, including a computer program, which implements the above-mentioned bird's-eye view panoramic image generation method when executed in a processor.

[0097] The computer program may be stored in a readable storage medium of a computer device or in the cloud; the processor of the computer device reads the computer program from the readable storage medium or in the cloud.

[0098] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0099] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0100] As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0101] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0102] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0103] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0104] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0105] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0106] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0107] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0108] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating a bird's-eye view panoramic image, characterized in that: include: Acquire images collected by various cameras in the vehicle, wherein the images collected by various cameras are combined to completely cover the ground scene around the vehicle; Determine the target pixel value corresponding to the original pixel value in the collected image through parallel processing, and fill the target pixel value into the bird's-eye view panoramic image template according to the pixel position mapping relationship to obtain the bird's-eye view panoramic image of the vehicle; The pixel position mapping relationship is a pixel position mapping relationship between the captured image and the bird's-eye view panoramic image template.

2. The method according to claim 1, characterized in that The process of obtaining the pixel position mapping relationship includes: Determine the external parameters of each camera through a calibration scene; wherein the calibration scene is a target placed around the vehicle; According to the external parameters and internal parameters of each camera, the pixel position mapping relationship between the collected image and the bird's-eye view panoramic image template is determined.

3. The method according to claim 2, characterized in that Determining the external parameters of each camera through the calibration scene includes: Acquire calibration scene images captured by each of the cameras, wherein the calibration scene images captured by each of the cameras are combined to completely cover the targets placed around the vehicle; For each of the cameras, the external parameters of the camera are determined according to the internal parameters of the camera, the relative positions of the corner points of the target and the vehicle, and the pixel coordinate values ​​of the corner points in the calibration scene image.

4. The method according to claim 3, characterized in that The cameras include four fisheye cameras, namely a front-view fisheye camera, a left-view fisheye camera, a right-view fisheye camera and a rear-view fisheye camera; The step of obtaining the calibration scene images collected by the cameras includes: Acquire a front-view calibration scene image captured by the front-view fisheye camera, a left-view calibration scene image captured by the left-view fisheye camera, a right-view calibration scene image captured by the right-view fisheye camera, and a rear-view calibration scene image captured by the rear-view fisheye camera.

5. The method according to claim 4, characterized in that The step of determining the external parameters of the camera according to the internal parameters of the camera, the relative position of the corner point of the target and the vehicle, and the pixel coordinate value of the corner point in the calibration scene image comprises: According to the relative position between the corner point of the target and the vehicle, determine the world coordinate value of the corner point in the world coordinate system, wherein the origin of the world coordinate system is the center point of the rear axle of the vehicle, the X-axis is parallel to the ground and points to the front of the vehicle, the Y-axis is parallel to the ground and points to the left side of the vehicle, and the Z-axis is perpendicular to the ground and faces upward; The extrinsic parameters of the camera are determined according to the intrinsic parameters of the camera, the pixel coordinate values ​​corresponding to the corner points, and the world coordinate values.

6. The method according to claim 5, characterized in that The determining the extrinsic parameters of the camera according to the intrinsic parameters of the camera, the pixel coordinate values ​​corresponding to the corner points, and the world coordinate values ​​includes: Determine the initial extrinsic parameters of the camera according to the intrinsic parameters of the camera, the pixel coordinate values ​​corresponding to the corner points, and the world coordinate values; Determine a world coordinate verification value corresponding to the corner point according to the initial external parameter, the intrinsic parameter of the camera and the pixel coordinate value corresponding to the corner point; The initial extrinsic parameters of the camera are optimized according to the difference between the world coordinate value and the world coordinate verification value.

7. The method according to claim 6, characterized in that Determining the pixel position mapping relationship between the captured image and the bird's-eye view panoramic image template according to the external parameters and internal parameters of each camera includes: Determine the pixel points in the bird's-eye view panorama template mapped to the world coordinate points of the world coordinate system; Determine, according to the intrinsic parameters of the camera and the extrinsic parameters of the camera, a camera coordinate point of a camera coordinate system of the camera mapped from the world coordinate point; Determine the pixel coordinate point mapped from the camera coordinate point to the acquired image; According to the pixel points in the bird's-eye view panoramic image template corresponding to the pixel points of the collected image, a pixel position mapping relationship between the collected image and the bird's-eye view panoramic image template is determined.

8. A device for generating a bird's-eye view panoramic image, characterized in that: include: An acquisition module is used to acquire images collected by various cameras in the vehicle, wherein the images collected by various cameras are combined to completely cover the ground scene around the vehicle; A processing module, configured to determine target pixel values ​​corresponding to original pixel values ​​in the collected image through parallel processing, and fill the target pixel values ​​into a bird's-eye view panoramic image template according to a pixel position mapping relationship, so as to obtain a bird's-eye view panoramic image of the vehicle; The pixel position mapping relationship is a pixel position mapping relationship between the captured image and the bird's-eye view panoramic image template.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores at least one computer program executable by the at least one processor, and the at least one computer program is executed by the at least one processor so as to enable the at least one processor to perform the bird's-eye view panoramic image generating method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed in a processor, the method for generating a bird's-eye view panoramic image according to any one of claims 1 to 7 is implemented.

Images