Image generation method and device, vehicle and storage medium

Abstract

The invention discloses an image generation method and device, a vehicle and a storage medium, relates to the technical field of image processing, and is used for enabling the color space of an all-round aerial view of the vehicle to be more balanced. The method includes; for each look-around camera deployed on the vehicle, performing coordinate mapping from a camera coordinate system to an imaging plane in a world coordinate system on a current scene image of the look-around camera to obtain a panoramic sub-image of the look-around camera; in the mapping process, pixel correction is carried out on the pixel value of the first pixel point in the current scene image of the look-around camera to obtain the pixel value of the first pixel point in the panoramic sub-image of the look-around camera; the first pixel point is a pixel point corresponding to an overlapping area of the shooting ranges of the surround-view camera and the adjacent surround-view camera; carrying out pixel fusion on the pixel values of the pixel points of the panoramic sub-images in the overlapping region; and splicing the overlapped region after pixel fusion and the non-overlapped region in the panoramic sub-image to obtain a look-around overhead view of the vehicle.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image generation method, device, vehicle and storage medium. Background Art

[0002] Vehicles are usually equipped with multiple cameras with different shooting angles. By stitching together the images taken by different cameras, a surround view image that can show the situation around the vehicle can be obtained. In practical applications, cameras are usually installed on the front, left, right and rear of the vehicle. The images taken by each camera are stitched together to obtain a bird's-eye view of the vehicle, so that the driver can better understand the environment around the vehicle and improve the safety of vehicle driving.

[0003] However, the installation angles, lighting conditions, and exposure levels of cameras with different shooting angles are all different to a certain extent, so the color spaces of various areas of the stitched panoramic bird's-eye view map will have certain differences, and the display effect of the panoramic bird's-eye view map is not good. Summary of the invention

[0004] The present application provides an image generation method, device, vehicle and storage medium, which are used to make the color space of a vehicle's surround view bird's-eye view image more balanced.

[0005] In order to achieve the above technical objectives, this application adopts the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides an image generation method, the method comprising:

[0007] Acquire the current scene images around the vehicle captured by each of the multiple surround view cameras deployed on the vehicle, where the shooting range of all surround view cameras deployed on the vehicle covers 360° around the vehicle;

[0008] For each surround-view camera, coordinate mapping is performed on the current scene image of the surround-view camera from the camera coordinate system to the imaging plane in the world coordinate system to obtain a panoramic sub-image of the surround-view camera; in the mapping process, pixel correction is performed on the pixel value of the first pixel point in the current scene image of the surround-view camera based on the pixel value of the first pixel point in the current scene image of the adjacent surround-view camera to obtain the pixel value of the first pixel point in the panoramic sub-image of the surround-view camera; the first pixel point is a pixel point corresponding to the overlapping area of ​​the shooting range of the surround-view camera and the adjacent surround-view camera;

[0009] Perform pixel fusion on the pixel values ​​of the pixels of the panoramic sub-images in the overlapping area;

[0010] The overlapping areas after pixel fusion and the non-overlapping areas in the panoramic sub-image are spliced ​​to obtain a bird's-eye view of the vehicle.

[0011] The technical solution provided by the present application brings at least the following beneficial effects: in the process of generating the surround view bird's-eye view map of the vehicle, it is necessary to map the coordinates of the current scene images taken by each surround view camera from the camera coordinate system to the imaging plane in the world coordinate system to obtain the panoramic sub-images corresponding to each current scene image. In the mapping process, each current scene image will be pixel-corrected based on the current scene image taken by the adjacent surround view camera. Therefore, the panoramic sub-image corresponding to the current scene image refers to the image taken by the adjacent surround view camera, and the brightness, contrast, chromaticity, etc. of the pixels in each panoramic sub-image are closer, that is, the color space of each panoramic sub-image is closer. In addition, when the panoramic sub-images are spliced, the pixels in the overlapping area are pixel-fused, and the display effect of the overlapping area in the final surround view bird's-eye view map is close to the display effect of the two adjacent panoramic sub-images. The transition of each area of ​​the surround view bird's-eye view map is natural, and the color space of each area in the entire surround view bird's-eye view map is more balanced, and the problem of excessive color space differences in different areas of the surround view bird's-eye view map is avoided as much as possible, and the display effect is better.

[0012] In a possible implementation, based on the pixel value of the first pixel in the current scene image of the adjacent surround view camera, pixel correction is performed on the pixel value of the first pixel in the current scene image of the surround view camera to obtain the pixel value of the first pixel in the panoramic sub-image of the surround view camera, including: weighted fusion of the pixel value of the first pixel in the current scene image of the surround view camera and the pixel value of the first pixel in the current scene image of the adjacent surround view camera to obtain a fused pixel value; and the product of a historical gain value and a gain coefficient plus the fused pixel value to obtain the pixel value of the first pixel in the panoramic sub-image of the surround view camera; wherein the historical gain value is used to characterize the degree of difference in shooting results of the surround view camera and the adjacent surround view camera for the spatial position corresponding to the first pixel within a historical time period.

[0013] In a possible implementation, the gain coefficient is determined based on the absolute value of the difference between the first fusion weight and the second fusion weight, the first fusion weight being the weight used when the pixel value of the first pixel in the current scene image of the surround view camera participates in the weighted fusion calculation, and the second fusion weight being the weight used when the pixel value of the first pixel in the current scene image of the adjacent surround view camera participates in the weighted fusion calculation.

[0014] In one possible implementation, when the absolute value of the difference between the first fusion weight and the second fusion weight is greater than a preset threshold, the gain coefficient is 0; or, when the absolute value of the difference between the first fusion weight and the second fusion weight is less than or equal to the preset threshold, the gain coefficient is 1.

[0015] In a possible implementation, a pixel value of a first pixel point in a current scene image of a surround-view camera and a pixel value of the first pixel point in a current scene image of an adjacent surround-view camera are weightedly fused to obtain a fused pixel value, including: obtaining multiple pixel values ​​of a second pixel point in multiple historical scene images taken by the surround-view camera, and multiple pixel values ​​of the second pixel point in multiple historical scene images taken by other surround-view cameras; the second pixel point is a pixel point in the historical scene image corresponding to the same spatial position as the first pixel point; a first variance value is calculated based on the multiple pixel values ​​of the second pixel point in the multiple historical scene images taken by the surround-view camera, and a second variance value is calculated based on the multiple pixel values ​​of the second pixel point in the multiple historical scene images taken by other surround-view cameras; The first fusion weight and the second fusion weight used by the surround-view camera at the previous moment are corrected according to the size relationship between the difference value and the second variance value, so as to obtain the first fusion weight and the second fusion weight used by the surround-view camera at the current moment; the first fusion weight is the weight used when the pixel value of the first pixel point in the current scene image of the surround-view camera participates in the weighted fusion calculation, and the second fusion weight is the weight used when the pixel value of the first pixel point in the current scene image of the adjacent surround-view camera participates in the weighted fusion calculation; based on the first fusion weight and the second fusion weight used by the surround-view camera at the current moment, the pixel value of the first pixel point in the current scene image of the surround-view camera and the pixel value of the first pixel point in the current scene image of the adjacent surround-view camera are weightedly fused to obtain the fused pixel value.

[0016] In a possible implementation, based on the size relationship between the first variance value and the second variance value, the first fusion weight and the second fusion weight used by the surround-view camera at the previous moment are corrected to obtain the first fusion weight and the second fusion weight used by the surround-view camera at the current moment, including: when the first variance value is less than the second variance value, the first fusion weight used by the surround-view camera at the previous moment is increased by a preset value to obtain the first fusion weight used by the surround-view camera at the current moment, and the second fusion weight used by the surround-view camera at the previous moment is reduced by a preset value to obtain the second fusion weight used by the surround-view camera at the current moment; in the When a variance value is greater than a second variance value, the first fusion weight used by the surround-view camera at the last moment is reduced by a preset value to obtain the first fusion weight used by the surround-view camera at the current moment, and the second fusion weight used by the surround-view camera at the last moment is increased by a preset value to obtain the second fusion weight used by the surround-view camera at the current moment; when the first variance value is equal to the second variance value, the first fusion weight used by the surround-view camera at the last moment is used as the first fusion weight used by the surround-view camera at the current moment, and the second fusion weight used by the surround-view camera at the last moment is used as the second fusion weight used by the surround-view camera at the current moment.

[0017] In a possible implementation, the first variance value and multiple pixel values ​​of the second pixel point in multiple historical scene images captured by the surround view camera satisfy the following relationship: in, Used to represent the first variance value, PixCur i It is used to represent the pixel value of the second pixel of the i-th pixel in the current scene image captured by the surround camera. is used to represent the average value of multiple pixel values ​​of the i-th second pixel point in multiple historical scene images taken by the surround view camera, and M is used to represent the number of second pixel points in the current scene image taken by the surround view camera; the second variance value and the multiple pixel values ​​of the second pixel point in multiple historical scene images taken by other surround view cameras satisfy the following relationship: in, Used to represent the second variance value, pixAdj j It is used to represent the pixel value of the jth second pixel in the current scene image captured by the adjacent surround view camera. It is used to represent the average value of multiple pixel values ​​of the j-th second pixel in multiple historical scene images taken by other surround view cameras, and N is used to represent the number of second pixels in the current scene image taken by adjacent surround view cameras; the fused pixel value of the first pixel in the panoramic sub-image of the surround view camera and the first fusion weight, the second fusion weight, the pixel value of the first pixel in the current scene image taken by the surround view camera, and the pixel value of the first pixel in the current scene image taken by the adjacent surround view camera satisfy the following relationship: Among them, Pix n It is used to represent the fused pixel value of the nth pixel in the panoramic sub-image of the surround camera, w1 is used to represent the first fusion weight, w2 is used to represent the second fusion weight, PixCur n PixAdj is used to represent the pixel value of the nth first pixel in the current scene image captured by the surround camera. n Used to indicate that PixAdj n It is used to represent the pixel value of the nth first pixel in the current scene image captured by the adjacent surround view camera, and a is used to represent the gain coefficient, PixGain n It is used to represent the historical gain value corresponding to the nth first pixel point; the historical gain value and the multiple pixel values ​​of the second pixel point in the multiple historical scene images taken by the surround view camera and the multiple pixel values ​​of the second pixel point in the multiple historical scene images taken by other surround view cameras satisfy the following relationship: Among them, PixGain m Used to represent the historical gain value of the mth second pixel point, It is used to represent the average value of multiple pixel values ​​of the m-th second pixel in multiple historical scene images taken by the surround camera. Used to represent the average value of multiple pixel values ​​of the m-th second pixel point in multiple historical scene images taken by other surround view cameras.

[0018] In one possible implementation, multiple pixel values ​​of a second pixel point in multiple historical scene images taken by a surround-view camera, as well as multiple pixel values ​​of the second pixel point in multiple historical scene images taken by other surround-view cameras, are obtained, including: obtaining vehicle displacement information corresponding to each frame of the historical scene image, the vehicle displacement information corresponding to the historical scene image being the vehicle displacement information from the shooting time of the historical scene image to the shooting time of the current scene image; for each frame of the historical scene image taken by the surround-view camera, based on the vehicle displacement information corresponding to the historical scene image and the pixel coordinates of the third pixel point on the current scene image taken by the surround-view camera, determining a first corresponding pixel point of the third pixel point from the historical scene image; using the first corresponding pixel point and the corresponding third pixel point that meet a preset condition as second pixel points representing the same spatial position, using the first corresponding pixel point and the corresponding third pixel point that do not meet the preset condition as pixel points in a three-dimensional obstacle, and determining that the pixel point in the three-dimensional obstacle does not belong to second pixel point; for each frame of historical scene image taken by other surround-view cameras, based on the vehicle displacement information corresponding to the historical scene image and the pixel coordinates of the fourth pixel point on the current scene image taken by the adjacent surround-view camera, determine the second corresponding pixel point of the fourth pixel point from the historical scene image; the second corresponding pixel point and the corresponding fourth pixel point that meet the preset conditions are used as the second pixel points representing the same spatial position, and the second corresponding pixel point and the corresponding fourth pixel point that do not meet the preset conditions are used as the pixel points in the three-dimensional obstacle, and determine that the pixel point in the three-dimensional obstacle does not belong to the second pixel point; the preset conditions include: the Euclidean distance corresponding to the corresponding pixel point satisfies the distance threshold condition, and the Euclidean distance corresponding to at least one adjacent pixel point of the corresponding pixel point satisfies the distance threshold condition, the Euclidean distance corresponding to the corresponding pixel point is used to represent the Euclidean distance between the corresponding pixel point and the corresponding pixel point in the current scene image in the color space, and the distance threshold condition is used to measure the color similarity between the two pixel points.

[0019] In a possible implementation, the vehicle displacement information corresponding to each frame of the historical scene image is obtained, including: for each frame of the historical scene image, obtaining the shooting time interval, the vehicle's moving speed within the shooting time interval, and the steering angle; the shooting time interval is the time interval from the shooting moment of the historical scene image to the shooting moment of the current scene image; based on the shooting time interval, the vehicle's moving speed within the shooting time interval, and the steering angle, the vehicle displacement information corresponding to the historical scene image is determined.

[0020] In a possible implementation, the vehicle displacement information satisfies the following relationship with the shooting time interval, the moving speed of the vehicle within the shooting time interval, and the steering angle: Δx=cos(θ a )VΔt;Δy=sin(θ a )VΔt; where Δx is the displacement of the vehicle in the X-axis direction in the world coordinate system with the vehicle body as the origin in the vehicle displacement information, Δy is the displacement of the vehicle in the Y-axis direction in the world coordinate system with the vehicle body as the origin in the vehicle displacement information, θ a is the steering angle of the vehicle during the shooting time interval, Δt is the shooting time interval, and V is the moving speed of the vehicle during the shooting time interval.

[0021] In one possible implementation, when the scene represented by the current scene image is a grass scene, the initial value of the first fusion weight is determined to be a first preset value, and the initial value of the second fusion weight is determined to be the first preset value; when the scene represented by the current scene image is a brightness mutation scene, the initial value of the first fusion weight is determined to be a second preset value, the initial value of the second fusion weight is determined to be a third preset value, the gain coefficient is determined to be 0, and the second preset value is greater than the third preset value.

[0022] In a possible implementation, the surround view camera is a fisheye camera, and current scene images around the vehicle captured by multiple surround view cameras deployed on the vehicle are obtained, including: obtaining a fisheye image captured by the fisheye camera; and performing distortion correction on the fisheye image based on internal and external camera parameters of the fisheye camera to obtain a current scene image.

[0023] In a second aspect, the present application provides an image generating device, comprising:

[0024] An acquisition module is used to acquire the current scene images around the vehicle captured by each of the multiple surround view cameras deployed on the vehicle. The shooting range of all surround view cameras deployed on the vehicle covers 360° around the vehicle.

[0025] A processing module is used to, for each surround-view camera, perform coordinate mapping of the current scene image of the surround-view camera from the camera coordinate system to the imaging plane in the world coordinate system to obtain a panoramic sub-image of the surround-view camera; in the mapping process, based on the pixel value of the first pixel point in the current scene image of the adjacent surround-view camera, perform pixel correction on the pixel value of the first pixel point in the current scene image of the surround-view camera to obtain the pixel value of the first pixel point in the panoramic sub-image of the surround-view camera; the first pixel point is a pixel point corresponding to an overlapping area of ​​the shooting range of the surround-view camera and the adjacent surround-view camera;

[0026] The processing module is also used to perform pixel fusion on pixel values ​​of pixel points in the overlapping area of ​​the panoramic sub-images;

[0027] The processing module is also used to stitch the overlapping areas after pixel fusion and the non-overlapping areas in the panoramic sub-image to obtain a bird's-eye view of the vehicle.

[0028] In one possible implementation, the processing module is specifically used to: perform weighted fusion on a pixel value of a first pixel in a current scene image of a surround-view camera and a pixel value of the first pixel in a current scene image of an adjacent surround-view camera to obtain a fused pixel value; add a product of a historical gain value and a gain coefficient to the fused pixel value to obtain a pixel value of the first pixel in a panoramic sub-image of the surround-view camera; wherein the historical gain value is used to characterize the degree of difference in shooting results of the surround-view camera and the adjacent surround-view camera for the spatial position corresponding to the first pixel within a historical time period.

[0029] In a possible implementation, the gain coefficient is determined based on the absolute value of the difference between the first fusion weight and the second fusion weight, the first fusion weight being the weight used when the pixel value of the first pixel in the current scene image of the surround view camera participates in the weighted fusion calculation, and the second fusion weight being the weight used when the pixel value of the first pixel in the current scene image of the adjacent surround view camera participates in the weighted fusion calculation.

[0030] In one possible implementation, when the absolute value of the difference between the first fusion weight and the second fusion weight is greater than a preset threshold, the gain coefficient is 0; or, when the absolute value of the difference between the first fusion weight and the second fusion weight is less than or equal to the preset threshold, the gain coefficient is 1.

[0031] In a possible implementation, the acquisition module is further used to obtain multiple pixel values ​​of the second pixel point in multiple historical scene images taken by the surround-view camera, and multiple pixel values ​​of the second pixel point in multiple historical scene images taken by other surround-view cameras; the second pixel point is a pixel point in the historical scene image that corresponds to the same spatial position as the first pixel point; the processing module is specifically used to: calculate a first variance value based on the multiple pixel values ​​of the second pixel point in the multiple historical scene images taken by the surround-view camera, and calculate a second variance value based on the multiple pixel values ​​of the second pixel point in the multiple historical scene images taken by other surround-view cameras; based on the size relationship between the first variance value and the second variance value, The first fusion weight and the second fusion weight adopted at the moment are corrected to obtain the first fusion weight and the second fusion weight adopted by the surround-view camera at the current moment; the first fusion weight is the weight adopted when the pixel value of the first pixel point in the current scene image of the surround-view camera participates in the weighted fusion calculation, and the second fusion weight is the weight adopted when the pixel value of the first pixel point in the current scene image of the adjacent surround-view camera participates in the weighted fusion calculation; based on the first fusion weight and the second fusion weight adopted by the surround-view camera at the current moment, the pixel value of the first pixel point in the current scene image of the surround-view camera and the pixel value of the first pixel point in the current scene image of the adjacent surround-view camera are weightedly fused to obtain a fused pixel value.

[0032] In a possible implementation, the processing module is specifically used to: when the first variance value is less than the second variance value, increase the first fusion weight used by the surround-view camera at the last moment by a preset value to obtain the first fusion weight used by the surround-view camera at the current moment, and reduce the second fusion weight used by the surround-view camera at the last moment by a preset value to obtain the second fusion weight used by the surround-view camera at the current moment; when the first variance value is greater than the second variance value, reduce the first fusion weight used by the surround-view camera at the last moment by a preset value to obtain the first fusion weight used by the surround-view camera at the current moment, and increase the second fusion weight used by the surround-view camera at the last moment by a preset value to obtain the second fusion weight used by the surround-view camera at the current moment; when the first variance value is equal to the second variance value, use the first fusion weight used by the surround-view camera at the last moment as the first fusion weight used by the surround-view camera at the current moment, and use the second fusion weight used by the surround-view camera at the last moment as the second fusion weight used by the surround-view camera at the current moment.

[0033] In a possible implementation, the first variance value and multiple pixel values ​​of the second pixel point in multiple historical scene images captured by the surround view camera satisfy the following relationship: in, Used to represent the first variance value, PixCur iIt is used to represent the pixel value of the second pixel of the i-th pixel in the current scene image captured by the surround camera. is used to represent the average value of multiple pixel values ​​of the i-th second pixel point in multiple historical scene images taken by the surround view camera, and M is used to represent the number of second pixel points in the current scene image taken by the surround view camera; the second variance value and the multiple pixel values ​​of the second pixel point in multiple historical scene images taken by other surround view cameras satisfy the following relationship: in, Used to represent the second variance value, PixAdj j It is used to represent the pixel value of the jth second pixel in the current scene image captured by the adjacent surround view camera. It is used to represent the average value of multiple pixel values ​​of the j-th second pixel in multiple historical scene images taken by other surround view cameras, and N is used to represent the number of second pixels in the current scene image taken by adjacent surround view cameras; the fused pixel value of the first pixel in the panoramic sub-image of the surround view camera and the first fusion weight, the second fusion weight, the pixel value of the first pixel in the current scene image taken by the surround view camera, and the pixel value of the first pixel in the current scene image taken by the adjacent surround view camera satisfy the following relationship: Among them, Pix n It is used to represent the fused pixel value of the nth pixel in the panoramic sub-image of the surround camera, w1 is used to represent the first fusion weight, w2 is used to represent the second fusion weight, PixCur n PixAdj is used to represent the pixel value of the nth first pixel in the current scene image captured by the surround camera. n Used to indicate that PixAdj n It is used to represent the pixel value of the nth first pixel in the current scene image captured by the adjacent surround view camera, and a is used to represent the gain coefficient, PixGain n It is used to represent the historical gain value corresponding to the nth first pixel point; the historical gain value and the multiple pixel values ​​of the second pixel point in the multiple historical scene images taken by the surround view camera and the multiple pixel values ​​of the second pixel point in the multiple historical scene images taken by other surround view cameras satisfy the following relationship: Among them, PixGain m Used to represent the historical gain value of the mth second pixel point, It is used to represent the average value of multiple pixel values ​​of the m-th second pixel in multiple historical scene images taken by the surround camera. Used to represent the average value of multiple pixel values ​​of the m-th second pixel point in multiple historical scene images taken by other surround view cameras.

[0034] In a possible implementation, the acquisition module is specifically used to acquire vehicle displacement information corresponding to each frame of historical scene image, where the vehicle displacement information corresponding to the historical scene image is the vehicle displacement information from the time when the historical scene image is shot to the time when the current scene image is shot; the processing module is also used to: for each frame of historical scene image shot by the surround-view camera, based on the vehicle displacement information corresponding to the historical scene image and the pixel coordinates of the third pixel point on the current scene image shot by the surround-view camera, determine the first corresponding pixel point of the third pixel point from the historical scene image; use the first corresponding pixel point and the corresponding third pixel point that meet the preset conditions as the second pixel point representing the same spatial position, use the first corresponding pixel point and the corresponding third pixel point that do not meet the preset conditions as the pixel point in the three-dimensional obstacle, and determine that the pixel point in the three-dimensional obstacle does not belong to the second pixel point; for each frame of historical scene image shot by other surround-view cameras, Based on the vehicle displacement information corresponding to the historical scene image and the pixel coordinates of the fourth pixel point on the current scene image taken by the adjacent surround view camera, a second corresponding pixel point of the fourth pixel point is determined from the historical scene image; the second corresponding pixel point that meets the preset conditions and the corresponding fourth pixel point are used as the second pixel point representing the same spatial position, and the second corresponding pixel point that does not meet the preset conditions and the corresponding fourth pixel point are used as the pixel points in the three-dimensional obstacle, and it is determined that the pixel point in the three-dimensional obstacle does not belong to the second pixel point; the preset conditions include: the Euclidean distance corresponding to the corresponding pixel point satisfies the distance threshold condition, and the Euclidean distance corresponding to at least one adjacent pixel point of the corresponding pixel point satisfies the distance threshold condition, the Euclidean distance corresponding to the corresponding pixel point is used to represent the Euclidean distance between the corresponding pixel point and the corresponding pixel point in the current scene image in the color space, and the distance threshold condition is used to measure the color similarity between the two pixel points.

[0035] In one possible implementation, the acquisition module is specifically used to obtain, for each frame of the historical scene image, a shooting time interval, a moving speed of the vehicle within the shooting time interval, and a turning angle; the shooting time interval is the time interval from the shooting moment of the historical scene image to the shooting moment of the current scene image; the processing module is also used to determine the vehicle displacement information corresponding to the historical scene image based on the shooting time interval, the moving speed of the vehicle within the shooting time interval, and the turning angle.

[0036] In a possible implementation, the vehicle displacement information satisfies the following relationship with the shooting time interval, the moving speed of the vehicle within the shooting time interval, and the steering angle: Δx=cos(θ a )VΔt;Δy=sin(θ a)VΔt; where Δx is the displacement of the vehicle in the X-axis direction in the world coordinate system with the vehicle body as the origin in the vehicle displacement information, Δy is the displacement of the vehicle in the Y-axis direction in the world coordinate system with the vehicle body as the origin in the vehicle displacement information, θ a is the steering angle of the vehicle during the shooting time interval, Δt is the shooting time interval, and V is the moving speed of the vehicle during the shooting time interval.

[0037] In one possible implementation, the processing module is also used to: when the scene represented by the current scene image is a grass scene, determine that the initial value of the first fusion weight is a first preset value, and determine that the initial value of the second fusion weight is the first preset value; when the scene represented by the current scene image is a brightness mutation scene, determine that the initial value of the first fusion weight is a second preset value, determine that the initial value of the second fusion weight is a third preset value, determine that the gain coefficient is 0, and the second preset value is greater than the third preset value.

[0038] In a possible implementation, the surround view camera is a fisheye camera, and the acquisition module is specifically used to acquire a fisheye image taken by the fisheye camera; the processing module is also used to perform distortion correction on the fisheye image based on internal and external camera parameters of the fisheye camera to obtain a current scene image.

[0039] In a third aspect, the present application provides a vehicle comprising multiple cameras with different shooting angles and an electronic device; the camera is used to capture images and send the images to the electronic device; the electronic device is used to receive images sent by each camera and execute the steps of any one of the image generation methods provided in the first aspect above.

[0040] In a fourth aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any one of the image generation methods provided in the first aspect.

[0041] In a fifth aspect, the present application provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of any one of the image generation methods provided in the first aspect above.

[0042] In a sixth aspect, the present application provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of any one of the image generation methods provided in the first aspect above.

[0043] For the specific description of the second to sixth aspects and their various implementations in the present application, reference may be made to the detailed description in the first aspect and its various implementations; and for the beneficial effects of the second to sixth aspects and their various implementations, reference may be made to the analysis of the beneficial effects in the first aspect and its various implementations, which will not be repeated here.

[0044] These and other aspects of the present application will become more apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of the architecture of a vehicle to which an image generation method provided in an embodiment of the present application is applicable;

[0046] Figure 2 A schematic diagram of the hardware composition of a computing device provided in an embodiment of the present application;

[0047] Figure 3 A schematic diagram of a flow chart of an image generation method provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of a use scenario of an image generation method provided in an embodiment of the present application Figure 1 ;

[0049] Figure 5 A schematic diagram of a use scenario of an image generation method provided in an embodiment of the present application Figure 2 ;

[0050] Figure 6 A schematic diagram of a use scenario of an image generation method provided in an embodiment of the present application Figure 3 ;

[0051] Figure 7 A schematic diagram of a use scenario of an image generation method provided in an embodiment of the present application Figure 4 ;

[0052] Figure 8 A schematic diagram of a use scenario of an image generation method provided in an embodiment of the present application Figure 5 ;

[0053] Fig. 9 A schematic diagram of a use scenario of an image generation method provided in an embodiment of the present application Figure 6 ;

[0054] Fig.10 A schematic diagram of a use scenario of an image generation method provided in an embodiment of the present application Figure 7 ;

[0055] Fig.11 A schematic diagram of a use scenario of an image generation method provided in an embodiment of the present application Figure 8 ;

[0056] Fig.12 A schematic diagram of the structure of an image generating device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. The terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, "multiple" means two or more.

[0059] Vehicles are usually equipped with multiple cameras with different shooting angles. By stitching together the images taken by different cameras, a surround view image that can show the situation around the vehicle can be obtained. In practical applications, cameras are usually installed on the front, left, right and rear of the vehicle. The images taken by each camera are stitched together to obtain a bird's-eye view of the vehicle, so that the driver can better understand the environment around the vehicle and improve the safety of vehicle driving.

[0060] However, the installation angles, lighting conditions, and exposure levels of cameras with different shooting angles are all different to a certain extent, so the color spaces of various areas of the stitched panoramic bird's-eye view map will have certain differences, and the display effect of the panoramic bird's-eye view map is not good.

[0061] In this regard, an embodiment of the present application provides an image generation method, which performs coordinate mapping from the camera coordinate system to the imaging plane in the world coordinate system for the current scene image captured by each surround-view camera of the vehicle, and obtains the panoramic sub-image corresponding to each current scene image. During the mapping process, each current scene image will be pixel-corrected based on the current scene image captured by the adjacent surround-view camera. Therefore, the panoramic sub-image corresponding to the current scene image refers to the image captured by the adjacent surround-view camera, and the brightness, contrast, chromaticity, etc. of the pixels in each panoramic sub-image are closer, that is, the color space of each panoramic sub-image is closer. In addition, when the panoramic sub-images are spliced, the pixels in the overlapping area are pixel-fused, and the display effect of the overlapping area in the final surround-view bird's-eye view image is close to the display effect of the two adjacent panoramic sub-images, and the transition of each area of ​​the surround-view bird's-eye view image is natural, and the color space of each area in the entire surround-view bird's-eye view image is more balanced, and the problem of excessive color space differences in different areas of the surround-view bird's-eye view image is avoided as much as possible, and the display effect is better.

[0062] Please refer to Figure 1 , which shows the vehicle to which the image generation method provided by the present application is applicable. Figure 1 As shown, the vehicle 1 includes: a plurality of surround view cameras 10 with different shooting angles and an electronic device 20 .

[0063] The surround view camera 10 and the electronic device 20 establish a communication connection in a direct or indirect manner. It should be understood that the connection method can be a wireless connection, such as a Bluetooth connection, a wireless fidelity (Wi-Fi) connection, etc.; or the connection method can also be a wired connection, such as an optical fiber connection, etc., which is not limited to this. For example, the surround view camera 10 and the electronic device 20 are connected via a wireless local area network.

[0064] In some embodiments, multiple surround view cameras 10 are installed at different positions of the vehicle, and the shooting range of all surround view cameras deployed on the vehicle covers 360° around the vehicle, so that each surround view camera 10 can capture images from different perspectives. For example, a surround view camera 10 is installed at the front, back, left, and right of the vehicle, respectively, so that the front view camera can capture the scene in front of the vehicle, the left view camera can capture the scene to the left of the vehicle, the right view camera can capture the scene to the right of the vehicle, and the rear view camera can capture the scene behind the vehicle. Finally, the images captured by these four cameras are stitched together to observe a panoramic image around the vehicle.

[0065] Specifically, the surround view camera 10 may be a fisheye camera, a monocular camera, a binocular camera, a trinocular camera, a wide-angle camera, etc., and the present application does not limit the specific implementation of the surround view camera 10. In practical applications, the surround view camera 10 is usually a fisheye camera.

[0066] In some embodiments, the electronic device 20 is used to receive images captured by each surround view camera 10, and to perform coordinate mapping of the current scene image captured by each surround view camera from the camera coordinate system to the imaging plane in the world coordinate system to obtain the panoramic sub-image of the surround view camera. During the mapping process, the electronic device 20 is also used to perform pixel correction on the pixel value of the first pixel point in the current scene image of the surround view camera based on the pixel value of the first pixel point in the current scene image of the adjacent surround view camera to obtain the pixel value of the first pixel point in the panoramic sub-image of the surround view camera. The first pixel point is a pixel point corresponding to the overlapping area of ​​the shooting range of the surround view camera and the adjacent surround view camera.

[0067] In some embodiments, the electronic device 20 is also used to perform pixel fusion on the pixel values ​​of the pixel points in the overlapping area of ​​the panoramic sub-image, and to splice the overlapping area after pixel fusion and the non-overlapping area in the panoramic sub-image to obtain a surround view bird's-eye view of the vehicle.

[0068] In some embodiments, the electronic device 20 may include a display for displaying to the user a bird's-eye view of the surrounding view obtained by stitching images captured by various surround view cameras on the vehicle. For example, the display may be a liquid crystal display, an organic light-emitting diode (OLED) display, etc., and the specific type, size, and resolution of the display are not limited.

[0069] Optionally, the electronic device 20 is connected to a display device, and the obtained surround-view bird's-eye view image is sent to the display device, so that the display device displays the surround-view bird's-eye view image to the user.

[0070] In some embodiments, the electronic device 20 can be a single server or a server cluster. Alternatively, the electronic device 20 can also be a terminal device, such as a personal computer (PC), a laptop computer, a mobile device, a tablet computer, a laptop computer, a vehicle-mounted terminal, etc. Alternatively, the electronic device 20 can also be the hardware operated by the system for controlling the vehicle's central control on the vehicle, such as a central processing unit (CPU), a general-purpose processor network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD) or any combination thereof. The electronic device 20 can also be other devices with processing functions, such as circuits, devices or software modules. The embodiments of the present application do not limit the specific form of the electronic device 20.

[0071] In some embodiments, multiple surround view cameras 10 and electronic devices 20 are installed on a vehicle. For example, the surround view cameras 10 are vehicle-mounted cameras with different shooting angles on the vehicle, and the electronic device 20 is a vehicle-mounted terminal on the vehicle.

[0072] Optionally, the electronic device 20 is not installed on the mobile target, and the surround-view camera 10 transmits the captured images to the electronic device 20. After the electronic device 20 splices the images captured by each surround-view camera 10, it transmits the obtained surround-view bird's-eye view to a display device for displaying the scene around the vehicle.

[0073] In some embodiments, the surround view camera 10 and the electronic device 20 may be configured as follows: Figure 1 The devices shown are independent of each other, or the electronic device 20 can also be integrated with any surround-view camera 10 on the vehicle, and the embodiments of the present application do not impose specific limitations on this.

[0074] The hardware structure of the electronic device 20 includes: Figure 2 The computing device shown in FIG. Figure 2 Taking the computing device shown as an example, the hardware structure of the electronic device 20 is introduced.

[0075] like Figure 2 As shown, the computing device may include a processor 301, a memory 302, a communication interface 303, and a bus 304. The processor 301, the memory 302, and the communication interface 303 may be connected via a bus 304.

[0076] The processor 301 is the control center of the computing device, which can be a processor or a general term for multiple processing elements. For example, the processor 301 can be a general-purpose central processing unit (CPU) or other general-purpose processors. Among them, the general-purpose processor can be a microprocessor or any conventional processor.

[0077] As an embodiment, the processor 301 may include one or more CPUs, such as Figure 2 CPU 0 and CPU 1 are shown in .

[0078] The memory 302 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0079] In a possible implementation, the memory 302 may exist independently of the processor 301, and the memory 302 may be connected to the processor 301 via a bus 304 to store instructions or program codes. When the processor 301 calls and executes the instructions or program codes stored in the memory 302, the model deployment method provided in the embodiment of the present application can be implemented.

[0080] In another possible implementation, the memory 302 may also be integrated with the processor 301 .

[0081] The communication interface 303 is used for connecting the computing device to other devices through a communication network, which may be Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. The communication interface 303 may include a receiving unit for receiving data and a sending unit for sending data.

[0082] The bus 304 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0083] It should be pointed out that Figure 2 The structure shown in the figure does not constitute a limitation on the computing device, except Figure 2In addition to the components shown, the computing device may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0084] The implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0085] like Figure 3 As shown, the embodiment of the present application provides an image generation method, which can be performed by the above electronic device 20, and the method includes the following steps:

[0086] S101. Acquire a current scene image around the vehicle captured by each of a plurality of surround view cameras deployed on the vehicle.

[0087] Among them, the shooting range of all surround-view cameras deployed on the vehicle covers 360° around the vehicle.

[0088] In some embodiments, when the surround view camera is a fisheye camera, the original image captured by the fisheye camera is distorted, for example Figure 4 The original image frame taken by the fisheye camera shown cannot be directly used for stitching to obtain a panoramic bird's-eye view. Therefore, it is necessary to perform distortion correction on the original image taken by the fisheye camera to obtain the current scene image.

[0089] Specifically, the electronic device can perform distortion correction on the original image taken by the fisheye camera through the intrinsic parameters and extrinsic parameters of the fisheye camera.

[0090] When the vehicle leaves the factory, the internal and external parameters of the fisheye camera can be obtained by calibrating the fisheye camera, and the internal and external parameters of the fisheye camera are stored and called in the process of obtaining the surround view and bird's-eye view. The internal parameters of the fisheye camera include distortion parameters k1, k2, k3, k4, center point parameter C x and C y , and the focal length parameter f x , f y . In practical applications, Zhang Zhengyou's checkerboard calibration scheme is usually used. By making a calibration plate, using the images taken by the fisheye camera and obtaining the corner coordinates of the calibration plate, the intrinsic parameters of the fisheye camera are calculated by minimizing the projection error using the feature points on multiple images taken by the fisheye camera. The extrinsic parameters of the fisheye camera generally include the camera position parameters (x, y, z) and the camera's pose angle parameters (yaw, pitch, roll). A joint calibration scheme is usually used to move the vehicle in a specific venue. The corner extraction algorithm is used to perform a preliminary calibration of the extrinsic parameters of the fisheye camera. Then, the optical flow fine-tuning method is used to perform a fusion calibration of the extrinsic parameters to obtain the optimal value of the extrinsic parameters with the minimum reprojection error.

[0091] If the surround-view camera is not a fisheye camera, the electronic device may also use other distortion correction methods to perform distortion correction on the original image taken by the surround-view camera, and this application does not limit this.

[0092] S102. For each surround-view camera, coordinate mapping is performed on the current scene image of the surround-view camera from the camera coordinate system to the imaging plane in the world coordinate system to obtain a panoramic sub-image of the surround-view camera; during the mapping process, based on the pixel value of the first pixel point in the current scene image of the adjacent surround-view camera, pixel correction is performed on the pixel value of the first pixel point in the current scene image of the surround-view camera to obtain the pixel value of the first pixel point in the panoramic sub-image of the surround-view camera.

[0093] Among them, the first pixel point is the pixel point corresponding to the overlapping area of ​​the shooting range of the surround camera and the adjacent surround camera. For example, within a certain shooting range of the vehicle, both the left-view camera and the front-view camera can shoot this range. This shooting range is the overlapping area between the shooting ranges of the left-view camera and the front-view camera. If there is a spatial point A in this shooting range, it corresponds to pixel point B in the current scene image shot by the left-view camera and corresponds to pixel point C in the current scene image shot by the front-view camera, then pixel point B and pixel point C are the first pixel points representing the same spatial position. The origin of the world coordinate system in this application is the vehicle itself. When the position of the vehicle changes, the origin of the world coordinate system also changes with the position of the vehicle. For example, the vehicle is at the first position at the first moment, then the origin of the world coordinate system at the first moment is the first position, and the vehicle is at the second position at the second moment, then the origin of the world coordinate system at the second moment is the second position. The imaging plane in the world coordinate system represents the display angle of the vehicle. For example, what is finally obtained in this application is a bird's-eye view of the vehicle, then the display angle of the vehicle is the angle of looking down on the ground from the sky, and the imaging plane in the world coordinate system is the ground.

[0094] The essence of a surround view bird's-eye view map is to present the three-dimensional spatial information around the vehicle in the form of a two-dimensional image. To achieve this, it is necessary to map the pixel points in the two-dimensional image captured by the surround view camera back to the real world coordinates in order to accurately reconstruct the spatial environment around the vehicle. Therefore, the current scene image of the surround view camera must be mapped from the camera coordinate system to the imaging plane in the world coordinate system to obtain the panoramic sub-image of the surround view camera. In addition, the camera coordinate system uses the camera as the coordinate origin. The application scenario of this application involves multiple surround view cameras, and the coordinate origin of each surround view camera is different. Therefore, the camera coordinate system corresponding to each surround view camera is also different. Pixels in different camera coordinate systems cannot be compared to determine whether they represent the same spatial position. Only by converting the camera coordinate system of each surround view camera to a unified world coordinate system can the correspondence between different pixels in the actual space be determined. When generating a surround-view bird's-eye view image based on the panoramic sub-images of each surround-view camera, the panoramic sub-images of multiple surround-view cameras need to be stitched together. In order to ensure the consistency and accuracy of the stitched image, the pixel points in the current scene image taken by each surround-view camera need to be converted to a unified world coordinate system to improve the stitching quality.

[0095] Taking the surround view camera as a fisheye camera as an example, the coordinate mapping of the current scene image of the surround view camera from the camera coordinate system to the imaging plane in the world coordinate system is performed. The specific conversion formula between the pixel coordinates of the pixel points in the current scene image and the actual world coordinates corresponding to the pixel points is as follows:

[0096]

[0097] in:

[0098]

[0099] T is the position parameter (x, y, z) of the fisheye camera, R is the pose angle parameter (yaw, pitch, roll) of the fisheye camera, Z c is the length of the optical axis of the fisheye camera, (u, v) is the pixel coordinate of the pixel point in the current scene image of the fisheye camera, (X w , Y w , Z w ) is the spatial coordinate of the actual spatial position represented by the pixel point in the world coordinate system.

[0100] Therefore, the mapping process between the pixel points corresponding to the same spatial position in different scene images and the world coordinates can be expressed by the formula:

[0101]

[0102] Since the application ultimately obtains a bird's-eye view of the vehicle, the bird's-eye view is an image of the vehicle from the sky. Therefore, the altitude of the pixel points in the world coordinates does not need to be paid attention to in the bird's-eye view. That is, the current scene image is projected onto the ground. Then Z w Taking 0, the above conversion formula can be rewritten as:

[0103]

[0104] Where H is the homography matrix, That is to say h 11 、h 12 、h 13 and h 21 、h 22 、h 23 Used to represent the 3 changes from the source image to the target image, including image rotation, scaling and translation. 31 and h 32 Used to represent the perspective transformation from the source image to the target image. They control the direction and depth of the projection. 33 Used for normalization operations, usually set to 3 to ensure that the determinant of the matrix is ​​not zero.

[0105] The distortion calibration calculation of the fisheye camera can be performed through the above internal parameters of the fisheye camera. The specific calculation formula is as follows:

[0106]

[0107] θ i ≈k0θ 1 +k1θ 3 +k2θ 5 +k3θ 7 +k4θ 9 ;

[0108]

[0109] Where k0 = 1; r d Represents the distance between the point in the distortion calibration image and the center of the image circle; r e Represents the distance between the point in the fisheye calibration image and the center of the image circle; θ i represents the angle between the vector from the point to the optical center and the plane in the distortion calibration image; θ represents the angle between the vector from the point to the optical center and the plane in the original image taken by the fisheye camera; (u eye , v eye ) represents the pixel coordinates of the midpoint of the original image taken by the fisheye camera.

[0110] For example, Figure 5The shaded area in the bird's-eye view image is the location of the vehicle. The pixel coordinates of the same spatial location in different images are different, for example Figure 5 The star-shaped pattern represents the same spatial position, and the pixel coordinates of the star-shaped pattern in different images are different.

[0111] Based on the above process, for each surround-view camera, the coordinates of the imaging plane in the current scene image of the surround-view camera can be mapped from the pixel coordinate system to the world coordinate system based on the intrinsic parameters and extrinsic parameters of the surround-view camera, so as to obtain the world coordinates corresponding to the pixel coordinates of the pixel points in the current scene image of the surround-view camera.

[0112] For example, Figure 6 As shown in FIG. 1 , the original image taken by the fisheye camera is subjected to distortion correction and coordinate mapping to obtain a panoramic sub-image.

[0113] In some embodiments, in consideration of the display range of the surround bird's-eye view, the electronic device can crop the panoramic sub-image. The specific cropping methods can be: 1. Directly perform a coordinate mapping process on the current scene image to obtain an initial panoramic sub-image, and crop the initial panoramic sub-image according to the range to be displayed to obtain a panoramic sub-image; 2. In the process of coordinate mapping the current scene image, only the mapping results of the position points within a preset distance from the vehicle are retained to obtain a panoramic sub-image.

[0114] In some embodiments, the electronic device performs pixel correction on the pixel value of the first pixel in the current scene image of the surround view camera based on the pixel value of the first pixel in the current scene image of the adjacent surround view camera to obtain the pixel value of the first pixel in the panoramic sub-image of the surround view camera. This can be specifically implemented as follows: the electronic device performs weighted fusion of the pixel value of the first pixel in the current scene image of the surround view camera and the pixel value of the first pixel in the current scene image of the adjacent surround view camera to obtain the fused pixel value; the electronic device adds the product of the historical gain value and the gain coefficient to the fused pixel value to obtain the pixel value of the first pixel in the panoramic sub-image of the surround view camera. The historical gain value is used to characterize the degree of difference in the shooting results of the surround view camera and the adjacent surround view camera for the spatial position corresponding to the first pixel in the historical time period.

[0115] The gain coefficient is determined according to the absolute value of the difference between the first fusion weight and the second fusion weight, the first fusion weight being the weight used when the pixel value of the first pixel point in the current scene image of the surround view camera participates in the weighted fusion calculation, and the second fusion weight being the weight used when the pixel value of the first pixel point in the current scene image of the adjacent surround view camera participates in the weighted fusion calculation.

[0116] Specifically, the fused pixel value of the first pixel in the panoramic sub-image of the surround view camera satisfies the following relationship with the first fusion weight, the second fusion weight, the pixel value of the first pixel in the current scene image taken by the surround view camera, and the pixel value of the first pixel in the current scene image taken by the adjacent surround view camera:

[0117]

[0118] Among them, Pix n It is used to represent the fused pixel value of the nth pixel in the panoramic sub-image of the surround camera, w1 is used to represent the first fusion weight, w2 is used to represent the second fusion weight, PixCur n PixAdj is used to represent the pixel value of the nth first pixel in the current scene image captured by the surround camera. n Used to indicate that PixAdj n It is used to represent the pixel value of the nth first pixel in the current scene image captured by the adjacent surround view camera, and a is used to represent the gain coefficient, PixGain n Used to indicate the historical gain value corresponding to the nth first pixel.

[0119] In particular, when there is no pixel in the current scene image of the surround view camera that represents the same spatial position as the pixel in the current scene image of the adjacent surround view camera, the second fusion weight corresponding to the current scene image of the surround view camera should be 0. In this way, when the gain coefficient is not 0, if the pixel in the non-overlapping area (there is no pixel in the current scene image of the adjacent surround view camera that represents the same spatial position as the pixel) is corrected, the historical gain of the historical scene image is taken into account, and the panoramic sub-image obtained by the current scene image has a natural transition with the panoramic sub-image corresponding to the historical scene image, then the final spliced ​​surround view image also has a natural transition with the previous surround view image, and the display effect is better.

[0120] When the absolute value of the difference between the first fusion weight and the second fusion weight is greater than the preset threshold, the gain coefficient is 0; or, when the absolute value of the difference between the first fusion weight and the second fusion weight is less than or equal to the preset threshold, the gain coefficient is 1. The typical value of the preset threshold is 0.5. When the preset threshold is 0.5, the relationship between the gain coefficient and the first fusion weight, the second fusion weight, and the preset threshold can be expressed by the formula:

[0121]

[0122] When the absolute value of the difference between the first fusion weight and the second fusion weight is greater than the preset threshold, it means that in the process of the panoramic sub-image of the surround-view camera, the current scene image of the surround-view camera or the color space of the current scene image of the adjacent surround-view camera is mainly considered, and considering the historical gain of the historical scene image will not bring better display effect. When the absolute value of the difference between the first fusion weight and the second fusion weight is less than or equal to the preset threshold, it means that in the process of the panoramic sub-image of the surround-view camera, the color space of the current scene image of the adjacent surround-view camera also occupies a large part of the consideration. At this time, the historical gain of the historical scene image can be taken into consideration, so that the panoramic sub-image corresponding to the current scene image and the panoramic sub-image of the previous frame have a natural transition. Therefore, by comparing the first fusion weight and the second fusion weight to determine the gain coefficient, in order to decide whether the historical gain of the historical scene image needs to be considered, different scenes can be adapted to obtain better display effect.

[0123] In some embodiments, the electronic device performs weighted fusion of the pixel value of the first pixel point in the current scene image of the surround-view camera and the pixel value of the first pixel point in the current scene image of the adjacent surround-view camera to obtain a fused pixel value, which can be specifically implemented as follows: the electronic device obtains multiple pixel values ​​of the second pixel point in multiple historical scene images taken by the surround-view camera, and multiple pixel values ​​of the second pixel point in multiple historical scene images taken by other surround-view cameras; the second pixel point is a pixel point in the historical scene image corresponding to the same spatial position as the first pixel point; the electronic device calculates a first variance based on the multiple pixel values ​​of the second pixel point in the multiple historical scene images taken by the surround-view camera value, calculates a second variance value based on multiple pixel values ​​of the second pixel point in multiple historical scene images taken by other surround-view cameras; the electronic device corrects the first fusion weight and the second fusion weight used by the surround-view camera at the previous moment based on the size relationship between the first variance value and the second variance value, and obtains the first fusion weight and the second fusion weight used by the surround-view camera at the current moment; the electronic device weightedly fuses the pixel value of the first pixel point in the current scene image of the surround-view camera with the pixel value of the first pixel point in the current scene image of the adjacent surround-view camera based on the first fusion weight and the second fusion weight used by the surround-view camera at the current moment, and obtains the fused pixel value.

[0124] Specifically, the first variance value and multiple pixel values ​​of the second pixel point in multiple historical scene images captured by the surround camera satisfy the following relationship:

[0125]

[0126] in, Used to represent the first variance value, PixCur i It is used to represent the pixel value of the second pixel of the i-th pixel in the current scene image captured by the surround camera. It is used to represent the average value of multiple pixel values ​​of the i-th second pixel point in multiple historical scene images taken by the surround-view camera, and M is used to represent the number of second pixel points in the current scene image taken by the surround-view camera.

[0127] The second variance value and multiple pixel values ​​of the second pixel point in multiple historical scene images taken by other surround view cameras satisfy the following relationship:

[0128]

[0129] in, Used to represent the second variance value, PixAdj j It is used to represent the pixel value of the jth second pixel in the current scene image captured by the adjacent surround view camera. It is used to represent the average value of multiple pixel values ​​of the j-th second pixel point in multiple historical scene images taken by other surround view cameras, and N is used to represent the number of second pixel points in the current scene image taken by adjacent surround view cameras.

[0130] Based on the above description, the historical gain value satisfies the following relationship with multiple pixel values ​​of the second pixel in multiple historical scene images captured by the surround view camera and multiple pixel values ​​of the second pixel in multiple historical scene images captured by other surround view cameras:

[0131]

[0132] Among them, PixGain m Used to represent the historical gain value of the mth second pixel point, It is used to represent the average value of multiple pixel values ​​of the m-th second pixel in multiple historical scene images taken by the surround camera. Used to represent the average value of multiple pixel values ​​of the m-th second pixel point in multiple historical scene images taken by other surround view cameras.

[0133] In some embodiments, the electronic device corrects the first fusion weight and the second fusion weight used by the surround-view camera at the last moment based on the size relationship between the first variance value and the second variance value to obtain the first fusion weight and the second fusion weight used by the surround-view camera at the current moment. This can be specifically implemented as follows: when the first variance value is less than the second variance value, the electronic device increases the first fusion weight used by the surround-view camera at the last moment by a preset value to obtain the first fusion weight used by the surround-view camera at the current moment, and decreases the second fusion weight used by the surround-view camera at the last moment by a preset value to obtain the second fusion weight used by the surround-view camera at the current moment; in the first When one variance value is greater than the second variance value, the electronic device reduces the first fusion weight used by the surround-view camera at the last moment by a preset value to obtain the first fusion weight used by the surround-view camera at the current moment, and increases the second fusion weight used by the surround-view camera at the last moment by a preset value to obtain the second fusion weight used by the surround-view camera at the current moment; when the first variance value is equal to the second variance value, the electronic device uses the first fusion weight used by the surround-view camera at the last moment as the first fusion weight used by the surround-view camera at the current moment, and uses the second fusion weight used by the surround-view camera at the last moment as the second fusion weight used by the surround-view camera at the current moment. Among them, the typical value of the preset value can be 0.01, and the last moment refers to the shooting moment of the scene image adjacent to the current scene image before the current moment.

[0134] The relationship between the first fusion weight and the second fusion weight used by the surround view camera at the current moment, and the first fusion weight and the second fusion weight, the preset value, the first variance value, and the second variance value used by the surround view camera at the previous moment can be expressed by the formula:

[0135]

[0136] Among them, w1 ′ It is used to represent the first fusion weight adopted by the surround-view camera at the current moment, w2′ is used to represent the second fusion weight adopted by the surround-view camera at the current moment, w1 is used to represent the first fusion weight adopted by the surround-view camera at the last moment, and w2 is used to represent the second fusion weight adopted by the surround-view camera at the last moment.

[0137] The first fusion weight and the second fusion weight used in each frame of the image are constantly changing, and can better adapt to the changes in the scene during the vehicle's driving process, so as to perform pixel correction on the current scene image of the surround view camera based on the first fusion weight and the second fusion weight to obtain a panoramic sub-image with better display effect.

[0138] In some embodiments, the first fusion weight and the second fusion weight in different scenes have corresponding initial values. When a vehicle drives into a new scene, the electronic device determines the first fusion weight and the second fusion weight used when converting the current scene image into a panoramic sub-image as the initial values ​​corresponding to the new scene. In the process of determining the panoramic sub-image corresponding to the next frame of the current scene image, the electronic device obtains the first fusion weight and the second fusion weight used when converting a new frame of the current scene image into a panoramic sub-image based on the initial values ​​corresponding to the first fusion weight and the second fusion weight.

[0139] For example, when a vehicle enters a scene with cluttered colors, such as a scene with grass, the colors in the 360° surroundings of the vehicle are different and the color changes are large, then the electronic device can determine that the initial value of the first fusion weight in such a scene is the first preset value, and the initial value of the second fusion weight is the first preset value. The typical value of the first preset value is 0.5.

[0140] Since the color changes in the scene around the vehicle are large, by setting the initial value of the first fusion weight equal to the initial value of the second fusion weight, the corresponding gain coefficient is 1. The color space of the final panoramic sub-image is more balanced with the color space of the images taken by the adjacent surround-view cameras, and the transition with the color space of the historical images is more natural. In this way, the display effect of the images originally taken by the surround-view camera itself is retained, and the display effect of the images taken by the adjacent surround-view cameras is taken into account. The gain effect of the historical images is also taken into account. The stitching display effect of the surround-view bird's-eye view map obtained by stitching the panoramic sub-images is more natural, the transition between frames is natural, and the color space between regions in the surround-view bird's-eye view map is more balanced.

[0141] In another example, when the scene represented by the current scene image is a scene of sudden brightness change, for example, a vehicle driving from a dark environment to a bright environment (for example, a vehicle driving out of a tunnel, etc.), a vehicle driving from a bright environment to a dark environment (for example, a vehicle driving into a tunnel, a vehicle driving into an underground garage, etc.), a vehicle driving in a dark environment encountering illumination from the lights of surrounding vehicles, and other scenes of sudden brightness change, the electronic device determines that the initial value of the first fusion weight is the second preset value, determines that the initial value of the second fusion weight is the third preset value, and determines that the gain coefficient is 0. Among them, the second preset value is greater than the third preset value, the typical value of the second preset value is 0.8, and the typical value of the third preset value is 0.2.

[0142] Since the current scene image represents a brightness mutation scene, the previous historical scene images are no longer of reference value. By setting the gain coefficient to 0, the influence of the historical environment can be ignored, and the color space changes of the current scene image in the brightness mutation scene can be better adapted to represent the flexibility of color space adjustment of the current scene image.

[0143] When a vehicle enters a scene with a sudden change in brightness, the brightness, contrast, and chromaticity in the color space of the current scene images captured by different surround view cameras will be quite different, which is most obvious as one image is too bright and the other is too dark. The display effect of the surround view bird's-eye view image obtained by stitching is not good. By setting the initial value of the first fusion weight greater than the second fusion weight in the scene with a sudden change in brightness, the panoramic sub-image of the surround view camera mainly considers the color space of its corresponding current scene image, and slightly considers the color space of the current scene image captured by the adjacent surround view camera, that is, it retains its original display effect, and can also balance the overly bright and dark images, so that the color space of the final stitched surround view bird's-eye view image is more balanced and the display effect is better.

[0144] Based on the above description, the process of determining the panoramic sub-image of the present application is introduced from the perspective of the overall process as follows.

[0145] For example, if a vehicle is equipped with four surround view cameras (a front view camera, a left view camera, a right view camera, and a rear view camera), the pixel correction process of the scene image captured by each surround view camera is as follows:

[0146] In the process of determining the panoramic sub-image corresponding to the current scene image taken by the front-view camera, obtain the X frames of historical scene images before the current scene image is taken by the front-view camera, and for each pixel point A in the current scene image of the front-view camera, find the pixel point representing the same spatial position as the pixel point A in the X frames of historical scene images, and find the average value of the pixel values ​​of the multiple pixel points representing the same spatial position as the pixel point A found in the X frames of historical scene images (it is not necessarily possible to find X pixels here, for example, there are 2 frames of historical scene images before the X frames of historical scene images that do not find the pixel point representing the same spatial position as the pixel point A, and the pixel points representing the same spatial position as the pixel point A are found in the other frames of historical scene images, then the average value of the pixel values ​​of (X-2) pixels should be found here), and then find the difference between each pixel point in the current scene image of the front-view camera and the corresponding average value, and finally obtain the variance value 1 corresponding to the front-view camera based on each difference. Wherein, X is a positive integer.

[0147] The shooting range of the front camera overlaps with that of the left camera. The left camera is an adjacent surround camera of the front camera. Then, the X frames of historical scene images before the left camera, the right camera and the rear camera are obtained. For each pixel point B in the current scene image of the left camera, the pixel point representing the same spatial position as the pixel point B is searched in the X frames of historical scene images of the three surround cameras. The average pixel value of the multiple pixel points representing the same spatial position as the pixel point B found in the 3X frames of historical scene images is calculated (similarly, 3X pixels may not be found here). Then, the difference between each pixel point in the current scene image of the left camera and its corresponding average value is calculated. Finally, the variance value 2 corresponding to the left camera is obtained based on each difference.

[0148] The shooting range of the front-view camera overlaps with the shooting range of the right-view camera. The right-view camera is an adjacent surround-view camera of the front-view camera. Then, the X frames of historical scene images before the left-view camera, the right-view camera and the rear-view camera are obtained. For each pixel point in the current scene image of the right-view camera, the pixel point representing the same spatial position as the pixel point C is searched in the X frames of historical scene images of the three surround-view cameras, and the average pixel value of the multiple pixel points representing the same spatial position as the pixel point C found in the 3X frames of historical scene images is calculated (similarly, 3X pixels may not be found here), and then the difference between each pixel point in the current scene image of the right-view camera and its corresponding average value is calculated, and finally the variance value 3 corresponding to the right-view camera is obtained based on each difference.

[0149] Finally, the current scene image of the front camera is pixel-corrected. If a pixel D1 in the current scene image of the front camera has a pixel D2 representing the same spatial position in the current scene image taken by the left camera, then the pixel value of the pixel D0 corresponding to the pixel D1 in the panoramic sub-image of the front camera is At this time, the first fusion weight and the second fusion weight are determined based on the variance value 1, the variance value 2, and the first fusion weight and the second fusion weight used when determining the panoramic sub-image of the front-view camera at the last moment. If a pixel point E1 in the current scene image of the front-view camera has a pixel point E2 representing the same spatial position in the current scene image taken by the right-view camera, then the pixel value of the pixel point E0 corresponding to the pixel point E1 in the panoramic sub-image of the front-view camera is At this time, the first fusion weight and the second fusion weight are determined based on the variance value 1, the variance value 3, and the first fusion weight and the second fusion weight used when the panoramic sub-image of the front-view camera was determined at the last moment.

[0150] Similarly, in the process of determining the panoramic sub-image of the left-view camera, if the front-view camera and the rear-view camera overlap with the shooting range of the left-view camera, the previous X frames of historical scene images of the left-view camera must be obtained to determine the variance value corresponding to the left-view camera. The previous X frames of historical scene images of the front-view camera, the rear-view camera and the right-view camera must be obtained to determine the variance values ​​corresponding to the front-view camera, the rear-view camera and the right-view camera. Finally, based on the variance values ​​corresponding to the left-view camera and the variance values ​​corresponding to the front-view camera, the pixel values ​​of the pixels in the current scene image taken by the left-view camera (there are pixels representing the same spatial position as the pixel in the current scene image taken by the front-view camera) are pixel-corrected. Based on the variance values ​​corresponding to the left-view camera and the variance values ​​corresponding to the rear-view camera, the pixel values ​​of the pixels in the current scene image taken by the left-view camera (there are pixels representing the same spatial position as the pixel in the current scene image taken by the rear-view camera) are pixel-corrected, and finally the panoramic sub-image of the left-view camera is obtained.

[0151] The process of determining the panoramic sub-image of the right-view camera and the panoramic sub-image of the rear-view camera is logically the same as the above process, and will not be repeated here.

[0152] In some embodiments, the electronic device obtains multiple pixel values ​​of a second pixel point in multiple historical scene images taken by a surround-view camera, as well as multiple pixel values ​​of the second pixel point in multiple historical scene images taken by other surround-view cameras. This can be specifically implemented as follows: the electronic device obtains vehicle displacement information corresponding to each frame of historical scene image, and the vehicle displacement information corresponding to the historical scene image is the vehicle displacement information from the shooting time of the historical scene image to the shooting time of the current scene image; for each frame of historical scene image taken by the surround-view camera, the electronic device determines a first corresponding pixel point of the third pixel point from the historical scene image based on the vehicle displacement information corresponding to the historical scene image and the pixel coordinates of the third pixel point on the current scene image taken by the surround-view camera; the electronic device uses the first corresponding pixel point and the corresponding third pixel point that meet the preset conditions as the second pixel point representing the same spatial position, and uses the first corresponding pixel point and the corresponding third pixel point that do not meet the preset conditions as the pixel points in the three-dimensional obstacle, and determines that the pixel points in the three-dimensional obstacle are not belongs to the second pixel point; for each frame of historical scene image taken by other surround-view cameras, the electronic device determines the second corresponding pixel point of the fourth pixel point from the historical scene image based on the vehicle displacement information corresponding to the historical scene image and the pixel coordinates of the fourth pixel point on the current scene image taken by the adjacent surround-view camera; the electronic device uses the second corresponding pixel point and the corresponding fourth pixel point that meet the preset conditions as the second pixel point representing the same spatial position, and uses the second corresponding pixel point and the corresponding fourth pixel point that do not meet the preset conditions as the pixel points in the three-dimensional obstacle, and determines that the pixel point in the three-dimensional obstacle does not belong to the second pixel point; wherein the preset conditions include: the Euclidean distance corresponding to the corresponding pixel point satisfies the distance threshold condition, and the Euclidean distance corresponding to at least one adjacent pixel point of the corresponding pixel point satisfies the distance threshold condition, the Euclidean distance corresponding to the corresponding pixel point is used to represent the Euclidean distance between the corresponding pixel point and the corresponding pixel point in the current scene image in the color space, and the distance threshold condition is used to measure the color similarity between the two pixel points.

[0153] Since the surround view camera is fixedly mounted on the vehicle, when the vehicle moves, it can also be regarded as the movement of the surround view camera. Therefore, the displacement between the world coordinates corresponding to two pixel points representing the same spatial position in two images taken by the same surround view camera should be opposite to the displacement direction of the vehicle in the shooting interval of the two images and equal in distance.

[0154] Exemplarily, after obtaining X frames of historical scene images of the vehicle's forward-looking camera, taking the pixel point A in the current scene image of the forward-looking camera as an example, searching for the pixel point representing the same spatial position in the historical scene image 1, the displacement of the vehicle from the shooting time of the historical scene image 1 to the shooting time of the current scene image is x1, and the pixel coordinates of the pixel point A are known. Then, based on the conversion relationship between the aforementioned pixel coordinate system and the world coordinate system with the vehicle body as the origin, the world coordinates of the pixel point A can be obtained. If the pixel point in the historical scene image 1 that represents the same spatial position as the pixel point A is the pixel point B, then the displacement between the world coordinates of the pixel point A and the world coordinates of the pixel point B should be equal in distance and opposite in direction to x1.

[0155] Therefore, based on the vehicle displacement and the world coordinates of the pixel points in the current scene image, it is possible to reversely calculate the pixel points in the historical scene image that represent the same spatial position as the pixel points in the current scene image.

[0156] like Figure 7 As shown in the figure, there are often various 3D obstacles in the driving environment of the vehicle. Since the colors of the 3D obstacles captured by the surround camera during the driving process are different at different angles, and the colors of the 3D obstacles in different scene images also change with the movement of the vehicle, the color difference between the two pixels representing the same spatial position in the two images will be large, resulting in poor pixel correction effect of the panoramic sub-image. Therefore, the pixels representing the 3D obstacles in the scene image should be removed without pixel correction process.

[0157] For example, in the process of searching for a pixel point representing the same spatial position as a pixel point in the current scene image from the historical scene image, if the conversion relationship between the pixel coordinates (u, v) of the pixel point A on the current scene image captured by the surround view camera at a certain moment and the world coordinates of its corresponding spatial position is as follows:

[0158]

[0159] When the vehicle moves, the conversion relationship between the pixel coordinates (u1, v1) of pixel point B representing the same spatial position as pixel point A in the newly captured scene image of the surround view camera and the world coordinates of its corresponding spatial position is as follows:

[0160]

[0161] Among them, Δx is the displacement of the vehicle on the X axis, and Δy is the displacement of the vehicle on the Y axis.

[0162] Then the Euclidean distance between pixel A and pixel B is:

[0163]

[0164] The Euclidean distance between pixel A and pixel B represents the similarity between the two pixels in color space. Since pixel A and pixel B represent the same spatial position, when the Euclidean distance between the two pixels is too large, it means that the color similarity between the two pixels is small, and this spatial position may be a point on a three-dimensional obstacle. In addition, by judging whether the spatial position represented by other pixels around the pixel is a point on a three-dimensional obstacle, it is possible to more accurately judge whether the spatial position represented by two pixels with low color similarity is a point on a three-dimensional obstacle, avoiding misjudgment caused by three-dimensional obstacle judgment based on the Euclidean distance between two pixels corresponding to a single spatial position.

[0165] In some embodiments, the electronic device obtains the vehicle displacement information corresponding to each frame of the historical scene image, which can be specifically implemented as follows: for each frame of the historical scene image, the electronic device obtains the shooting time interval, the vehicle's moving speed within the shooting time interval, and the steering angle; the shooting time interval is the time interval from the shooting moment of the historical scene image to the shooting moment of the current scene image; the electronic device determines the vehicle displacement information corresponding to the historical scene image based on the shooting time interval, the vehicle's moving speed within the shooting time interval, and the steering angle.

[0166] Specifically, the vehicle displacement information satisfies the following relationship with the shooting time interval, the moving speed of the vehicle within the shooting time interval, and the steering angle:

[0167] Δx=cos(θ a )VΔt;

[0168] Δy=sin(θ a )VΔt;

[0169] Wherein, Δx is the displacement of the vehicle in the X-axis direction in the world coordinate system with the vehicle body as the origin in the vehicle displacement information, Δy is the displacement of the vehicle in the Y-axis direction in the world coordinate system with the vehicle body as the origin in the vehicle displacement information, and θ a is the steering angle of the vehicle during the shooting time interval, Δt is the shooting time interval, and V is the moving speed of the vehicle during the shooting time interval.

[0170] For example, during the driving process of the vehicle, a measuring device for measuring vehicle driving related parameters is installed on the vehicle, which can detect the vehicle driving speed V and the steering angle θ of the vehicle. a, the first position is the position of the vehicle when the first historical scene image is generated, and the second position is the position of the vehicle when the first perspective image is generated. The speed of the vehicle in the positive direction can be set as positive, and the right direction of the turn can be set as negative. The initial position of the vehicle is recorded as (x1, y1), and the position of the vehicle after moving is recorded as (x2, y2), then x2 = x1 + Δx, y2 = y1 + Δy, Δx = cos(θ a )VΔt,Δy=sin(θ a )VΔt,Δx=cos(θ a )VΔt,Δy=sin(θ a )VΔt. When the vehicle is traveling in a straight line, the vehicle travels as follows Figure 8 As shown, x2 = x1 + cos(θ a )VΔt,y2=y1+sin(θ a )VΔt. When the vehicle is in the process of turning, the vehicle driving process is as follows Fig. 9 As shown, x2 = x1 + R × (cos (θ a1 )-cos(θ a )), y2=y1+R×(sin(θ a1 )-sin(θ a )),θ a represents the steering angle of the vehicle in the first position, θ a1 Indicates the steering angle of the vehicle in the second position.

[0171] Based on the above process, the vehicle displacement information corresponding to each frame of the historical scene image can be obtained. When the vehicle moves, the points in the actual space can be considered to be moving in the opposite direction relative to the vehicle. The displacement of the two pixel points representing the same spatial position in the two frames of images directly in the world coordinate system with the vehicle body as the origin should match the vehicle displacement information corresponding to the two frames of images. Then, an inverse mapping can be performed based on the historical displacement information to determine the pixel points in the current scene image corresponding to the pixel points in the historical scene image, which is convenient for calculating the aforementioned first variance value, second variance value, historical gain value and other values, thereby realizing pixel correction of the current scene image of the surround view camera.

[0172] S103: Perform pixel fusion on the pixel values ​​of the pixels of the panoramic sub-images in the overlapping area.

[0173] For example, Fig.10As shown, areas 1-9 are areas divided by the surround view bird's-eye view map corresponding to the vehicle with surround view cameras with four shooting angles, and area 5 is the area where the vehicle is located. Areas 1-3 are areas where the panoramic sub-image of the vehicle's front view camera is located, that is, areas 1-3 are the visible areas of the vehicle's front view camera in the surround view bird's-eye view map. Areas 1, 4, and 7 are areas where the panoramic sub-image of the vehicle's left view camera is located, that is, areas 1, 4, and 7 are the visible areas of the vehicle's left view camera in the surround view bird's-eye view map; areas 3, 6, and 9 are areas where the panoramic sub-image of the vehicle's right view camera is located, and areas 3, 6, and 9 are the visible areas of the vehicle's rear view camera in the surround view bird's-eye view map; areas 7-9 are areas where the panoramic sub-image of the vehicle's rear view camera is located, and areas 7-9 are the visible areas of the vehicle's rear view camera in the surround view bird's-eye view map. Fig.10 Region 10 is an area not shown in the image taken by the front camera of the vehicle, region 11 is an area not shown in the image taken by the left camera of the vehicle, region 12 is an area not shown in the image taken by the right camera of the vehicle, and region 13 is an area not shown in the image taken by the rear camera of the vehicle. Region 1 is an overlapping area between the panoramic sub-image of the front camera of the vehicle and the panoramic sub-image of the left camera, region 3 is an overlapping area between the panoramic sub-image of the front camera of the vehicle and the panoramic sub-image of the right camera, region 7 is an overlapping area between the panoramic sub-image of the left camera of the vehicle and the panoramic sub-image of the rear camera, and region 9 is an overlapping area between the panoramic sub-image of the right camera of the vehicle and the panoramic sub-image of the rear camera.

[0174] by Fig.10 For example, the pixel value of pixel point A in area 1 of the panoramic view bird's-eye view image is obtained by pixel fusion of the pixel value of pixel point A in the panoramic subimage of the left-view camera and the pixel value of pixel point A in the panoramic subimage of the front-view camera. In actual fusion, a weight can be assigned to each surround-view camera, and the pixels are fused according to the weight. Usually, the same weight is assigned to each surround-view camera. For example, the pixel value of pixel point A in area 1 of the panoramic view bird's-eye view image is obtained by multiplying the pixel value of pixel point A in the panoramic subimage of the left-view camera by Add the pixel value of pixel A in the panoramic sub-image of the front-view camera and multiply it by The weights corresponding to the left-view camera and the front-view camera are

[0175] S104: splicing the overlapping areas after pixel fusion and the non-overlapping areas in the panoramic sub-image to obtain a bird's-eye view of the vehicle.

[0176] For example, assuming that the vehicle is equipped with surround view cameras at the front, back, left, and right sides, each surround view camera is Figure 6 The conversion process shown in FIG. 4 obtains four panoramic sub-images. After stitching the panoramic sub-images corresponding to the four surround-view cameras, the following can be obtained: Fig.11The bird's-eye view shown, Fig.11 The blank area in the middle indicates the vehicle's location.

[0177] Figure 3 The technical solution shown brings at least the following beneficial effects: for the current scene image captured by each surround-view camera of the vehicle, coordinate mapping is performed from the camera coordinate system to the imaging plane in the world coordinate system to obtain the panoramic sub-image corresponding to each current scene image. During the mapping process, each current scene image will be pixel-corrected based on the current scene image captured by the adjacent surround-view camera. Therefore, the panoramic sub-image corresponding to the current scene image refers to the image captured by the adjacent surround-view camera, and the brightness, contrast, chromaticity, etc. of the pixels in each panoramic sub-image are closer, that is, the color space of each panoramic sub-image is closer. In addition, when the panoramic sub-images are spliced, the pixels in the overlapping area are pixel-fused, and the display effect of the overlapping area in the final surround-view bird's-eye view image is close to the display effect of the two adjacent panoramic sub-images, and the transition of each area of ​​the surround-view bird's-eye view image is natural, and the color space of each area in the entire surround-view bird's-eye view image is more balanced, and the problem of excessive color space differences in different areas of the surround-view bird's-eye view image is avoided as much as possible, and the display effect is better.

[0178] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. It should be easy to realize that the technical goals in this field are combined with the units and algorithm steps of each example described in the embodiments disclosed in this article, and the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical goals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0179] like Fig.12 As shown, the embodiment of the present application further provides an image generation device, which is used in the image generation method shown in the above method embodiment. The image generation device 400 includes: an acquisition module 401 and a processing module 402.

[0180] Among them, the acquisition module 401 is used to obtain the current scene image around the vehicle captured by each of the multiple surround view cameras deployed on the vehicle, and the shooting range of all the surround view cameras deployed on the vehicle covers 360° around the vehicle; the processing module 402 is used to, for each surround view camera, perform coordinate mapping of the current scene image of the surround view camera from the camera coordinate system to the imaging plane in the world coordinate system to obtain a panoramic sub-image of the surround view camera; in the mapping process, based on the pixel value of the first pixel point in the current scene image of the adjacent surround view camera, the pixel value of the first pixel point in the current scene image of the surround view camera is pixel corrected to obtain the pixel value of the first pixel point in the panoramic sub-image of the surround view camera; the first pixel point is the pixel point corresponding to the overlapping area of ​​the shooting range of the surround view camera and the adjacent surround view camera; the processing module 402 is also used to perform pixel fusion on the pixel values ​​of the pixel points in the overlapping area of ​​the panoramic sub-image; the processing module 402 is also used to splice the overlapping area after pixel fusion and the non-overlapping area in the panoramic sub-image to obtain a surround bird's-eye view of the vehicle.

[0181] In one possible implementation, the processing module 402 is specifically used to: perform weighted fusion on the pixel value of the first pixel in the current scene image of the surround-view camera and the pixel value of the first pixel in the current scene image of the adjacent surround-view camera to obtain a fused pixel value; add the product of the historical gain value and the gain coefficient to the fused pixel value to obtain the pixel value of the first pixel in the panoramic sub-image of the surround-view camera; wherein the historical gain value is used to characterize the degree of difference in the shooting results of the surround-view camera and the adjacent surround-view camera for the spatial position corresponding to the first pixel within a historical time period.

[0182] In another possible implementation, the gain coefficient is determined based on the absolute value of the difference between the first fusion weight and the second fusion weight, the first fusion weight being the weight used when the pixel value of the first pixel in the current scene image of the surround-view camera participates in the weighted fusion calculation, and the second fusion weight being the weight used when the pixel value of the first pixel in the current scene image of the adjacent surround-view camera participates in the weighted fusion calculation.

[0183] In another possible implementation, when the absolute value of the difference between the first fusion weight and the second fusion weight is greater than a preset threshold, the gain coefficient is 0; or, when the absolute value of the difference between the first fusion weight and the second fusion weight is less than or equal to the preset threshold, the gain coefficient is 1.

[0184] In another possible implementation, the acquisition module 401 is further used to obtain multiple pixel values ​​of the second pixel point in multiple historical scene images taken by the surround-view camera, and multiple pixel values ​​of the second pixel point in multiple historical scene images taken by other surround-view cameras; the second pixel point is a pixel point in the historical scene image that corresponds to the same spatial position as the first pixel point; the processing module 402 is specifically used to: calculate a first variance value based on the multiple pixel values ​​of the second pixel point in the multiple historical scene images taken by the surround-view camera, and calculate a second variance value based on the multiple pixel values ​​of the second pixel point in the multiple historical scene images taken by other surround-view cameras; based on the size relationship between the first variance value and the second variance value, The first fusion weight and the second fusion weight used at the previous moment are corrected to obtain the first fusion weight and the second fusion weight used by the surround-view camera at the current moment; the first fusion weight is the weight used when the pixel value of the first pixel point in the current scene image of the surround-view camera participates in the weighted fusion calculation, and the second fusion weight is the weight used when the pixel value of the first pixel point in the current scene image of the adjacent surround-view camera participates in the weighted fusion calculation; based on the first fusion weight and the second fusion weight used by the surround-view camera at the current moment, the pixel value of the first pixel point in the current scene image of the surround-view camera and the pixel value of the first pixel point in the current scene image of the adjacent surround-view camera are weightedly fused to obtain a fused pixel value.

[0185] In another possible implementation, the processing module 402 is specifically used to: when the first variance value is less than the second variance value, increase the first fusion weight used by the surround-view camera at the last moment by a preset value to obtain the first fusion weight used by the surround-view camera at the current moment, and reduce the second fusion weight used by the surround-view camera at the last moment by a preset value to obtain the second fusion weight used by the surround-view camera at the current moment; when the first variance value is greater than the second variance value, reduce the first fusion weight used by the surround-view camera at the last moment by a preset value to obtain the first fusion weight used by the surround-view camera at the current moment, and increase the second fusion weight used by the surround-view camera at the last moment by a preset value to obtain the second fusion weight used by the surround-view camera at the current moment; when the first variance value is equal to the second variance value, use the first fusion weight used by the surround-view camera at the last moment as the first fusion weight used by the surround-view camera at the current moment, and use the second fusion weight used by the surround-view camera at the last moment as the second fusion weight used by the surround-view camera at the current moment.

[0186] In another possible implementation, the first variance value and multiple pixel values ​​of the second pixel point in multiple historical scene images captured by the surround camera satisfy the following relationship: in, Used to represent the first variance value, PixCur iIt is used to represent the pixel value of the second pixel of the i-th pixel in the current scene image captured by the surround camera. is used to represent the average value of multiple pixel values ​​of the i-th second pixel point in multiple historical scene images taken by the surround view camera, and M is used to represent the number of second pixel points in the current scene image taken by the surround view camera; the second variance value and the multiple pixel values ​​of the second pixel point in multiple historical scene images taken by other surround view cameras satisfy the following relationship: in, Used to represent the second variance value, PixAdj j It is used to represent the pixel value of the jth second pixel in the current scene image captured by the adjacent surround view camera. It is used to represent the average value of the second pixel in the current scene image taken by the adjacent surround view cameras, and N is used to represent the number of the jth second pixel in multiple historical scene images taken by other surround view cameras; the fused pixel value of the first pixel in the panoramic sub-image of the surround view camera and the first fusion weight, the second fusion weight, the pixel value of the first pixel in the current scene image taken by the surround view camera, and the pixel value of the first pixel in the current scene image taken by the adjacent surround view cameras satisfy the following relationship: Among them, Pix n It is used to represent the fused pixel value of the nth pixel in the panoramic sub-image of the surround camera, w1 is used to represent the first fusion weight, w2 is used to represent the second fusion weight, PixCur n PixAdj is used to represent the pixel value of the nth first pixel in the current scene image captured by the surround camera. n Used to indicate that PixAdj n It is used to represent the pixel value of the nth first pixel in the current scene image captured by the adjacent surround view camera, and a is used to represent the gain coefficient, PixGain n It is used to represent the historical gain value corresponding to the nth first pixel point; the historical gain value and the multiple pixel values ​​of the second pixel point in the multiple historical scene images taken by the surround view camera and the multiple pixel values ​​of the second pixel point in the multiple historical scene images taken by other surround view cameras satisfy the following relationship: Among them, PixGain m Used to represent the historical gain value of the mth second pixel point, It is used to represent the average value of multiple pixel values ​​of the m-th second pixel in multiple historical scene images taken by the surround camera. Used to represent the average value of multiple pixel values ​​of the m-th second pixel point in multiple historical scene images taken by other surround view cameras.

[0187] In another possible implementation, the acquisition module 401 is specifically used to obtain the vehicle displacement information corresponding to each frame of the historical scene image, and the vehicle displacement information corresponding to the historical scene image is the vehicle displacement information from the shooting time of the historical scene image to the shooting time of the current scene image; the processing module 402 is also used to: for each frame of the historical scene image shot by the surround-view camera, based on the vehicle displacement information corresponding to the historical scene image, and the pixel coordinates of the third pixel point on the current scene image shot by the surround-view camera, determine the first corresponding pixel point of the third pixel point from the historical scene image; use the first corresponding pixel point and the corresponding third pixel point that meet the preset conditions as the second pixel point representing the same spatial position, use the first corresponding pixel point and the corresponding third pixel point that do not meet the preset conditions as the pixel point in the three-dimensional obstacle, and determine that the pixel point in the three-dimensional obstacle does not belong to the second pixel point; for each frame of the historical scene shot by other surround-view cameras An image is obtained by determining a second corresponding pixel point of the fourth pixel point from the historical scene image based on the vehicle displacement information corresponding to the historical scene image and the pixel coordinates of the fourth pixel point on the current scene image taken by the adjacent surround view camera; the second corresponding pixel point and the corresponding fourth pixel point that meet the preset conditions are used as the second pixel point representing the same spatial position, and the second corresponding pixel point and the corresponding fourth pixel point that do not meet the preset conditions are used as the pixel points in the three-dimensional obstacle, and it is determined that the pixel point in the three-dimensional obstacle does not belong to the second pixel point; the preset conditions include: the Euclidean distance corresponding to the corresponding pixel point satisfies the distance threshold condition, and the Euclidean distance corresponding to at least one adjacent pixel point of the corresponding pixel point satisfies the distance threshold condition, the Euclidean distance corresponding to the corresponding pixel point is used to represent the Euclidean distance between the corresponding pixel point and the pixel point in the current scene image corresponding to it in the color space, and the distance threshold condition is used to measure the color similarity between the two pixel points.

[0188] In another possible implementation, the acquisition module 401 is specifically used to obtain, for each frame of the historical scene image, the shooting time interval, the vehicle's moving speed within the shooting time interval, and the steering angle; the shooting time interval is the time interval from the shooting moment of the historical scene image to the shooting moment of the current scene image; the processing module 402 is also used to determine the vehicle displacement information corresponding to the historical scene image based on the shooting time interval, the vehicle's moving speed within the shooting time interval, and the steering angle.

[0189] In another possible implementation, the vehicle displacement information satisfies the following relationship with the shooting time interval, the moving speed of the vehicle within the shooting time interval, and the steering angle: Δx=cos(θ a )VΔt;Δy=sin(θ a)VΔt; where Δx is the displacement of the vehicle in the X-axis direction in the world coordinate system with the vehicle body as the origin in the vehicle displacement information, Δy is the displacement of the vehicle in the Y-axis direction in the world coordinate system with the vehicle body as the origin in the vehicle displacement information, θ a is the steering angle of the vehicle during the shooting time interval, Δt is the shooting time interval, and V is the moving speed of the vehicle during the shooting time interval.

[0190] In another possible implementation, the processing module 402 is also used for: when the scene represented by the current scene image is a grass scene, determining that the initial value of the first fusion weight is a first preset value, and determining that the initial value of the second fusion weight is the first preset value; when the scene represented by the current scene image is a brightness mutation scene, determining that the initial value of the first fusion weight is a second preset value, determining that the initial value of the second fusion weight is a third preset value, determining that the gain coefficient is 0, and the second preset value is greater than the third preset value.

[0191] In another possible implementation, the surround view camera is a fisheye camera, and the acquisition module 401 is specifically used to obtain the fisheye image taken by the fisheye camera; the processing module 402 is also used to perform distortion correction on the fisheye image based on the internal and external camera parameters of the fisheye camera to obtain the current scene image.

[0192] It should be noted that Fig.12 The division of modules in the example is schematic and is only a logical function division. There may be other division methods in actual implementation. For example, two or more functions may be integrated into one processing module. The above integrated modules may be implemented in the form of hardware or software function modules.

[0193] Another embodiment of the present application further provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the image generation method shown in the above embodiment.

[0194] In actual implementation, the acquisition module 401 and the processing module 402 can be implemented by the processor of the computer device calling the computer program code in the memory. The specific execution process can refer to the description of the above image generation method part, which will not be repeated here.

[0195] Another embodiment of the present application further provides a computer-readable storage medium, on which computer instructions are stored. When the computer instructions are executed by a processor, the steps of the image generation method shown in the above embodiment are implemented.

[0196] In another embodiment of the present application, a computer program product is provided. The computer program product includes computer instructions. When the computer instructions are executed by a processor, the steps of the image generation method shown in the above embodiment are implemented.

[0197] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer execution instruction is loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instruction can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that can be accessed by the computer or a data storage device such as a server, data center, etc. that contains one or more servers that can be integrated with the medium. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), etc.

[0198] The above is only a specific implementation of the present application. Those skilled in the art may conceive of changes or substitutions based on the specific implementation provided by the present application, which should all be included in the protection scope of the present application.

Claims

1. An image generation method, characterized in that: The method comprises: Acquire a current scene image around the vehicle captured by each of multiple surround view cameras deployed on the vehicle, wherein the shooting range of all surround view cameras deployed on the vehicle covers 360° around the vehicle; For each surround-view camera, coordinate mapping is performed on the current scene image of the surround-view camera from the camera coordinate system to the imaging plane in the world coordinate system to obtain a panoramic sub-image of the surround-view camera; in the mapping process, based on the pixel value of the first pixel point in the current scene image of the adjacent surround-view camera, pixel correction is performed on the pixel value of the first pixel point in the current scene image of the surround-view camera to obtain the pixel value of the first pixel point in the panoramic sub-image of the surround-view camera; the first pixel point is a pixel point corresponding to an overlapping area of ​​the shooting range of the surround-view camera and the adjacent surround-view camera; Performing pixel fusion on pixel values ​​of pixel points of the panoramic sub-image in the overlapping area; The overlapping areas after pixel fusion and the non-overlapping areas in the panoramic sub-image are spliced ​​to obtain a bird's-eye view of the vehicle.

2. The method according to claim 1, characterized in that The method of performing pixel correction on the pixel value of the first pixel in the current scene image of the neighboring surround view camera based on the pixel value of the first pixel in the current scene image of the surround view camera to obtain the pixel value of the first pixel in the panoramic sub-image of the surround view camera includes: Performing weighted fusion on a pixel value of the first pixel point in the current scene image of the surround view camera and a pixel value of the first pixel point in the current scene image of an adjacent surround view camera to obtain a fused pixel value; The product of the historical gain value and the gain coefficient is added to the fused pixel value to obtain the pixel value of the first pixel in the panoramic sub-image of the surround-view camera; wherein the historical gain value is used to characterize the degree of difference in the shooting results of the spatial position corresponding to the first pixel between the surround-view camera and the adjacent surround-view camera within a historical time period.

3. The method according to claim 2, characterized in that The gain coefficient is determined based on the absolute value of the difference between a first fusion weight and a second fusion weight, wherein the first fusion weight is the weight used when the pixel value of the first pixel point in the current scene image of the surround-view camera participates in a weighted fusion calculation, and the second fusion weight is the weight used when the pixel value of the first pixel point in the current scene image of the adjacent surround-view camera participates in a weighted fusion calculation.

4. The method according to claim 3, characterized in that When the absolute value of the difference between the first fusion weight and the second fusion weight is greater than a preset threshold, the gain coefficient is 0; or, when the absolute value of the difference between the first fusion weight and the second fusion weight is less than or equal to the preset threshold, the gain coefficient is 1.

5. The method according to claim 2, characterized in that: The step of weighting and fusing the pixel value of the first pixel point in the current scene image of the surround view camera with the pixel value of the first pixel point in the current scene image of an adjacent surround view camera to obtain a fused pixel value includes: Acquire multiple pixel values ​​of a second pixel point in multiple historical scene images taken by the surround-view camera, and multiple pixel values ​​of the second pixel point in multiple historical scene images taken by other surround-view cameras; the second pixel point is a pixel point in the historical scene image corresponding to the same spatial position as the first pixel point; Calculating a first variance value based on multiple pixel values ​​of the second pixel point in multiple historical scene images taken by the surround view camera, and calculating a second variance value based on multiple pixel values ​​of the second pixel point in multiple historical scene images taken by the other surround view cameras; Based on the size relationship between the first variance value and the second variance value, the first fusion weight and the second fusion weight used by the surround-view camera at the previous moment are corrected to obtain the first fusion weight and the second fusion weight used by the surround-view camera at the current moment; the first fusion weight is the weight used when the pixel value of the first pixel point in the current scene image of the surround-view camera participates in the weighted fusion calculation, and the second fusion weight is the weight used when the pixel value of the first pixel point in the current scene image of the adjacent surround-view camera participates in the weighted fusion calculation; Based on the first fusion weight and the second fusion weight adopted by the surround view camera at the current moment, the pixel value of the first pixel point in the current scene image of the surround view camera and the pixel value of the first pixel point in the current scene image of the adjacent surround view camera are weightedly fused to obtain a fused pixel value.

6. The method according to claim 5, characterized in that The first fusion weight and the second fusion weight adopted by the surround-view camera at the last moment are corrected based on the size relationship between the first variance value and the second variance value to obtain the first fusion weight and the second fusion weight adopted by the surround-view camera at the current moment, including: When the first variance value is less than the second variance value, the first fusion weight used by the surround-view camera at the last moment is increased by a preset value to obtain the first fusion weight used by the surround-view camera at the current moment, and the second fusion weight used by the surround-view camera at the last moment is reduced by the preset value to obtain the second fusion weight used by the surround-view camera at the current moment; When the first variance value is greater than the second variance value, reducing the first fusion weight used by the surround-view camera at the last moment by the preset value to obtain the first fusion weight used by the surround-view camera at the current moment, and increasing the second fusion weight used by the surround-view camera at the last moment by the preset value to obtain the second fusion weight used by the surround-view camera at the current moment; When the first variance value is equal to the second variance value, the first fusion weight used by the surround-view camera at the previous moment is used as the first fusion weight used by the surround-view camera at the current moment, and the second fusion weight used by the surround-view camera at the previous moment is used as the second fusion weight used by the surround-view camera at the current moment.

7. The method according to claim 5, characterized in that The first variance value and multiple pixel values ​​of the second pixel point in multiple historical scene images captured by the surround view camera satisfy the following relationship: in, For the first variance value, PixCur i is used to represent the pixel value of the i-th second pixel point in the current scene image captured by the surround view camera, is used to represent the average value of multiple pixel values ​​of the i-th second pixel point in multiple historical scene images taken by the surround-view camera, and M is used to represent the number of second pixel points in the current scene image taken by the surround-view camera; The second variance value and multiple pixel values ​​of the second pixel point in multiple historical scene images taken by the other surround view cameras satisfy the following relationship: in, Used to represent the second variance value, PixAdj j is used to represent the pixel value of the j-th second pixel point in the current scene image captured by the adjacent surround view camera, is used to represent the average value of multiple pixel values ​​of the j-th second pixel point in multiple historical scene images taken by the other surround view cameras, and N is used to represent the number of second pixel points in the current scene image taken by the adjacent surround view camera; The following relationship is satisfied between the fused pixel value of the first pixel in the panoramic sub-image of the surround view camera, the first fusion weight, the second fusion weight, the pixel value of the first pixel in the current scene image taken by the surround view camera, and the pixel value of the first pixel in the current scene image taken by the adjacent surround view camera: Among them, Pix n It is used to represent the fused pixel value of the nth first pixel point in the panoramic sub-image of the surround view camera, w1 is used to represent the first fusion weight, w2 is used to represent the second fusion weight, PixCur n PixAdj is used to represent the pixel value of the nth first pixel in the current scene image captured by the surround view camera. n Used to indicate that PixAdj n It is used to represent the pixel value of the nth first pixel point in the current scene image captured by the adjacent surround camera, and a is used to represent the gain coefficient, PixGain n Used to represent the historical gain value corresponding to the nth first pixel point; The historical gain value and the multiple pixel values ​​of the second pixel point in the multiple historical scene images taken by the surround view camera and the multiple pixel values ​​of the second pixel point in the multiple historical scene images taken by the other surround view cameras satisfy the following relationship: Among them, PixGain m Used to represent the historical gain value of the mth second pixel point, used to represent the average value of multiple pixel values ​​of the mth second pixel point in multiple historical scene images taken by the surround view camera, Used to represent the average value of multiple pixel values ​​of the mth second pixel point in multiple historical scene images taken by the other surround-view cameras.

8. The method according to claim 5, characterized in that The obtaining of multiple pixel values ​​of the second pixel point in multiple historical scene images taken by the surround view camera, and multiple pixel values ​​of the second pixel point in multiple historical scene images taken by other surround view cameras, includes: Acquire vehicle displacement information corresponding to each frame of the historical scene image, where the vehicle displacement information corresponding to the historical scene image is the vehicle displacement information from the shooting time of the historical scene image to the shooting time of the current scene image; For each frame of the historical scene image captured by the surround-view camera, based on the vehicle displacement information corresponding to the historical scene image and the pixel coordinates of the third pixel point on the current scene image captured by the surround-view camera, determine a first corresponding pixel point of the third pixel point from the historical scene image; use the first corresponding pixel point and the corresponding third pixel point that meet a preset condition as second pixel points representing the same spatial position, use the first corresponding pixel point and the corresponding third pixel point that do not meet the preset condition as pixel points in a three-dimensional obstacle, and determine that the pixel point in the three-dimensional obstacle does not belong to the second pixel point; For each frame of the historical scene image captured by the other surround view cameras, based on the vehicle displacement information corresponding to the historical scene image and the pixel coordinates of the fourth pixel point on the current scene image captured by the adjacent surround view camera, determine the second corresponding pixel point of the fourth pixel point from the historical scene image; use the second corresponding pixel point and the corresponding fourth pixel point that meet the preset condition as the second pixel point representing the same spatial position, use the second corresponding pixel point and the corresponding fourth pixel point that do not meet the preset condition as the pixel point in the three-dimensional obstacle, and determine that the pixel point in the three-dimensional obstacle does not belong to the second pixel point; The preset conditions include: the Euclidean distance corresponding to the corresponding pixel point satisfies the distance threshold condition, and the Euclidean distance corresponding to at least one adjacent pixel point of the corresponding pixel point satisfies the distance threshold condition, the Euclidean distance corresponding to the corresponding pixel point is used to represent the Euclidean distance between the corresponding pixel point and the corresponding pixel point in the current scene image in the color space, and the distance threshold condition is used to measure the color similarity between two pixel points.

9. The method according to claim 8, characterized in that The obtaining of the vehicle displacement information corresponding to each frame of the historical scene image includes: For each frame of the historical scene image, obtain the shooting time interval, the moving speed of the vehicle within the shooting time interval, and the steering angle; the shooting time interval is the time interval from the shooting moment of the historical scene image to the shooting moment of the current scene image; based on the shooting time interval, the moving speed of the vehicle within the shooting time interval, and the steering angle, determine the vehicle displacement information corresponding to the historical scene image.

10. The method according to claim 9, characterized in that The vehicle displacement information satisfies the following relationship with the shooting time interval, the moving speed of the vehicle within the shooting time interval, and the steering angle: Δx=cos(θ a )VΔt; Δy=sin(θ a )VΔt; Wherein, Δx is the displacement of the vehicle in the X-axis direction in the world coordinate system with the vehicle body as the origin in the vehicle displacement information, Δy is the displacement of the vehicle in the Y-axis direction in the world coordinate system with the vehicle body as the origin in the vehicle displacement information, and θ a is the steering angle of the vehicle within the shooting time interval, Δt is the shooting time interval, and V is the moving speed of the vehicle within the shooting time interval.

11. The method according to any one of claims 3 to 10, characterized in that: In a case where the scene represented by the current scene image is a grass scene, determining that an initial value of the first fusion weight is a first preset value, and determining that an initial value of the second fusion weight is the first preset value; In the case where the scene represented by the current scene image is a scene with sudden brightness change, the initial value of the first fusion weight is determined to be a second preset value, the initial value of the second fusion weight is determined to be a third preset value, the gain coefficient is determined to be 0, and the second preset value is greater than the third preset value.

12. The method according to any one of claims 1 to 10, characterized in that: The surround view camera is a fisheye camera, and the acquiring of current scene images around the vehicle captured by each of a plurality of surround view cameras deployed on the vehicle includes: Acquire a fisheye image captured by the fisheye camera; Based on the internal and external camera parameters of the fisheye camera, the fisheye image is subjected to distortion correction to obtain the current scene image.

13. An image generating device, characterized in that: include: An acquisition module is used to acquire a current scene image around the vehicle captured by each of multiple surround view cameras deployed on the vehicle, wherein the shooting range of all surround view cameras deployed on the vehicle covers 360° around the vehicle; A processing module, configured to perform coordinate mapping of a current scene image of each surround-view camera from a camera coordinate system to an imaging plane in a world coordinate system, so as to obtain a panoramic sub-image of the surround-view camera; During the mapping process, based on the pixel value of the first pixel point in the current scene image of the adjacent surround view camera, pixel correction is performed on the pixel value of the first pixel point in the current scene image of the surround view camera to obtain the pixel value of the first pixel point in the panoramic sub-image of the surround view camera; The first pixel point is a pixel point corresponding to an overlapping area of ​​the shooting range of the surround view camera and the adjacent surround view camera; The processing module is further used to perform pixel fusion on pixel values ​​of pixel points of the panoramic sub-image in the overlapping area; The processing module is also used to stitch the overlapping areas after pixel fusion and the non-overlapping areas in the panoramic sub-image to obtain a surround bird's-eye view of the vehicle.

14. The device according to claim 13, characterized in that The processing module is specifically used to: perform weighted fusion on the pixel value of the first pixel point in the current scene image of the surround view camera and the pixel value of the first pixel point in the current scene image of the adjacent surround view camera to obtain a fused pixel value; add the product of the historical gain value and the gain coefficient to the fused pixel value to obtain the pixel value of the first pixel point in the panoramic sub-image of the surround view camera; wherein the historical gain value is used to characterize the degree of difference between the shooting results of the surround view camera and the adjacent surround view camera for the spatial position corresponding to the first pixel point within a historical time period; and / or, The gain coefficient is determined according to the absolute value of the difference between the first fusion weight and the second fusion weight, the first fusion weight being the weight used when the pixel value of the first pixel point in the current scene image of the surround view camera participates in the weighted fusion calculation, and the second fusion weight being the weight used when the pixel value of the first pixel point in the current scene image of the adjacent surround view camera participates in the weighted fusion calculation; and / or, When the absolute value of the difference between the first fusion weight and the second fusion weight is greater than a preset threshold, the gain coefficient is 0; or, when the absolute value of the difference between the first fusion weight and the second fusion weight is less than or equal to the preset threshold, the gain coefficient is 1; and / or, The acquisition module is also used to obtain multiple pixel values ​​of a second pixel point in multiple historical scene images taken by the surround-view camera, and multiple pixel values ​​of the second pixel point in multiple historical scene images taken by other surround-view cameras; the second pixel point is a pixel point in the historical scene image corresponding to the same spatial position as the first pixel point; the processing module is specifically used to: calculate a first variance value based on the multiple pixel values ​​of the second pixel point in the multiple historical scene images taken by the surround-view camera, and calculate a second variance value based on the multiple pixel values ​​of the second pixel point in the multiple historical scene images taken by the other surround-view cameras; based on the size relationship between the first variance value and the second variance value, The first fusion weight and the second fusion weight are corrected to obtain the first fusion weight and the second fusion weight used by the surround-view camera at the current moment; the first fusion weight is the weight used when the pixel value of the first pixel point in the current scene image of the surround-view camera participates in the weighted fusion calculation, and the second fusion weight is the weight used when the pixel value of the first pixel point in the current scene image of the adjacent surround-view camera participates in the weighted fusion calculation; based on the first fusion weight and the second fusion weight used by the surround-view camera at the current moment, the pixel value of the first pixel point in the current scene image of the surround-view camera and the pixel value of the first pixel point in the current scene image of the adjacent surround-view camera are weightedly fused to obtain a fused pixel value; and / or, The processing module is specifically used for: when the first variance value is less than the second variance value, increasing the first fusion weight used by the surround-view camera at the last moment by a preset value to obtain the first fusion weight used by the surround-view camera at the current moment, and reducing the second fusion weight used by the surround-view camera at the last moment by the preset value to obtain the second fusion weight used by the surround-view camera at the current moment; when the first variance value is greater than the second variance value, reducing the first fusion weight used by the surround-view camera at the last moment by the preset value to obtain the first fusion weight used by the surround-view camera at the current moment, and increasing the second fusion weight used by the surround-view camera at the last moment by the preset value to obtain the second fusion weight used by the surround-view camera at the current moment; when the first variance value is equal to the second variance value, using the first fusion weight used by the surround-view camera at the last moment as the first fusion weight used by the surround-view camera at the current moment, and using the second fusion weight used by the surround-view camera at the last moment as the second fusion weight used by the surround-view camera at the current moment; and / or, The first variance value and multiple pixel values ​​of the second pixel point in multiple historical scene images captured by the surround view camera satisfy the following relationship: in, For the first variance value, PixCur i is used to represent the pixel value of the i-th second pixel point in the current scene image captured by the surround view camera, is used to represent the average value of multiple pixel values ​​of the i-th second pixel point in multiple historical scene images taken by the surround view camera, M is used to represent the number of second pixel points in the current scene image taken by the surround view camera; the second variance value and the multiple pixel values ​​of the second pixel point in the multiple historical scene images taken by the other surround view cameras satisfy the following relationship: in, is used to represent the second variance value, is used to represent the pixel value of the j-th second pixel point in the current scene image captured by the adjacent surround view camera, is used to represent the average value of multiple pixel values ​​of the j-th second pixel in multiple historical scene images taken by the other surround view cameras, and N is used to represent the number of second pixels in the current scene image taken by the adjacent surround view camera; the fused pixel value of the first pixel in the panoramic sub-image of the surround view camera and the first fusion weight, the second fusion weight, the pixel value of the first pixel in the current scene image taken by the surround view camera, and the pixel value of the first pixel in the current scene image taken by the adjacent surround view camera satisfy the following relationship: Among them, Pix n It is used to represent the fused pixel value of the nth first pixel point in the panoramic sub-image of the surround view camera, w1 is used to represent the first fusion weight, w2 is used to represent the second fusion weight, PixCur n PixAdj is used to represent the pixel value of the nth first pixel in the current scene image captured by the surround view camera. n Used to indicate that PixAdj n It is used to represent the pixel value of the nth first pixel point in the current scene image captured by the adjacent surround camera, and a is used to represent the gain coefficient, PixGain n Used to represent the historical gain value corresponding to the nth first pixel point; the historical gain value and the multiple pixel values ​​of the second pixel point in the multiple historical scene images taken by the surround view camera, and the multiple pixel values ​​of the second pixel point in the multiple historical scene images taken by the other surround view cameras satisfy the following relationship: Among them, PixGain m Used to represent the historical gain value of the mth second pixel point, used to represent the average value of multiple pixel values ​​of the mth second pixel point in multiple historical scene images taken by the surround view camera, used to represent the average value of multiple pixel values ​​of the mth second pixel point in multiple historical scene images taken by the other surround view cameras; and / or, The acquisition module is specifically used to acquire vehicle displacement information corresponding to each frame of the historical scene image, where the vehicle displacement information corresponding to the historical scene image is the vehicle displacement information from the shooting time of the historical scene image to the shooting time of the current scene image; the processing module is also used to: for each frame of the historical scene image shot by the surround-view camera, based on the vehicle displacement information corresponding to the historical scene image and the pixel coordinates of the third pixel point on the current scene image shot by the surround-view camera, determine the first corresponding pixel point of the third pixel point from the historical scene image; use the first corresponding pixel point and the corresponding third pixel point that meet the preset conditions as the second pixel point representing the same spatial position, use the first corresponding pixel point and the corresponding third pixel point that do not meet the preset conditions as the pixel point in the three-dimensional obstacle, and determine that the pixel point in the three-dimensional obstacle does not belong to the second pixel point; for each frame of the historical scene image shot by the other surround-view cameras, based on the historical scene image The vehicle displacement information corresponding to the historical scene image, and the pixel coordinates of the fourth pixel point on the current scene image taken by the adjacent surround view camera, determine the second corresponding pixel point of the fourth pixel point from the historical scene image; use the second corresponding pixel point and the corresponding fourth pixel point that meet the preset condition as the second pixel point representing the same spatial position, use the second corresponding pixel point and the corresponding fourth pixel point that do not meet the preset condition as the pixel point in the three-dimensional obstacle, and determine that the pixel point in the three-dimensional obstacle does not belong to the second pixel point; the preset condition includes: the Euclidean distance corresponding to the corresponding pixel point satisfies the distance threshold condition, and the Euclidean distance corresponding to at least one adjacent pixel point of the corresponding pixel point satisfies the distance threshold condition, the Euclidean distance corresponding to the corresponding pixel point is used to represent the Euclidean distance between the corresponding pixel point and the corresponding pixel point in the current scene image in the color space, and the distance threshold condition is used to measure the color similarity between two pixels; and / or, The acquisition module is specifically used to acquire, for each frame of the historical scene image, a shooting time interval, a moving speed of the vehicle within the shooting time interval, and a steering angle; the shooting time interval is a time interval from the shooting time of the historical scene image to the shooting time of the current scene image; the processing module is also used to determine the vehicle displacement information corresponding to the historical scene image based on the shooting time interval, the moving speed of the vehicle within the shooting time interval, and the steering angle; and / or, The vehicle displacement information satisfies the following relationship with the shooting time interval, the moving speed of the vehicle within the shooting time interval, and the steering angle: Δx=cos(θ a )VΔt;Δy=sin(θ a )VΔt; wherein Δx is the displacement of the vehicle in the X-axis direction in the world coordinate system with the vehicle body as the origin in the vehicle displacement information, Δy is the displacement of the vehicle in the Y-axis direction in the world coordinate system with the vehicle body as the origin in the vehicle displacement information, θ a is the steering angle of the vehicle within the shooting time interval, Δt is the shooting time interval, and V is the moving speed of the vehicle within the shooting time interval; and / or, The processing module is further used for: when the scene represented by the current scene image is a grass scene, determining that the initial value of the first fusion weight is a first preset value, and determining that the initial value of the second fusion weight is the first preset value; when the scene represented by the current scene image is a brightness mutation scene, determining that the initial value of the first fusion weight is a second preset value, determining that the initial value of the second fusion weight is a third preset value, determining that the gain coefficient is 0, and the second preset value is greater than the third preset value; and / or, The surround view camera is a fisheye camera, and the acquisition module is specifically used to acquire the fisheye image taken by the fisheye camera; the processing module is also used to perform distortion correction on the fisheye image based on internal and external camera parameters of the fisheye camera to obtain the current scene image.

15. A vehicle, characterized in that: Includes multiple cameras and electronic equipment for different shooting angles; The camera is used to capture images and send the images to the electronic device; The electronic device is used to receive images sent by each of the cameras and execute the image generation method described in any one of claims 1 to 12.

16. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images