Splicing method, device and equipment of vehicle around-view transparent chassis image and medium

Abstract

The invention provides a splicing method, device and equipment for a vehicle look-around transparent chassis image and a medium. The method comprises the steps that the look-around chassis image of a vehicle at the current moment and actually-measured vehicle driving data are acquired; filtering the actually measured vehicle driving data by using a filtering algorithm to obtain predicted vehicle driving data; intercepting a target chassis image from the look-around transparent chassis image at the previous moment according to the predicted vehicle driving data; and obtaining a look-around transparent chassis image at the current moment based on the target chassis image and the look-around chassis image at the current moment. Thus, the actually measured vehicle driving data is filtered by using the filtering algorithm, the corresponding position information of the vehicle at the current moment in the look-around transparent image at the previous moment can be more accurately determined, a more accurate target chassis image is obtained, the splicing dislocation of the target chassis image and the look-around chassis image at the current moment is avoided, and the accuracy of the target chassis image is improved. And the display effect of the look-around transparent chassis image is improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle-mounted panoramic imaging technology, and specifically to a method, device, equipment and medium for stitching vehicle surround transparent chassis images. Background Art

[0002] Transparent chassis is a common technology in the automotive field in recent years that is configured for panoramic imaging functions. It refers to the transmission of real-time images from under the chassis of the car to the large central control screen inside the car through the body camera, allowing the driver to intuitively see the condition of the road surface under the car, thus achieving a true "panoramic" image. With the further development of panoramic imaging functions in the automotive industry, how to optimize the display effect of the transparent chassis has gradually become a major research topic in the automotive industry. Among them, the splicing misalignment problem between the vehicle chassis area and other areas (such as the vehicle surround area) seriously affects the display effect of the transparent chassis. Therefore, how to solve the splicing misalignment of the transparent chassis is a problem that needs to be solved urgently. Summary of the invention

[0003] The present application provides a method, device, equipment and medium for splicing images of a vehicle's surround transparent chassis. The method can avoid misalignment in splicing of the vehicle's transparent chassis and improve the display effect of the transparent chassis.

[0004] The technical solution of this application is implemented as follows:

[0005] An embodiment of the present application provides a method for stitching a vehicle surround-view transparent chassis image, comprising: acquiring a surround-view chassis image of a vehicle at a current moment, and measured vehicle driving data corresponding to the vehicle at the current moment; filtering the measured vehicle driving data using a filtering algorithm to obtain predicted vehicle driving data corresponding to the vehicle at the current moment; based on the predicted vehicle driving data, extracting a target chassis image from a surround-view transparent chassis image at a previous moment; and performing stitching processing based on the target chassis image and the surround-view chassis image at the current moment to obtain a surround-view transparent chassis image at the current moment.

[0006] According to the above technical means, the measured vehicle driving data is filtered by using a filtering algorithm to remove the noise in the measured vehicle data, so that the corresponding position information of the current vehicle in the surround-view transparent image at the previous moment can be more accurately determined based on the preset vehicle driving data obtained after filtering, thereby obtaining a more accurate target chassis image, avoiding the splicing misalignment of the target chassis image and the surround-view chassis image at the current moment, and improving the display effect of the surround-view transparent chassis image at the current moment.

[0007] Furthermore, the measured vehicle driving data includes the measured displacement of the vehicle, and the predicted vehicle driving data includes the predicted displacement of the vehicle from the previous moment to the current moment; the filtering algorithm includes the Kalman filtering algorithm; the use of the wave algorithm to predict the measured vehicle driving data to obtain the predicted vehicle driving data corresponding to the vehicle at the current moment includes: creating a state transfer matrix and a state observation matrix of the Kalman filter; determining the Kalman update formula according to the state transfer matrix, the state observation matrix, the preset Kalman prediction formula and the covariance matrix; updating the measured displacement based on the Kalman update formula to obtain the predicted displacement.

[0008] According to the above technical means, the Kalman prediction formula and the Kalman update formula are constructed to realize the update and prediction of the measured displacement, so as to obtain a more accurate displacement, so that the vehicle position information can be calculated more accurately based on the predicted displacement in the future, thereby improving the accuracy of the target chassis image finally captured.

[0009] Furthermore, the predicted vehicle driving data also includes a first predicted vehicle speed; the method also includes: determining a delay time from when the vehicle sensor collects the measured vehicle driving data to when the on-board electronic device acquires the measured vehicle driving data; determining a compensation displacement based on the delay time and the first predicted vehicle speed; correcting the predicted displacement based on the compensation displacement to obtain a new predicted displacement; and based on the new predicted displacement, extracting the target chassis image from the surround transparent chassis image at the previous moment.

[0010] According to the above technical means, the predicted displacement determined by the Kalman filter algorithm is corrected by the delay time from the vehicle sensor collecting the actual vehicle driving data to the on-board electronic equipment obtaining the measured vehicle driving data and the compensation displacement determined by the first predicted vehicle speed, so that the target chassis image finally captured from the surround transparent chassis image at the previous moment can be more accurate.

[0011] Furthermore, the method of extracting a target chassis image from the surround-view transparent chassis image at the previous moment based on the predicted vehicle driving data includes: determining position information of four vertices of the vehicle at the current moment in the surround-view transparent chassis image at the previous moment based on the predicted vehicle driving data; determining a first area from the surround-view transparent chassis image at the previous moment based on the position information; and extracting an image corresponding to the first area to obtain the target chassis image.

[0012] According to the above technical means, the position information of the four vertices of the vehicle at the current moment in the surround-view transparent chassis image at the previous moment is determined by predicting the vehicle driving data, so that the target chassis image can be intercepted from the surround-view transparent chassis image at the previous moment according to the position information, thereby realizing the acquisition of the transparent chassis image.

[0013] Furthermore, the predicted vehicle driving data includes a predicted displacement of the vehicle from a previous moment to a current moment; and determining the position information of four vertices of the vehicle at the current moment in the previous surround transparent chassis image frame based on the predicted vehicle driving data includes: determining a rotation angle of the vehicle at the current moment relative to the previous moment based on the predicted displacement; and determining the position information of four vertices of the vehicle at the current moment in the previous surround transparent chassis image frame based on the rotation angle and the position information of the vehicle's rotation center.

[0014] According to the above technical means, the rotation angle of the vehicle at the current moment relative to the previous moment can be determined by predicting the displacement, and the position information of the vehicle's rotation center can be further determined based on the rotation angle, so the position information of the four vertices of the vehicle at the current moment in the previous surround transparent chassis image frame can be accurately calculated.

[0015] Further, the method of extracting a target chassis image from the surround-view transparent chassis image at a previous moment based on the predicted vehicle driving data includes: extracting a reference chassis image from the surround-view transparent chassis image at a previous moment based on the predicted vehicle driving data; if there is a block shadow area in the reference chassis image, re-capturing the surround-view transparent chassis image at the previous moment to obtain the target chassis image; the target chassis image does not include the block shadow area, and the size of the target chassis image is larger than the size of the reference chassis image.

[0016] According to the above-mentioned technical means, when the reference chassis image includes block shadows, the surround transparent chassis image of the previous moment is re-captured, so that the target chassis image includes more visible or more image content, so that the block shadows can be excluded from the actual vehicle chassis area in the subsequent stitching process, and a clearer vehicle chassis area image can be obtained.

[0017] Further, the stitching processing based on the target chassis image and the surround-view chassis image at the current moment to obtain the surround-view transparent chassis image corresponding to the current moment includes: enlarging the target chassis image by a preset multiple to obtain a first candidate image; determining the overlapping area between the enlarged surround-view transparent chassis image corresponding to the previous moment and the surround-view chassis image at the current moment; stitching the image of the overlapping area with the first candidate image to obtain a second candidate image; reducing the second candidate image by the preset multiple to obtain a third candidate image; and stitching the third candidate image with the surround-view chassis image at the current moment to obtain the surround-view transparent chassis image corresponding to the current moment.

[0018] According to the above technical means, by magnifying the captured target chassis image by a preset multiple, the third candidate image obtained by reducing the second candidate image can retain more pixel content, thereby effectively reducing the image blur in the transparent chassis area that is common when the vehicle is at low speed, and making the final surround transparent chassis image at the current moment clearer.

[0019] Furthermore, the stitching processing of the third candidate image and the surround-view chassis image at the current moment to obtain the surround-view transparent chassis image corresponding to the current moment includes: determining a first weight of corresponding image pixel values ​​in the vehicle surround-view area in the third candidate image, and a second weight of corresponding image pixel values ​​in the vehicle surround-view area in the surround-view chassis image at the current moment; the first weight of image pixels close to the vehicle chassis area is greater than the second weight; determining an image of the vehicle surround-view area at the current moment based on the first weight, the second weight, image pixel values ​​in the vehicle surround-view area in the third candidate image, and image pixel values ​​in the vehicle surround-view area in the surround-view chassis image at the current moment; and stitching processing of the image of the vehicle surround-view area and the image corresponding to the vehicle chassis area in the third candidate image to obtain the surround-view transparent chassis image at the current moment.

[0020] According to the above technical means, the image pixel values ​​of the vehicle surround view area of ​​the third candidate image and the surround view chassis image at the current moment are assigned different weights for fusion, so that the image content in the vehicle surround view area in the final transparent chassis image at the current moment is closer to the actual scene. Thereafter, the image of the vehicle surround view area obtained after the fusion and the image of the chassis area in the surround view chassis image at the current moment are spliced ​​to obtain a surround view transparent chassis image with a good display effect.

[0021] Furthermore, the predicted vehicle driving data includes a first predicted vehicle speed; the method also includes: determining the sharpness value of the surround-view transparent chassis image at the current moment according to the first predicted vehicle speed and a preset correspondence between the predicted vehicle speed and the sharpness value of the surround-view transparent chassis image; based on the sharpness value, sharpening the surround-view transparent chassis image at the current moment to obtain the target surround-view transparent chassis image at the current moment.

[0022] According to the above technical means, by determining the sharpness value of the surround-view transparent chassis image corresponding to the first predicted vehicle speed, and sharpening the surround-view transparent chassis image at the current moment based on the sharpness value, the blur problem caused by the previous enlargement of the surround-view transparent chassis image can be solved, and the clarity of the surround-view transparent chassis image at the current moment can be improved.

[0023] The embodiment of the present application provides a splicing device for a vehicle surround transparent chassis image, comprising:

[0024] A data acquisition module, used to acquire a surround chassis image of the vehicle at the current moment, and actual vehicle driving data corresponding to the vehicle at the current moment;

[0025] A filtering processing module, used to filter the measured vehicle driving data using a filtering algorithm to obtain predicted vehicle driving data corresponding to the vehicle at the current moment;

[0026] An image capture module, which captures a target chassis image from the surround transparent chassis image at the last moment according to the predicted vehicle driving data;

[0027] The splicing processing module is used to perform splicing processing based on the target chassis image and the surround chassis image at the current moment to obtain the surround transparent chassis image at the current moment.

[0028] The present application embodiment provides a device for splicing a vehicle surround transparent chassis image, including:

[0029] A memory for storing computer programs that can be run on the processor;

[0030] The processor is used to execute the vehicle surround transparent chassis image stitching method provided in the embodiment of the present application when running the computer program.

[0031] An embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. The computer-executable instructions are configured to execute the above-mentioned method for stitching the vehicle surround transparent chassis image. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1A schematic diagram of a process of stitching a vehicle surround view transparent chassis image provided in an embodiment of the present application;

[0033] Figure 2 A schematic diagram of a model of instantaneous motion of the rear axle center of a vehicle provided in an embodiment of the present application;

[0034] Figure 3 A flowchart illustrating a method for implementing a transparent chassis based on multi-data fusion provided in an embodiment of the present application;

[0035] Figure 4 A schematic diagram of the effect of chassis splicing optimization provided in an embodiment of the present application;

[0036] Figure 5 A schematic diagram of the effect of chassis area image blur optimization provided by an embodiment of the present application;

[0037] Figure 6 A schematic diagram of an optimization effect obtained by a transparent chassis splicing method based on multi-data fusion provided in an embodiment of the present application;

[0038] Figure 7 A schematic diagram of the structure of a splicing device for a vehicle surround view transparent chassis image provided in an embodiment of the present application;

[0039] Figure 8 A schematic diagram of the composition structure of a vehicle surround view transparent chassis image splicing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0041] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0042] In the following description, reference is made to “some embodiments\other embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments\other embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0043] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0045] In the related technology, there is a transparent chassis optimization method, device, equipment and storage medium. The method crops a portion of the image at a specified position from the historical surround stitching image for stitching, and then performs filtering on the stitched chassis image. The filtering process constructs a frequency domain filtering operator based on the characteristics of the chassis image, thereby ensuring that most water ripples are filtered out with minimal loss of detail, thereby improving the display effect of the transparent chassis. The optimization direction of this patent is water ripples. It does not provide optimization ideas for problems such as misalignment between vehicle chassis area images and other area images and blurring of vehicle chassis area images at slow speeds, and thus cannot solve common misalignment and blurring problems.

[0046] In the related art, there is a method and system for realizing a transparent chassis function, which includes: when the vehicle is at location A, obtaining a road surface image in the direction in which the vehicle is about to move; combining the vehicle speed and steering wheel, when the vehicle drives to the corresponding road area location B, the road surface image acquired in advance at location A and the images of the four directions of location B in front, behind, left, and right are spliced ​​to obtain a 360° panoramic view of the vehicle in front, behind, left, and right and a 180° chassis perspective image on the ground, that is, a 540° surround view image of the vehicle body including the transparent chassis. The use of surround view to realize the transparent chassis function can enable the driver to intuitively perceive the position of the tires and obstacles such as potholes, stones, small animals or manhole covers on the road surface, eliminate the driver's blind spots, avoid safety hazards during driving, and improve the safety performance of the car. This method only provides a basic method for obtaining images of the vehicle chassis area, and does not consider issues such as misalignment optimization that may occur later. Therefore, the accuracy and clarity of the vehicle chassis area image cannot be guaranteed.

[0047] Based on the problems existing in the related art, the embodiment of the present application provides a method for splicing a vehicle surround transparent chassis image, which can be applied to a splicing device for a vehicle surround transparent chassis image, and can avoid the splicing misalignment of the vehicle transparent chassis and improve the display effect of the transparent chassis. Figure 1FIG. 1 is a flow chart of a method for stitching a vehicle surround view transparent chassis image provided by an embodiment of the present application, the method comprising the following steps:

[0048] S101, obtaining a surround chassis image of the vehicle at the current moment, and actually measured vehicle driving data corresponding to the vehicle at the current moment.

[0049] It should be noted that the surround chassis image of the vehicle at the current moment may be an image corresponding to the vehicle chassis area and the area surrounding the chassis (or the vehicle surround area) acquired at the current moment; the measured vehicle driving data may be the vehicle driving data actually measured by the vehicle's sensors, and the measured vehicle driving data may include the displacement and speed of the vehicle from the previous moment to the current moment.

[0050] In some embodiments, the surround chassis image may be an image frame extracted from a video captured by the vehicle during driving, and the surround chassis image may be a top-view image of the vehicle. Due to occlusion, the actual road conditions below the vehicle chassis cannot be captured, so there is no image content in the vehicle chassis area in the surround chassis image. The actual vehicle speed can be obtained through the vehicle speed sensor, and the actual vehicle displacement can be obtained through the vehicle displacement sensor.

[0051] S102: Filter the measured vehicle driving data using a filtering algorithm to obtain predicted vehicle driving data corresponding to the vehicle at the current moment.

[0052] In some embodiments, there may be noise in the measured vehicle driving data, which may come from the measurement error of the sensor, the error in the transmission process of the sensor after measuring the corresponding vehicle driving data, etc. Therefore, the measured vehicle driving data is filtered by a filtering algorithm to remove the noise in the measured vehicle driving data, thereby obtaining smoother vehicle driving data and predicting vehicle driving data. The filtering algorithm can be a Kalman filtering algorithm, an unscented Kalman filtering algorithm, a limiting filtering method, an algorithmic average filtering method, etc.

[0053] S103: According to the predicted vehicle driving data, a target chassis image is intercepted from the surround transparent chassis image at the last moment.

[0054] In some embodiments, based on the predicted vehicle driving data, the corresponding position information of the vehicle chassis area of ​​the current vehicle in the surround-view transparent chassis image at the previous moment can be determined, and then the area range of the image to be intercepted can be determined based on the position information. By intercepting the area range in the surround-view transparent chassis image at the previous moment, the predicted vehicle driving data corresponding to the vehicle at the current moment can be obtained.

[0055] S104: performing stitching processing based on the target chassis image and the surround chassis image at the current moment to obtain the surround transparent chassis image at the current moment.

[0056] In some embodiments, the target chassis image includes an image of a vehicle chassis area and an image of a vehicle surround area, wherein the image of the vehicle chassis area includes visible image content, and the image corresponding to the vehicle chassis area in the surround chassis image at the current moment includes non-visual content. Therefore, by splicing the target chassis image and the surround chassis image at the current moment, a surround transparent chassis image with visible image content in the vehicle chassis area can be obtained.

[0057] In some embodiments, the stitching process may include pixel fusion or image block connection processing of the target chassis image and the surround view chassis image at the current moment. For example, the vehicle chassis area image and the vehicle surround view area image in the target chassis image may be fused with the image of the vehicle chassis area in the surround view chassis image at the current moment, and the vehicle surround view area in the surround view transparent chassis image at the current moment and the fused image may be stitched to obtain the surround view transparent chassis image at the current moment.

[0058] In an embodiment of the present application, a surround chassis image of the vehicle at the current moment and the measured vehicle driving data corresponding to the vehicle at the current moment are obtained; the measured vehicle driving data are filtered using a filtering algorithm to obtain the predicted vehicle driving data corresponding to the vehicle at the current moment; the target chassis image is intercepted from the surround transparent chassis image at the previous moment according to the predicted vehicle driving data; the surround transparent chassis image at the current moment is obtained by splicing the target chassis image and the surround chassis image at the current moment. In this way, by filtering the measured vehicle driving data using a filtering algorithm, the noise in the measured vehicle data is removed, so that the corresponding position information of the vehicle at the current moment in the surround transparent image at the previous moment can be more accurately determined based on the preset vehicle driving data obtained after filtering, thereby obtaining a more accurate target chassis image, avoiding the splicing misalignment of the target chassis image and the surround chassis image at the current moment, and improving the display effect of the surround transparent chassis image at the current moment finally obtained.

[0059] In some embodiments of the present application, the measured vehicle driving data includes the measured displacement of the vehicle, and the predicted vehicle driving data includes the predicted displacement of the vehicle from the previous moment to the current moment; the filtering algorithm includes a Kalman filtering algorithm, based on which the Kalman filtering algorithm is used to perform predictive processing on the measured vehicle driving data to obtain the predicted vehicle driving data corresponding to the vehicle at the current moment, that is, the above-mentioned step S102 can be implemented by the following steps S1021 to S1023, and each step is described separately below.

[0060] S1021. Create a state transfer matrix and a state observation matrix for Kalman filtering.

[0061] In some embodiments, the state transfer matrix can be determined based on the conversion relationship between the vehicle driving data at the previous moment and the vehicle driving data at the current moment, such as the conversion relationship between the displacement distance of the vehicle at the previous moment and the displacement distance at the current moment, the displacement speed of the vehicle at the previous moment and the displacement speed of the vehicle at the current moment, and the displacement acceleration of the vehicle at the previous moment and the displacement acceleration of the vehicle at the current moment. The dimension of the state transfer matrix can be determined by the number of state quantities.

[0062] In some embodiments, the state observation matrix can be a relationship matrix between state quantities and observation values. The state quantities can include the displacement distance, displacement speed, displacement acceleration, etc. of the center position of the rear axle of the vehicle. The observation values ​​can be the displacement of the center of the rear axle of the vehicle obtained by a displacement sensor, the vehicle speed measured by an inertial sensor, etc.

[0063] S1022. Determine a Kalman update formula according to a state transfer matrix, a state observation matrix, a preset Kalman prediction formula and a covariance matrix.

[0064] In some embodiments, a Kalman filter formula can be determined based on a state transfer matrix, a preset Kalman prediction formula and a covariance matrix, and a Kalman update formula can be determined based on the Kalman filter formula and a state observation matrix.

[0065] For example, the preset Kalman prediction formula can be expressed by formula (1):

[0066]

[0067] Where A is the state transfer matrix, A={{0,dt,0.5*dt 2},{0,1,dt},{0,0,1}}, dt is the time interval between the previous moment and the current moment, which represents the formula for predicting the current moment state based on the data state at the previous moment, which represents the model of this system; P t - and Pt-1 They represent the covariance matrix of the current state estimate and the covariance matrix of the state estimate at time t-1 respectively; Q is the process noise matrix, which can be a three-dimensional matrix, corresponding to the process variance of distance, speed, and acceleration, respectively. and They represent the state quantity estimated at time t and the state quantity estimated at time t-1 respectively.

[0068] The Kalman update formula can be expressed by formula (2):

[0069]

[0070] Among them, K represents the Kalman gain, C is the state observation matrix, and R is the measurement noise matrix, which can be a two-dimensional matrix, corresponding to the measurement variance of distance and speed respectively.

[0071] S1023. Update the measured displacement based on the Kalman update formula to obtain the predicted displacement.

[0072] In some embodiments, the measured displacement may be introduced into the Kalman update formula, the measured displacement may be updated, and the updated result may be introduced into the Kalman prediction formula to obtain the predicted displacement.

[0073] According to the above technical means, the Kalman prediction formula and the Kalman update formula are constructed to realize the update and prediction of the measured displacement, so as to obtain a more accurate displacement, so that the vehicle position information can be calculated more accurately based on the predicted displacement in the future, thereby improving the accuracy of the target chassis image finally captured.

[0074] In some embodiments of the present application, the predicted vehicle driving data also includes a first predicted vehicle speed. After the measured displacement is updated based on the Kalman update formula to obtain the predicted displacement, that is, step S1023, the following steps S201 to S204 can also be executed. Each step is described below.

[0075] S201. Determine the delay time from when the vehicle sensor collects the measured vehicle driving data to when the on-board electronic device obtains the measured vehicle driving data.

[0076] In some embodiments, the delay duration may represent the time interval between the vehicle-mounted electronic device actually acquiring the measured vehicle driving data and the vehicle sensor acquiring the measured vehicle driving data. When determining the delay duration, the delay duration may be determined by recording the timestamp of the vehicle sensor acquiring the measured vehicle driving data and the timestamp of the vehicle-mounted electronic device acquiring the measured vehicle driving data. By determining the difference between the two timestamps, the delay duration may be determined.

[0077] In some embodiments, the measured vehicle driving data may include the measured displacement of the center position of the rear axle of the vehicle and the vehicle speed. The delay time may be the time it takes for the displacement sensor to collect the displacement of the center position of the rear axle and transmit the measured displacement to the on-board electronic device, or it may be the time it takes for the speed sensor or inertial sensor to detect the speed of the vehicle and transmit the speed to the on-board electronic device.

[0078] S202: Determine a compensation displacement according to the delay duration and the first predicted vehicle speed.

[0079] In some embodiments, the first predicted vehicle speed may be the vehicle speed obtained by removing noise from the vehicle speed at the current moment through a Kalman filter algorithm, and the compensation displacement may represent the displacement offset of the center position of the rear axle of the vehicle during the period from when the vehicle sensor collects the measured vehicle driving data to when the on-board electronic device obtains the measured vehicle driving data. After determining the delay time from when the vehicle sensor collects the measured vehicle driving data to when the on-board electronic device obtains the measured vehicle driving data, the product of the delay time and the first predicted vehicle speed may be used as the compensation displacement.

[0080] S203: Correct the predicted displacement based on the compensation displacement to obtain a new predicted displacement.

[0081] In some embodiments, there is a time delay between the time when the vehicle sensor collects the measured vehicle travel data and the time when the on-board electronic device receives the measured vehicle travel data, and the Kalman filter algorithm performs filtering processing on the measured displacement at the current moment only after the on-board electronic device receives the measured vehicle travel data, therefore, after obtaining the predicted displacement through Kalman filtering, it is also necessary to consider the distance traveled by the vehicle within the delay period, that is, the predicted displacement needs to be corrected.

[0082] In some embodiments, the new predicted displacement may be the displacement obtained after compensating the predicted displacement, the new predicted displacement is greater than the predicted displacement, the correction processing may be updating or modifying the predicted displacement, and the sum of the predicted displacement and the compensated displacement may be determined as the new predicted displacement.

[0083] S204: based on the new predicted displacement, extract the target chassis image from the surround transparent chassis image at the previous moment.

[0084] In some embodiments, after determining the new predicted displacement, the position of the vehicle chassis area at the current moment corresponding to the surround-view transparent chassis image at the previous moment can be determined based on the new predicted displacement, thereby intercepting the surround-view transparent chassis image at the previous moment to obtain the target chassis image.

[0085] According to the above technical means, the predicted displacement determined by the Kalman filter algorithm is corrected by the delay time from the vehicle sensor collecting the actual vehicle driving data to the on-board electronic equipment obtaining the measured vehicle driving data and the compensation displacement determined by the first predicted vehicle speed, so that the target chassis image finally captured from the surround transparent chassis image at the previous moment can be more accurate.

[0086] In some embodiments of the present application, based on the predicted vehicle driving data, the target chassis image is captured from the surround transparent chassis image at the previous moment, that is, the above-mentioned step S103 can be implemented by the following steps S1031A to S1033A, and each step is described separately below.

[0087] S1031A. Based on the predicted vehicle driving data, determine the position information of the four vertices of the vehicle at the current moment in the surround transparent chassis image at the previous moment.

[0088] In some embodiments, the predicted vehicle driving data may include the distance traveled by the vehicle from the previous moment to the current moment, that is, the predicted displacement. The predicted displacement may be the displacement of the center of the rear axle of the vehicle. Based on the predicted displacement, the position information of the four vertices of the vehicle at the current moment in the surround transparent chassis image at the previous moment can be determined.

[0089] In some embodiments, the position information of the four vertices of the vehicle at the current moment in the surround transparent chassis image at the previous moment can be the position coordinates of the four vertices of the vehicle. For example, a rectangular coordinate system is established with the center of the rear axle of the vehicle at the previous moment as the origin, and the rotation center of the vehicle or the rear axle center of the vehicle is determined. The position coordinates of the four vertices of the vehicle at the current moment are determined under the rectangular coordinate system and the rotation center.

[0090] S1032A: Determine a first area from the surround transparent chassis image at the last moment according to the position information.

[0091] In some embodiments, after determining the position information of the four vertices of the vehicle at the current moment in the surround-view transparent chassis image at the previous moment, the first area can be determined from the surround-view transparent chassis image at the previous moment based on the position information and the position of the center of the vehicle's rear axle relative to the four vertices of the vehicle. The first area can be the area within the rectangle formed by the four vertices of the vehicle, that is, the area corresponding to the chassis of the vehicle.

[0092] S1033A: intercept the image corresponding to the first area to obtain the target chassis image.

[0093] In some embodiments, after determining the first area corresponding to the surround transparent chassis image of the four vertices of the vehicle at the previous moment, the image corresponding to the first area can be cut out to obtain the target chassis image.

[0094] According to the above technical means, the position information of the four vertices of the vehicle at the current moment in the surround-view transparent chassis image at the previous moment is determined by predicting the vehicle driving data, so that the target chassis image can be intercepted from the surround-view transparent chassis image at the previous moment according to the position information, thereby realizing the acquisition of the transparent chassis image.

[0095] In some embodiments of the present application, the predicted vehicle driving data includes the predicted displacement of the vehicle from the previous moment to the current moment. Based on this, based on the predicted vehicle driving data, the position information of the four vertices of the vehicle at the current moment in the previous surround transparent chassis image frame is determined, that is, the above-mentioned step S1031A can be implemented by the following steps S301 to S302, and each step is explained separately below.

[0096] S301. Determine the rotation angle of the vehicle at the current moment relative to the previous moment based on the predicted displacement.

[0097] In some embodiments, the predicted displacement may be the distance traveled by the center position of the vehicle's rear axle from the previous moment to the current moment. If the wheels of the vehicle rotate from the previous moment to the current moment, the predicted displacement may be the arc length of the movement of the center position of the vehicle's rear axle from the previous moment to the current moment.

[0098] In some embodiments, after the predicted displacement is determined, the rotation angle of the rear axle center position of the vehicle at the current moment relative to the previous moment can be determined based on the predicted displacement and the rotation radius of the rear axle center position of the vehicle from the previous moment to the current moment. The rotation angle can be radians, and the rotation angle can be obtained by dividing the predicted displacement by the rotation radius based on the relationship between the arc length, radius and radians.

[0099] S302: Determine the position information of the four vertices of the vehicle in the previous surround transparent chassis image frame at the current moment according to the rotation angle and the position information of the vehicle rotation center.

[0100] In some embodiments, the position information of the vehicle's rotation center may be coordinate information of the vehicle's rotation center in a rectangular coordinate system established with the center position of the vehicle's rear axle at a previous moment as the origin. The position information of the vehicle's rotation center at the current moment in the rectangular coordinate system may be determined based on the rotation radius. The position information may be the position coordinates of the rotation center. Based on the position coordinates of the rotation center and the rotation angle, the position information of the four vertices of the vehicle at the current moment in the previous surround transparent chassis image frame may be determined.

[0101] For example, Figure 2 As shown in the figure, if the rotation radius is R, the position coordinates of the rotation center relative to the center position of the rear axle of the vehicle at the previous moment (i.e., the origin of the rectangular coordinate system) are (x0, y0), and the rotation angle is θ, then the position coordinates (x', y') of the four vertices of the vehicle can be calculated by formula (2):

[0102]

[0103] Wherein, x and y represent the horizontal and vertical coordinates of the center position of the rear axle of the vehicle at the current moment, respectively, x=tanθ·R, y=0.

[0104] According to the above technical means, the rotation angle of the vehicle at the current moment relative to the previous moment can be determined by predicting the displacement, and the position information of the vehicle's rotation center can be further determined based on the rotation angle, so the position information of the four vertices of the vehicle at the current moment in the previous surround transparent chassis image frame can be accurately calculated.

[0105] In some embodiments of the present application, based on the predicted vehicle driving data, the target chassis image is captured from the surround transparent chassis image at the previous moment, that is, step S103 can be implemented by the following steps S1031B to S1032B, and each step is described separately below.

[0106] S1031B. According to the predicted vehicle driving data, a reference chassis image is extracted from the surround transparent chassis image at the previous moment.

[0107] In some embodiments, the distance moved by the center position of the rear axle of the vehicle from the previous moment to the current moment can be determined based on the predicted vehicle driving data. The position information of the vehicle at the current moment in the surround-view transparent chassis image at the previous moment can be determined based on the moving distance and the angle of rotation of the vehicle from the previous moment to the current moment. The first area to be intercepted from the surround-view transparent chassis image at the previous moment is determined based on the position information. The image content corresponding to the first area is intercepted from the surround-view transparent chassis image at the previous moment, and a reference chassis image can be obtained.

[0108] S1032B: If there is a block shadow area in the reference chassis image, the surround transparent chassis image at the previous moment is re-captured to obtain the target chassis image.

[0109] In some embodiments, the block shadow area may be a shadow area existing in the surround-view transparent chassis image obtained at the previous moment due to sunlight or other illumination. If it is determined that the captured reference chassis image has a block shadow area, the surround-view transparent chassis image at the previous moment may be re-captured to obtain the target chassis image.

[0110] In some embodiments, since there are block shadow areas in the reference chassis image, in order to ensure that a clear transparent chassis image can be obtained later, the range of capture can be expanded during the re-capture of the surround transparent chassis image at the previous moment, that is, the capture can be performed outside the first area determined according to the four vertex positions of the vehicle to obtain a target chassis image, which does not include block shadow areas and has a size larger than that of the reference chassis image.

[0111] In some embodiments, in the process of re-capturing the surround-view transparent chassis image of the previous moment, only the area on the vehicle side corresponding to the block shadow area (for example, the area in front of the vehicle) can be expanded, or the corresponding areas around the vehicle can be expanded. The range of expansion and capture can be determined according to the size or number of pixels of the surround-view transparent chassis image of the previous moment. For example, the first area can be expanded outward by 10 pixels or 20 pixels for capture. The range of re-capturing the surround-view transparent chassis image of the previous moment here is only an exemplary description, and the present application does not limit this.

[0112] According to the above-mentioned technical means, when the reference chassis image includes block shadows, the surround transparent chassis image of the previous moment is re-captured, so that the target chassis image includes more visible or more image content, so that the block shadows can be excluded from the actual vehicle chassis area in the subsequent stitching process, and a clearer vehicle chassis area image can be obtained.

[0113] In some embodiments of the present application, a stitching process is performed based on the target chassis image and the surround chassis image at the current moment to obtain a surround transparent chassis image corresponding to the current moment, that is, the above-mentioned step S104 can be implemented by the following steps S1041 to S1045, and each step is described separately below.

[0114] S1041. Enlarge the target chassis image by a preset multiple to obtain a first candidate image.

[0115] In some embodiments, the preset multiple may be any multiple value, for example, it may be 0.5 times, 1 times, 1.5 times, etc. of the target chassis image. By magnifying the target chassis image by the preset multiple, a first candidate image with more pixels may be obtained. In the process of magnifying the target chassis image, the pixels may be expanded by a pixel interpolation method, thereby obtaining a first candidate image with richer pixels. The size of the first candidate image is larger than the size of the target chassis image, and the number of pixels of the first candidate image is also greater than the number of pixels of the target chassis image.

[0116] S1042: Determine the area where the enlarged surround-view transparent chassis image corresponding to the previous moment overlaps with the surround-view chassis image at the current moment.

[0117] In some embodiments, the enlarged surround transparent chassis image corresponding to the previous moment may be a surround transparent chassis image obtained by enlarging the surround transparent chassis image corresponding to the previous moment, for example, the surround transparent chassis image of the previous moment is enlarged by a preset multiple to obtain the enlarged surround transparent chassis image, and the surround chassis image of the current moment may be the original or unenlarged surround chassis image of the current moment. The enlarged surround transparent chassis image corresponding to the previous moment may be obtained by splicing the transparent chassis surround images before the previous moment.

[0118] In some embodiments, the surround view chassis image at the current moment may be rotated relative to the enlarged surround view transparent chassis image corresponding to the previous moment. In the process of determining the overlapping area between the enlarged surround view transparent chassis image corresponding to the previous moment and the surround view chassis image at the current moment, the surround view chassis image at the current moment can be enlarged to the same size as the enlarged surround view transparent chassis image corresponding to the previous moment, so as to better determine the overlapping area between the two images.

[0119] S1043: perform splicing processing on the image in the overlapping area and the first candidate image to obtain a second candidate image.

[0120] In some embodiments, after determining the overlapping area between the enlarged surround transparent image corresponding to the previous moment and the surround transparent chassis image at the current moment, the image corresponding to the overlapping area can be cut out, and then the image corresponding to the new area and the first candidate image are spliced ​​to obtain the second candidate image.

[0121] S1044: Reduce the second candidate image by a preset multiple to obtain a third candidate image.

[0122] In some embodiments, the image size of the second candidate image is a preset multiple of the size of the surround-view chassis image corresponding to the current moment. Therefore, after the second candidate image is obtained, the size of the second candidate image needs to be adjusted. By reducing the second candidate image by a preset multiple, a third candidate image with the same size as the surround-view chassis image corresponding to the current moment can be obtained.

[0123] S1045: perform splicing processing on the third candidate image and the surround-view chassis image at the current moment to obtain a surround-view transparent chassis image corresponding to the current moment.

[0124] In some embodiments, in the process of stitching the third candidate image and the surround view chassis image at the current moment, the image of the vehicle chassis area in the third candidate image and the image of the vehicle chassis area in the surround view chassis image at the current moment can be fused, and the image of the vehicle surround view area in the third candidate image and the image of the vehicle surround view area in the surround view chassis image at the current moment can be fused.

[0125] According to the above technical means, by magnifying the captured target chassis image by a preset multiple, the third candidate image obtained by reducing the second candidate image can retain more pixel content, thereby effectively reducing the image blur in the transparent chassis area that is common when the vehicle is at low speed, and making the final surround transparent chassis image at the current moment clearer.

[0126] In some embodiments of the present application, the third candidate image and the surround-view chassis image at the current moment are spliced ​​to obtain the surround-view transparent chassis image corresponding to the current moment, that is, step S1045 can be implemented by the following steps S401 to S403, and each step is described separately below.

[0127] S401, determining a first weight of a corresponding image pixel value in a vehicle surround view area in a third candidate image, and a second weight of a corresponding image pixel value in a vehicle surround view area in a surround view chassis image at a current moment.

[0128] In some embodiments, both the first weight and the second weight may be pre-set weight values. The first weights corresponding to different image pixel values ​​in the vehicle's surround view area may be different, and the second weights may also be different. The first weight of image pixels close to the vehicle chassis area is greater than the second weight, and the first weight may be greater than, less than, or equal to the second weight. For the same image pixel value in the vehicle's surround view area, the sum of the first weight and the second weight may be 1. For example, the first weight may be 0.2 and the second weight may be 0.8. The values ​​of the first weight and the second weight here are merely exemplary and are not limited in this application.

[0129] In some embodiments, since the third candidate image is determined based on the surround-view transparent chassis image at the previous moment, the scene content of the area near the transparent chassis (such as the vehicle surround-view area) will change during the vehicle's travel from the previous moment to the current moment. Therefore, in the process of splicing the third candidate image and the surround-view chassis image at the current moment, the image pixel values ​​corresponding to the vehicle surround-view area in the surround-view chassis image at the current moment can be assigned a higher weight than the corresponding image pixel values ​​in the vehicle surround-view area in the third candidate image, so that the final spliced ​​surround-view transparent chassis image corresponding to the current moment is more accurate.

[0130] S402: Determine an image of the vehicle surround view area at the current moment based on the first weight, the second weight, the image pixel values ​​within the vehicle surround view area in the third candidate image, and the image pixel values ​​within the vehicle surround view area in the surround view chassis image at the current moment.

[0131] In some embodiments, for the same pixel point in the vehicle's surround view area, the product of the image pixel value in the third candidate image and the first weight, and the sum of the product of the image pixel value in the surround view chassis image at the current moment and the second weight can be determined as the image pixel value corresponding to the pixel point. By performing this processing on the pixel value of each pixel point in the vehicle's surround view area in turn, the image of the vehicle's surround view area at the current moment can be obtained.

[0132] S403: splicing the image of the vehicle surround area and the image corresponding to the vehicle chassis area in the third candidate image to obtain a surround transparent chassis image at the current moment.

[0133] In some embodiments, the image corresponding to the vehicle chassis area in the third candidate image includes visible image content. Therefore, in the process of stitching the surround-view chassis image at the current moment and the third candidate image, the image of the vehicle chassis area in the third candidate image can be determined as the content of the chassis area in the surround-view transparent chassis image at the current moment. By stitching the image corresponding to the vehicle chassis area in the third candidate image and the image of the vehicle surround-view area, a complete surround-view transparent chassis image at the current moment can be obtained.

[0134] According to the above technical means, the image pixel values ​​of the vehicle surround view area of ​​the third candidate image and the surround view chassis image at the current moment are assigned different weights for fusion, so that the image content in the vehicle surround view area in the final transparent chassis image at the current moment is closer to the actual scene. Thereafter, the image of the vehicle surround view area obtained after the fusion and the image of the chassis area in the surround view chassis image at the current moment are spliced ​​to obtain a surround view transparent chassis image with a good display effect.

[0135] In some embodiments of the present application, the predicted vehicle driving data includes a first predicted vehicle speed. Based on this, after splicing the target chassis image and the surround chassis image at the current moment to obtain the surround transparent chassis image at the current moment, that is, after step S104, the following steps S501 to S502 can also be executed, and each step is described below.

[0136] S501: Determine the sharpness value of the surround view transparent chassis image at the current moment according to the first predicted vehicle speed and a preset correspondence between the predicted vehicle speed and the sharpness value of the surround view transparent chassis image.

[0137] In some embodiments, a preset correspondence between the predicted vehicle speed and the sharpness value of the surround-view transparent chassis image can be created in advance, and the preset correspondence can be represented by a preset correspondence table between the predicted vehicle speed and the sharpness value of the surround-view transparent chassis image. Different predicted vehicle speeds correspond to different sharpness values. When the first predicted vehicle speed corresponding to the current moment is determined, the sharpness value of the surround-view transparent chassis image at the current moment can be found from the preset relationship correspondence table.

[0138] In some embodiments, if the vehicle is traveling at a low speed, multiple sharpening of the surround-view transparent chassis image may cause the obtained surround-view transparent chassis image to become blurred. Therefore, the surround-view transparent chassis image sharpness value corresponding to a small predicted vehicle speed value may be smaller than the surround-view transparent chassis image sharpness value corresponding to a large predicted vehicle speed value. For example, if the predicted vehicle speed A is greater than the predicted vehicle speed B, the surround-view transparent chassis image sharpness value corresponding to A is greater than the surround-view transparent chassis image sharpness value corresponding to B.

[0139] S502: Based on the sharpening degree value, the surround-view transparent chassis image at the current moment is sharpened to obtain the target surround-view transparent chassis image at the current moment.

[0140] In some embodiments, since the pixels in the surround-view transparent chassis image are continuously interpolated, the image will gradually become blurred. Therefore, by sharpening the surround-view transparent chassis image at the current moment, the boundaries between different objects in the scene (such as road obstacles) can be increased, thereby increasing the clarity of the surround-view transparent chassis image.

[0141] According to the above technical means, by determining the sharpness value of the surround-view transparent chassis image corresponding to the first predicted vehicle speed, and sharpening the surround-view transparent chassis image at the current moment based on the sharpness value, the blur problem caused by the previous enlargement of the surround-view transparent chassis image can be solved, and the clarity of the surround-view transparent chassis image at the current moment can be improved.

[0142] In the embodiment of the present application, the surround chassis image of the vehicle at the current moment and the measured vehicle driving data corresponding to the vehicle at the current moment are obtained; the measured vehicle driving data are filtered by using a filtering algorithm to obtain the predicted vehicle driving data corresponding to the vehicle at the current moment; the target chassis image is intercepted from the surround transparent chassis image at the previous moment according to the predicted vehicle driving data; the surround transparent chassis image at the current moment is obtained by splicing the target chassis image and the surround chassis image at the current moment. In this way, by filtering the measured vehicle driving data by using a filtering algorithm, the noise in the measured vehicle data is removed, so that the corresponding position information of the vehicle at the current moment in the surround transparent image at the previous moment can be more accurately determined based on the preset vehicle driving data obtained after filtering, thereby obtaining a more accurate target chassis image, avoiding the splicing misalignment of the target chassis image and the surround chassis image at the current moment, and improving the display effect of the surround transparent chassis image at the current moment finally obtained.

[0143] Next, the implementation process of the application embodiment in the actual application scenario is introduced.

[0144] Figure 3 A flowchart of a method for implementing a transparent chassis based on multi-data fusion provided in an embodiment of the present application is provided. The method can be implemented through the following steps S601 to S608, and each step is described separately below.

[0145] S601, obtaining the previous frame of vehicle data.

[0146] In some embodiments, the vehicle driving data of the previous frame includes data such as wheel speed pulse, steering wheel angle, and vehicle speed.

[0147] S602: Use the Kalman filter algorithm to fuse the vehicle data to obtain the vehicle speed and the displacement of the center position of the rear axle of the vehicle (equivalent to the "predicted displacement" in other embodiments).

[0148] In some embodiments, vehicle speed data can be added to the wheel speed pulse data, and the Kalman filter algorithm can be used to fuse the above vehicle data to obtain a more accurate speed of the rear axle center, and then more accurately estimate the movement distance between each frame, and then more accurately calculate the positions of the four vertices to avoid the problem of transparent chassis splicing misalignment caused by data jitter.

[0149] In some embodiments, the distance and speed can be estimated more accurately based on the measured distance (wheel speed pulse sensor) and speed (other sensors such as IMU) after Kalman filtering integration and filtering. The data itself is more accurate and smoother. The Kalman filtering algorithm uses the model shown in formula (4):

[0150]

[0151] In this scheme, since there is no output u, B·u is omitted, and the prediction formula and update formula of Kalman filter are respectively expressed as formula (1) and formula (2). The state acquisition formula is as formula (5):

[0152]

[0153] Among them, x is the state quantity, x={s1,v1,a1}, s1,v1,a1 They represent the displacement distance, displacement speed and displacement acceleration of the center position of the rear axle of the vehicle respectively; Z is the measured value, Z = {s2, v2}, s2 represents the displacement of the center of the rear axle obtained by taking the average of the two rear wheel speed pulses of the vehicle, and v2 is the vehicle speed measured by other on-board sensors such as (inertial sensors); D is the delay matrix, D = {{0, delay, 0.5*delay} 2},{0,1,delay},{0,0,1}}, represents the vehicle state after time delay in the current state. Delay is equal to the signal delay from the actual vehicle signal to the algorithm, which is used to avoid inaccurate data calculation caused by signal delay.

[0154] S603, performing delayed state processing on the displacement of the center position of the rear axle of the vehicle according to the vehicle speed to obtain a compensation displacement (equivalent to the "new predicted displacement" in other embodiments), and determining the rotation matrix corresponding to the common motion trajectory of each point in the chassis area according to the compensation displacement.

[0155] In some embodiments, the displacement of the center position of the rear axle of the vehicle can be compensated for the delay state according to the delay error, and a more accurate motion arc length value can be obtained by adding a certain compensation value to the displacement of the center position of the rear axle of the vehicle. Then, a rotation matrix is ​​determined according to the motion arc length value, and the rotation matrix may include a rotation angle and a rotation radius corresponding to the rotation center.

[0156] S604. Determine the position coordinates of the four vertices of the vehicle according to the rotation matrix (equivalent to “the position information of the four vertices of the vehicle in the surround transparent chassis image at the previous moment” in other embodiments).

[0157] In some embodiments, the position coordinates of the four vertices in the previous frame top view at the current moment can be calculated based on the motion arc length value, rotation angle, rotation radius and other data. The basic principle of the transparent chassis algorithm is to calculate the position of the vehicle at a certain moment based on the known driving parameters, so as to find the corresponding area of ​​the vehicle chassis in the pre-stored historical frame image, and fill the content of the area into the vehicle chassis area at the current moment. Figure 2As shown in the figure, the instantaneous motion of the vehicle is analyzed based on the center position of the vehicle's rear axle. The instantaneous motion of the entire vehicle can be approximated as a circular motion. All points of the vehicle rotate around an external point at the same angular velocity, and this point is the center of rotation. When the arc length of the vehicle's rear axle center, wheelbase L, steering wheel angle δ0, and angular transmission ratio parameter α are known, we can calculate the corresponding positions of the four vertices of the vehicle in the previous frame image at this moment.

[0158] Assuming that the radius corresponding to the instantaneous rotation center O0 when the rear wheel center point makes a circular motion is R, a coordinate system is established with the rear axle center as the origin, and the position of the rotation center O0 relative to the center of the vehicle's rear wheel (x0, y0) can be expressed as (-R, 0); based on the known wheel angle δ and the vehicle's front and rear wheelbase L, the rotation radius R = L / tan(δ), where the wheel angle δ can be obtained based on the square plate angle δ0 and the angular transmission ratio parameter α: δ = α·δ0; then, based on the movement distance of the vehicle's rear axle center between the two frames as the arc length S, when the rotation arc length S and radius R are known, the rotation arc θ between the two frames can be obtained; when the rotation arc θ is known and the position of the vehicle's rear wheel center (x0, y0), since the coordinates of the vehicle's four vertices relative to the rear axle center are known, the coordinates of the vehicle's four vertices after the rotation arc θ can be calculated by formula (3).

[0159] Through the above steps, we can get the corresponding positions of the four vertices of the vehicle in the previous frame image at this moment, capture the image of the area contained in the four vertices of the vehicle chassis, and draw it in the vehicle chassis area of ​​the current frame to achieve the effect of the road surface outside the chassis area entering the chassis area. After the processing is completed and displayed, the current frame becomes the old frame and participates in the calculation of the next frame, and so on.

[0160] The above is the basic core idea of ​​the transparent chassis algorithm. Since the coordinates of each point in the chassis area of ​​each frame are fixed for the entire top view, the coordinates of the four vertices of the chassis area at the corresponding moment of the "next frame" relative to the current top view frame can be calculated according to the above theory, so that the rendering interface can be used to intercept the area and fill it into the chassis area of ​​the "next frame". That is, a double buffer is required to store the top view image of the previous frame and the top view image of the current frame respectively, and after the current frame is displayed, it is stored as the previous frame.

[0161] S605. According to the position coordinates of the four vertices of the vehicle, a chassis area image corresponding to the current time point is obtained from the overhead image of the previous frame with a multiple size (equivalent to the "target chassis image" in other embodiments).

[0162] Before explaining this step, there are two prerequisites that need to be explained:

[0163] Prerequisite 1: Save the top view image of the previous frame that is larger than the current size for use in the next frame to obtain the chassis image, such as a top view image of 1.5 times or 2 times the size. It should be emphasized that a top view larger than the normal top view image size is used for the subsequent processing of cropping the chassis area. This solution can obtain more effective pixels when shrinking to the normal size compared to the ordinary method, thereby effectively reducing the common transparent chassis area image blurring at low speeds;

[0164] Premise 2: Expand the intercepted area to reduce the block shadows caused by direct sunlight, that is, set a "pseudo chassis area" slightly larger than the actual chassis area to exclude block shadows from the actual chassis area.

[0165] In some embodiments, a rendering interface can be used to capture an image of the chassis area of ​​the current frame in the top view image of the previous frame. At this time, the image size is enlarged to be greater than 1 times the size, and the captured area is also slightly larger than the actual chassis area as mentioned in premise 2, such as expanding outward by 50 pixels.

[0166] S606 . Dynamically sharpen the chassis area image corresponding to the current time point according to the vehicle speed.

[0167] Since the image of the chassis area is rotated in each frame, the corresponding pixels are continuously interpolated and saved, thus becoming blurred. Appropriate sharpening can restore clarity, and determining the degree of sharpening based on vehicle speed can ensure that the same clarity is maintained at different vehicle speeds, because the same area will be rotated and sharpened more times at low speeds than at high speeds.

[0168] S607: Render the sharpened chassis area image to the vacant chassis area on the current top view to obtain a chassis area image of a multiple size corresponding to the current time point.

[0169] In some embodiments, the chassis area image of multiple sizes obtained by sharpening after interception can be spliced ​​to the vacant chassis area on the current top view, so as to obtain the chassis area image of multiple sizes corresponding to the current time point.

[0170] S608. Scale the chassis area image of multiple sizes corresponding to the current time point to obtain a chassis area image of normal size corresponding to the current time point (equivalent to the "surrounding transparent chassis image corresponding to the current moment" in other embodiments).

[0171] In some embodiments, by scaling the chassis area image of a multiple size corresponding to the current time, the obtained chassis area image can be made to have the same size as the normal chassis area image.

[0172] Exemplarily, four buffers can be used as buffers for image calculation, buffer 1 and buffer 2 are buffers of the size of the top view area, buffer 3 and buffer 4 are buffers several times the size of the chassis area (including pixels expanded outward), and according to the previously calculated vehicle vertex coordinates, the rendering process is as follows: according to the calculated vehicle vertex coordinates, the image in the historical frame buffer 1 is intercepted and stretched and stored in buffer 3; according to the previously calculated vehicle displacement, the overlapping area between the image in the historical frame buffer 4 and the current vehicle is calculated, and the image of the overlapping area is copied to buffer 3; after scaling the entire buffer 3 image, the image is copied to buffer 2, and superimposed and fused with the weight map prepared in advance, the weight map makes the image pre-fetched on the chassis completely rotate into the chassis area, and the screenshot outside the chassis is fused with the image of the same area in the current frame using weights, so that the splicing transition between the two frames is smooth to reduce the presence of water ripples; in the next frame calculation, the identifiers of the current frame and the historical frame of buffer 1 and 2 are exchanged, and the identifiers of the current frame and the historical frame of buffer 3 and 4 are exchanged.

[0173] Through the above rendering process, after the image is captured from the top view, it is placed in a larger buffer for subsequent displacement operations, which can greatly reduce the loss of image accuracy during the displacement process, because the displacement is not always a complete translation. When rotation is involved, many pixels will be interpolated and displayed. When continuously rotated, they will be continuously interpolated. Using a larger buffer can minimize the loss of accuracy in interpolation, thereby maximizing the reduction of blur.

[0174] For example, Figure 4 The figure shows a schematic diagram of the effect of chassis splicing optimization provided by the present application, from Figure 4 It can be seen from the figure that the splicing method provided by the present application can obtain a transparent chassis image with no splicing misalignment and clearer clarity; Figure 5 As shown in the figure, it is a schematic diagram of the effect of fuzzy optimization of the chassis area image. Figure 5 It can be seen that the chassis area image can be made clearer by sharpening, intercepting the chassis image at multiple times the size, and other processing methods; Figure 6 As shown in FIG. 1 , a schematic diagram of the optimization effect obtained by a transparent chassis splicing method based on multi-data fusion provided by the present application is shown. Figure 6 It can be seen that through fuzzy optimization and stitching misalignment optimization, the final transparent chassis image can be made clearer and there is no problem of stitching misalignment.

[0175] The implementation method of the transparent chassis based on multi-data fusion provided in the present application avoids the problem of transparent chassis splicing misalignment caused by data jitter by adopting the Kalman filter algorithm to process multiple data processing methods; solves the splicing misalignment problem caused by data delay by increasing the constant of delay state prediction; uses a large image to calculate a small image display solution to avoid blurring caused by loss of accuracy of the chassis area when the vehicle is driving at low speed; and dynamically sharpens the chassis area image according to the vehicle speed, further optimizing the blurring problem of the transparent chassis image.

[0176] The present application provides a device for splicing vehicle surround transparent chassis images. Figure 7 A schematic diagram of the structure of a vehicle surround view transparent chassis image splicing device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the splicing device 700 for the vehicle surround transparent chassis image includes:

[0177] The data acquisition module 701 is used to acquire the surround chassis image of the vehicle at the current moment, and the measured vehicle driving data corresponding to the vehicle at the current moment;

[0178] A filtering processing module 702 is used to filter the measured vehicle driving data using a filtering algorithm to obtain predicted vehicle driving data corresponding to the vehicle at the current moment;

[0179] An image capture module 703 captures a target chassis image from the previous surround transparent chassis image according to the predicted vehicle driving data;

[0180] The stitching processing module 704 is used to perform stitching processing based on the target chassis image and the surround chassis image at the current moment to obtain the surround transparent chassis image at the current moment.

[0181] In some embodiments, the measured vehicle driving data includes the measured displacement of the vehicle, and the predicted vehicle driving data includes the predicted displacement of the vehicle from the previous moment to the current moment; the filtering algorithm includes a Kalman filtering algorithm; and the filtering processing module 702 includes:

[0182] The first creation submodule is used to create a state transfer matrix and a state observation matrix of a Kalman filter;

[0183] A first determination submodule is used to determine a Kalman update formula according to the state transfer matrix, the state observation matrix, a preset Kalman prediction formula and a covariance matrix;

[0184] The first updating module is used to update the measured displacement based on the Kalman update formula to obtain the predicted displacement.

[0185] In some embodiments, the predicted vehicle driving data further includes a first predicted vehicle speed; the vehicle surround transparent chassis image stitching device 700 further includes:

[0186] A first determination module is used to determine the delay time from when the vehicle sensor collects the measured vehicle driving data to when the on-board electronic device obtains the measured vehicle driving data;

[0187] A second determination module, configured to determine a compensation displacement according to the delay duration and the first predicted vehicle speed;

[0188] A first correction module, configured to correct the predicted displacement based on the compensation displacement to obtain a new predicted displacement;

[0189] The first interception module is used to intercept the target chassis image from the surround transparent chassis image at the previous moment based on the new predicted displacement.

[0190] In some embodiments, the image capture module 703 includes:

[0191] A second determination submodule is used to determine, based on the predicted vehicle driving data, position information of four vertices of the vehicle at a current moment in the surround transparent chassis image at a previous moment;

[0192] A third determining submodule is used to determine the first area from the surround transparent chassis image at the last moment according to the position information;

[0193] The first interception submodule is used to intercept the image corresponding to the first area to obtain the target chassis image.

[0194] In some embodiments, the predicted vehicle driving data includes the predicted displacement of the vehicle from the previous moment to the current moment; the second determination submodule includes:

[0195] a first determining unit, configured to determine a rotation angle of the vehicle at a current moment relative to a previous moment based on the predicted displacement;

[0196] The second determining unit is used to determine the position information of the four vertices of the vehicle in the previous surround transparent chassis image frame at the current moment according to the rotation angle and the position information of the vehicle rotation center.

[0197] In some embodiments, the image capture module 703 includes:

[0198] A second interception submodule is used to intercept a reference chassis image from the surround transparent chassis image at the last moment according to the predicted vehicle driving data;

[0199] The third capture submodule is used to re-capture the surround transparent chassis image at the previous moment to obtain the target chassis image if there is a block shadow area in the reference chassis image; the target chassis image does not include the block shadow area, and the size of the target chassis image is larger than that of the reference chassis image.

[0200] In some embodiments, the splicing processing module 704 includes:

[0201] A first acquisition submodule, used for magnifying the target chassis image by a preset multiple to obtain a first candidate image;

[0202] A third determining submodule is used to determine an area where the enlarged surround transparent chassis image corresponding to the previous moment overlaps with the surround chassis image at the current moment;

[0203] A first stitching submodule, used for stitching the image of the overlapped area and the first candidate image to obtain a second candidate image;

[0204] A first image size processing submodule, used for reducing the second candidate image by the preset multiple to obtain a third candidate image;

[0205] The second stitching submodule is used to stitch the third candidate image and the surround-view chassis image at the current moment to obtain the surround-view transparent chassis image corresponding to the current moment.

[0206] In some embodiments, the second splicing submodule includes:

[0207] A third determining unit is used to determine a first weight of a corresponding image pixel value in the vehicle surround view area in the third candidate image, and a second weight of a corresponding image pixel value in the vehicle surround view area in the surround view chassis image at the current moment; the first weight of the image pixel close to the vehicle chassis area is greater than the second weight;

[0208] A fourth determining unit, configured to determine an image of the vehicle surround view area at the current moment based on the first weight, the second weight, the image pixel values ​​within the vehicle surround view area in the third candidate image, and the image pixel values ​​within the vehicle surround view area in the surround view chassis image at the current moment;

[0209] The first splicing processing unit is used to splice the image of the vehicle surround area and the image corresponding to the vehicle chassis area in the third candidate image to obtain the surround transparent chassis image at the current moment.

[0210] In some embodiments, the predicted vehicle driving data includes a first predicted vehicle speed; the vehicle surround transparent chassis image stitching device 700 includes:

[0211] A third determination module, configured to determine the sharpness value of the surround view transparent chassis image at the current moment according to the first predicted vehicle speed and a preset correspondence between the predicted vehicle speed and the sharpness value of the surround view transparent chassis image;

[0212] The sharpening processing module is used to perform sharpening processing on the surround-view transparent chassis image at the current moment based on the sharpening degree value to obtain the target surround-view transparent chassis image at the current moment.

[0213] It should be noted that the description of the splicing device of the vehicle surround transparent chassis image in the embodiment of the present application is similar to the description of the corresponding method embodiment above, and has similar beneficial effects as the method embodiment, so it is not repeated. For technical details not disclosed in this embodiment, please refer to the description of the method embodiment of the present application for understanding.

[0214] It should be noted that in the embodiments of the present application, if the above-mentioned method for splicing the vehicle surround transparent chassis image is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant solution can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0215] Accordingly, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for stitching the vehicle surround view transparent chassis image provided in the above embodiment is implemented.

[0216] Figure 8 A schematic diagram of the structure of a vehicle surround transparent chassis image splicing device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the vehicle surround transparent chassis image stitching device 800 includes: a memory 801, a processor 802, a communication interface 803 and a communication bus 804. Among them, the memory 801 is used to store instructions for downloading executable vehicle upgrade files; the processor 802 is used to execute the executable vehicle surround transparent chassis image stitching instructions stored in the first memory 801 to implement the vehicle surround transparent chassis image stitching method provided in the above embodiment.

[0217] The description of the above vehicle surround transparent chassis image stitching device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the vehicle surround transparent chassis image stitching device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0218] It should be noted that, in this article, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises at least one ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0219] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0220] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0221] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0222] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, disks or optical disks.

[0223] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a product to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0224] The above are only implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technical object familiar with the technical field can be easily thought of within the technical scope disclosed in the present application. Changes or substitutions should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for stitching vehicle surround transparent chassis images, characterized in that: include: Acquire a surround chassis image of the vehicle at the current moment, and actual vehicle driving data corresponding to the vehicle at the current moment; Filtering the measured vehicle driving data using a filtering algorithm to obtain predicted vehicle driving data corresponding to the vehicle at the current moment; According to the predicted vehicle driving data, a target chassis image is intercepted from the surround transparent chassis image at the last moment; The target chassis image and the surround-view chassis image at the current moment are stitched together to obtain the surround-view transparent chassis image at the current moment.

2. The method according to claim 1, characterized in that The measured vehicle driving data includes the measured displacement of the vehicle, and the predicted vehicle driving data includes the predicted displacement of the vehicle from the last moment to the current moment; the filtering algorithm includes a Kalman filtering algorithm; The method of using a filtering algorithm to perform prediction processing on the measured vehicle driving data to obtain predicted vehicle driving data corresponding to the vehicle at the current moment includes: Create the state transfer matrix and state observation matrix of Kalman filter; Determine a Kalman update formula according to the state transfer matrix, the state observation matrix, the preset Kalman prediction formula and the covariance matrix; The measured displacement is updated based on the Kalman update formula to obtain the predicted displacement.

3. The method according to claim 2, characterized in that The predicted vehicle driving data also includes a first predicted vehicle speed; the method further includes: Determine the delay time from when the vehicle sensor collects the measured vehicle driving data to when the on-board electronic device obtains the measured vehicle driving data; determining a compensation displacement according to the delay duration and the first predicted vehicle speed; Correcting the predicted displacement based on the compensation displacement to obtain a new predicted displacement; Based on the new predicted displacement, the target chassis image is intercepted from the surround transparent chassis image at the previous moment.

4. The method according to claim 1, characterized in that The method of extracting a target chassis image from the surround transparent chassis image at the last moment according to the predicted vehicle driving data includes: Based on the predicted vehicle driving data, determining the position information of the four vertices of the vehicle at the current moment in the surround transparent chassis image at the previous moment; According to the position information, determining a first area from the surround transparent chassis image at the last moment; An image corresponding to the first area is intercepted to obtain the target chassis image.

5. The method according to claim 4, characterized in that The predicted vehicle travel data includes the predicted displacement of the vehicle from the last moment to the current moment; The step of determining the position information of the four vertices of the vehicle at the current moment in the previous surround transparent chassis image frame based on the predicted vehicle driving data includes: Based on the predicted displacement, determining a rotation angle of the vehicle at a current moment relative to a previous moment; According to the rotation angle and the position information of the vehicle rotation center, the position information of the four vertices of the vehicle in the previous surround transparent chassis image frame is determined at the current moment.

6. The method according to claim 1, characterized in that The method of extracting a target chassis image from the surround transparent chassis image at the last moment according to the predicted vehicle driving data includes: According to the predicted vehicle driving data, a reference chassis image is intercepted from the surround transparent chassis image at the last moment; If there is a block shadow area in the reference chassis image, the surround transparent chassis image of the previous moment is re-captured to obtain the target chassis image; the target chassis image does not include the block shadow area, and the size of the target chassis image is larger than that of the reference chassis image.

7. The method according to claim 1, characterized in that The step of performing stitching processing based on the target chassis image and the surround chassis image at the current moment to obtain the surround transparent chassis image corresponding to the current moment includes: Enlarging the target chassis image by a preset multiple to obtain a first candidate image; Determine an area where the enlarged surround-view transparent chassis image corresponding to the previous moment overlaps with the surround-view chassis image at the current moment; Performing splicing processing on the image of the overlapping area and the first candidate image to obtain a second candidate image; Reducing the second candidate image by the preset multiple to obtain a third candidate image; The third candidate image and the surround-view chassis image at the current moment are spliced ​​to obtain a surround-view transparent chassis image corresponding to the current moment.

8. The method according to claim 7, characterized in that The step of performing splicing processing on the third candidate image and the surround-view chassis image at the current moment to obtain the surround-view transparent chassis image corresponding to the current moment includes: Determine a first weight of a corresponding image pixel value in the vehicle surround view area in the third candidate image, and a second weight of a corresponding image pixel value in the vehicle surround view area in the surround view chassis image at the current moment; the first weight of the image pixel close to the vehicle chassis area is greater than the second weight; Determine an image of the vehicle surround view area at the current moment based on the first weight, the second weight, the image pixel value in the vehicle surround view area in the third candidate image, and the image pixel value in the vehicle surround view area in the surround view chassis image at the current moment; The image of the vehicle surround view area and the image corresponding to the vehicle chassis area in the third candidate image are spliced ​​to obtain the surround view transparent chassis image at the current moment.

9. The method according to claim 1, characterized in that: The predicted vehicle travel data includes a first predicted vehicle speed; the method further includes: Determining the sharpness value of the surround-view transparent chassis image at the current moment according to the first predicted vehicle speed and a preset correspondence between the predicted vehicle speed and the sharpness value of the surround-view transparent chassis image; Based on the sharpening degree value, the surround-view transparent chassis image at the current moment is sharpened to obtain the target surround-view transparent chassis image at the current moment.

10. A device for splicing vehicle surround transparent chassis images, characterized in that: include: A data acquisition module, used to acquire a surround chassis image of the vehicle at the current moment, and actual vehicle driving data corresponding to the vehicle at the current moment; A filtering processing module, used to filter the measured vehicle driving data using a filtering algorithm to obtain predicted vehicle driving data corresponding to the vehicle at the current moment; An image capture module, which captures a target chassis image from the surround transparent chassis image at the last moment according to the predicted vehicle driving data; The splicing processing module is used to perform splicing processing based on the target chassis image and the surround chassis image at the current moment to obtain the surround transparent chassis image at the current moment.

11. A device for splicing vehicle surround transparent chassis images, characterized in that: include: processor and memory; wherein, The memory is used to store a computer program that can be run on the processor; The processor is configured to execute the method according to any one of claims 1 to 9 when running the computer program.

12. A computer-readable storage medium, characterized in that: The storage medium stores computer program code, and when the computer program code is executed by a computer, the method according to any one of claims 1 to 9 is executed.

Images