Image processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides an image processing method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: predicting an output timestamp once every a preset period, then obtaining an environment image taking a vehicle position as an image center, and recording the obtained timestamp; then obtaining a timestamp and real-time pose information of the vehicle based on the predicted output timestamp, and calculating a target output field of view at the output timestamp; and intercepting a region image in a set region of the environment image, and processing the region image into a target output image according to the field of view of the region image and the target output field of view. According to the invention, the environment image is obtained by real-time updating based on the images acquired by the plurality of cameras carried by the vehicle, any visual image needing to be perceived in the driving process of the vehicle can be output based on the environment image, and decoupling between image input of the vehicle cameras and various visual output images is realized. There is no need to equip a plurality of independent visual systems for a vehicle in the prior art, thereby reducing the consumption of hardware resources.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] With the continuous advancement of automobile technology, in-vehicle vision systems play an increasingly important role in improving driving safety and convenience. At present, there are many in-vehicle vision-related functions on the market, such as 360-degree surround view system (capturing images through multiple cameras and processing these images to generate a bird's-eye view of the vehicle's surroundings), streaming rearview mirror (using cameras instead of traditional rearview mirrors to display real-time images behind the vehicle on the mirror inside the car), transparent chassis (using cameras installed under the vehicle to transmit images under the vehicle chassis to the in-vehicle display screen, allowing the driver to "see through" the situation under the chassis) and Advanced Driving Assistance System (ADAS), etc. These functions all rely on the acquisition and processing of surrounding environment images by in-vehicle cameras.

[0003] However, in existing in-vehicle vision implementations, each vision system is independently equipped with a camera and a processing unit. This independent configuration leads to a huge waste of hardware resources. Summary of the invention

[0004] The object of the present invention is to provide an image processing method, device, electronic device and computer-readable storage medium to improve the problems existing in the prior art.

[0005] The embodiments of the present invention can be implemented as follows:

[0006] In a first aspect, the present invention provides an image processing method, comprising:

[0007] After predicting the output timestamp once every preset period, an environmental image with the vehicle position as the image center is obtained and the acquisition timestamp is recorded; the environmental image is obtained by real-time updating based on the collected images of multiple cameras carried by the vehicle;

[0008] Based on the predicted output timestamp, the acquisition timestamp and the real-time posture information of the vehicle, a target output field of view at the output timestamp is calculated; the target output field of view is the field of view of any visual image that needs to be perceived during the vehicle's driving process;

[0009] A regional image is intercepted in a set area of ​​the environment image, and the regional image is processed into a target output image according to the field of view of the regional image and the target output field of view.

[0010] In an optional implementation manner, the updating process of the environment image includes:

[0011] Obtain the collected images and calculated optical flow information collected by each camera on the vehicle at the current moment;

[0012] For each of the collected images, adjusting the position and shape of the moving object in the collected image based on a preset time delay and optical flow information of the collected image, so as to obtain a delayed collected image corresponding to each collected image;

[0013] fusing each of the delayed acquisition images with different regions of the environment image at the previous moment to obtain the initial environment image at the current moment;

[0014] Based on the position information of the vehicle at the current moment and the previous moment, a movement operation is performed on the initial environment image to obtain the environment image at the current moment.

[0015] In an optional implementation, the step of adjusting the position and shape of the moving object in the collected image based on the preset time delay and the optical flow information of the collected image includes:

[0016] Based on the optical flow information of the collected image, searching for the original object contour of at least one moving object in the collected image; the optical flow information of the collected image includes an optical flow vector of each pixel in the image, and the optical flow vector represents the moving speed of the pixel;

[0017] For each target pixel point within any of the original object contours, calculating a displacement distance based on the preset time delay and an optical flow vector of the target pixel point;

[0018] From the collected image, find a pixel point to be determined that is spaced from the target pixel point by the displacement distance in the direction indicated by the optical flow vector thereof;

[0019] After replacing the pixel value of the undetermined pixel point with the pixel value of the target pixel point, the pixel value of the target pixel point is set to zero;

[0020] Traversing each target pixel point of each original object contour to obtain an intermediate acquisition image;

[0021] For each pixel to be corrected whose pixel value is zero in the intermediate acquisition image, a non-zero pixel closest to the target pixel is searched in the intermediate acquisition image and the non-zero pixel is used to correct the pixel to be corrected, so as to obtain the delayed acquisition image.

[0022] In an optional implementation, the step of correcting the pixel to be corrected by using the non-zero pixel includes:

[0023] Replacing the pixel value of the pixel to be corrected with the pixel value of the non-zero pixel;

[0024] Acquire real-time pixel values ​​outside the edge of the moving object perceived by the camera corresponding to the captured image;

[0025] Based on a result of weighted summation of the current pixel value of the pixel to be corrected and the real-time pixel value, a final pixel value of the pixel to be corrected is determined.

[0026] In an optional implementation, the step of fusing each of the delayed acquisition images with different regions of the environment image at the previous moment to obtain the initial environment image at the current moment includes:

[0027] For any of the delayed acquisition images, calculating a local fusion area of ​​the delayed acquisition image within the environment image based on the camera viewing angle of the delayed acquisition image;

[0028] Based on the difference between the camera viewing angle of the delayed acquisition image and the imaging viewing angle of the local fusion area in the environmental image, the delayed acquisition image is deformed to obtain a fusion input image;

[0029] Fusing the fused input image with the image of the local fusion area in the environment image at the previous moment;

[0030] Each of the delayed acquisition images is traversed to obtain the initial environment image at the current moment.

[0031] In an optional implementation, the step of performing a movement operation on the initial environment image based on the respective position information of the vehicle at the current moment and the previous moment to obtain the environment image at the current moment includes:

[0032] Obtain the vehicle's historical position information at the previous moment and the current position information at the current moment;

[0033] Calculating the posture change information between the historical posture information and the current posture information;

[0034] Using the posture change information, constructing a movement transformation matrix;

[0035] The initial environment image is adjusted to the environment image at the current moment by using the movement transformation matrix.

[0036] In an optional embodiment, the step of calculating the target output field of view at the output timestamp based on the predicted output timestamp, the acquisition timestamp and the real-time posture information of the vehicle includes:

[0037] Calculating the time difference between the output timestamp and the acquisition timestamp;

[0038] Obtain the current vehicle speed and current vehicle position from the vehicle's real-time posture information;

[0039] Determining a current output field of view based on a current vehicle position;

[0040] Calculating the field of view movement distance based on the current vehicle speed and the time difference;

[0041] The current output field of view is translated in the direction of the current vehicle speed by the field of view movement distance to obtain the target output field of view at the output timestamp.

[0042] In an optional implementation, the step of processing the regional image into a target output image according to the field of view of the regional image and the target output field of view includes:

[0043] Calculating a field of view conversion matrix between the field of view of the regional image and the target output field of view;

[0044] The regional image is processed using the field of view conversion matrix to obtain the target output image.

[0045] In a second aspect, the present invention provides an image processing device, comprising:

[0046] An output start module is used to predict an output timestamp once every preset period and then acquire an environment image with the vehicle position as the image center and record the acquisition timestamp; the environment image is obtained by real-time updating based on the collected images of multiple cameras carried by the vehicle;

[0047] A field of view calculation module, used to calculate a target output field of view at the output timestamp based on the predicted output timestamp, the acquisition timestamp and the real-time posture information of the vehicle; the target output field of view is the field of view of any visual image that needs to be perceived during the vehicle's driving process;

[0048] The image output module is used to intercept a regional image in a set area of ​​the environmental image, and process the regional image into a target output image according to the field of view of the regional image and the target output field of view.

[0049] In a third aspect, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory stores a software program, and when the electronic device is running, the processor executes the software program to implement the image processing method as described in the first aspect above.

[0050] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image processing method described in the first aspect is implemented.

[0051] In a fifth aspect, the present invention provides a program product, which, when executed by a processor, implements the image processing method described in the first aspect.

[0052] Compared with the prior art, the embodiment of the present invention provides an image processing method, device, electronic device and computer-readable storage medium, which obtains an environmental image with the vehicle position as the image center and records the acquisition timestamp after predicting the output timestamp once every preset period; then, based on the predicted output timestamp, the acquisition timestamp and the real-time posture information of the vehicle, the target output field of view at the output timestamp is calculated; the target output field of view is the field of view of any visual image among the panoramic view of the vehicle, the left rearview mirror image, the right rearview mirror image and the transparent chassis image; the regional image is intercepted in the set area of ​​the environmental image, and the regional image is processed into the target output image according to the field of view of the regional image and the target output field of view. In the present invention, the environmental image is obtained by real-time updating based on the collected images of multiple cameras carried by the vehicle, and supports outputting any visual image that needs to be perceived during the driving process of the vehicle based on the environmental image, realizing the decoupling between the image input of the vehicle camera and various visual output images, and there is no need to equip the vehicle with multiple independent visual systems as in the prior art, thereby reducing the consumption of hardware resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 One of the flowchart diagrams of an image processing method provided by an embodiment of the present invention.

[0055] Figure 2 A schematic diagram of the relationship between a set area of ​​a reversing visible map and a vehicle position provided by an embodiment of the present invention.

[0056] Figure 3 The second flowchart of an image processing method provided by an embodiment of the present invention.

[0057] Figure 4 One of the schematic diagrams of adjusting a moving object in a captured image provided by an embodiment of the present invention.

[0058] Figure 5 The second schematic diagram of adjusting a moving object in a captured image provided by an embodiment of the present invention.

[0059] Figure 6 A comparison diagram of changes during an environment image update process provided by an embodiment of the present invention.

[0060] Figure 7 A schematic diagram of converting from a current output field of view to a target output field of view is provided in an embodiment of the present invention.

[0061] Figure 8 A schematic diagram of the structure of an image processing device provided by an embodiment of the present invention.

[0062] Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0064] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0066] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0067] Here, the key words or key terms involved in the present invention are first introduced.

[0068] 1. Yaw acceleration: refers to the rotational acceleration of the vehicle body around the vertical axis (Z axis) when the vehicle is turning. When the vehicle is turning, the body will have a rotational motion around the vertical axis. The acceleration of this rotational motion is the yaw acceleration. It reflects the dynamic response of the vehicle when turning and is an important parameter for measuring the vehicle's handling stability.

[0069] 2. Yaw: The yaw angle refers to the angle at which the vehicle rotates around the vertical axis (Z axis) relative to the initial direction (usually the direction the vehicle is moving forward). It describes the deflection angle of the front of the vehicle during a turn, that is, the change in direction of the vehicle.

[0070] 3. Tilt: The tilt angle refers to the rotation of the vehicle around its longitudinal axis (Y axis), that is, the angle at which the vehicle tilts left and right. When the vehicle is turning, due to the centrifugal force, the body will tilt to the outside of the curve, which is the tilt. The tilt angle can reflect the lateral stability of the vehicle and the performance of the suspension system.

[0071] In the existing technology, the implementation of vehicle vision has the following limitations:

[0072] First, each vision system is equipped with an independent camera and processing unit. This independent configuration leads to a huge waste of hardware resources. Since each vision system needs to be independently designed, produced and maintained, it not only increases the cost but also increases the complexity of the system.

[0073] Secondly, due to the natural contradiction between image resolution and transmission bandwidth, traditional solutions face real-time challenges while pursuing high-resolution images. High-resolution images require more transmission bandwidth to ensure real-time data transmission, which may increase system latency and affect the driving experience. Reducing resolution to reduce bandwidth requirements will affect image quality and the performance of the visual system.

[0074] In summary, the existing technologies have the problems of resource waste and performance limitation when implementing the vehicle vision function. Therefore, it is necessary to develop an efficient and integrated vehicle vision system to achieve efficient acquisition and processing of surrounding environment images, while improving the overall performance of the system and driving experience.

[0075] Based on the discovery of the above technical problems, the inventors have proposed the following technical solutions to solve or improve the above problems through creative work. It should be noted that the defects in the solutions in the above prior art are the results obtained by the inventors after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed in the embodiments of the present application for the above problems below should all be the contributions made by the inventors to the present application in the process of invention and creation, and should not be understood as technical contents known to those skilled in the art.

[0076] The embodiment of the present invention provides an image processing method, which can be applied to an electronic device, which can be a server, a personal computer, or a car or a main control unit of a car. The method updates the vehicle's environmental image in real time based on the images collected by multiple cameras carried by the vehicle, and the image center of the environmental image is the vehicle position; based on different areas in the real-time updated environmental image, after image processing, visual images such as a bird's-eye view panoramic image, a left rearview mirror image, a right rearview mirror image, and a transparent chassis image can be output, without the need to equip the vehicle with multiple independent visual systems as in the prior art, thereby reducing hardware resource consumption.

[0077] Please refer to Figure 1 , Figure 1 The present invention provides a flowchart of an image processing method according to an embodiment of the present invention. The method includes the following steps S200 to S203.

[0078] S200: predict an output timestamp every preset period.

[0079] In this embodiment, a visual image needs to be output once every preset period. To output a visual image, it is first necessary to predict the output timestamp. The output timestamp can be the current timestamp plus the processing delay for outputting the visual image once (i.e., the time required to execute steps S200 to S203 once). The processing delay can be calibrated in advance.

[0080] The size of the preset period may depend on the preset output frame rate of the visual image. Visual images include categories such as overhead panoramic images, left rearview mirror images, right rearview mirror images, and transparent chassis images during vehicle driving, and the preset output frame rates of different categories of visual images may be different or the same. Among them, the preset output frame rate can be flexibly set based on the visual perception needs of the car. For example, the preset output frame rate may be 25 frames / second, 40 frames / second, 60 frames / second, etc. This example is only an example and is not limited here.

[0081] S201, acquiring an environment image with the vehicle position as the image center and recording an acquisition timestamp.

[0082] In this embodiment, the environment image is obtained by real-time updating based on the images collected by multiple cameras carried by the vehicle, and the environment image is stored in the cache of the main control unit.

[0083] It can be understood that the center of the environment image represents the real-time vehicle position. In essence, the environment image is a global environment image from a bird's-eye view, which includes the real-time surrounding traffic / road environment and the ground environment under the vehicle during the vehicle's driving process. However, the vehicle itself does not exist in the environment image, and the area of ​​the vehicle itself is essentially a real-time ground environment image of the vehicle driving.

[0084] S202: Calculate the target output field of view at the output timestamp based on the predicted output timestamp, the acquisition timestamp, and the real-time posture information of the vehicle.

[0085] In this embodiment, the target output field of view is the field of view of any visual image of the vehicle's overhead panoramic image, left rearview mirror image, right rearview mirror image, transparent chassis image, panoramic transparent chassis image, and reversing visible image;

[0086] S203 , intercepting a regional image in a set area of ​​the environment image, and processing the regional image into a target output image according to the field of view of the regional image and the target output field of view.

[0087] In this embodiment, the target output image is any visual image of the overhead panoramic image, the left rearview mirror image, the right rearview mirror image, the transparent chassis image, the panoramic transparent chassis image, and the reversing visible image during the vehicle driving process. To output different visual images, the set area that needs to be intercepted in the environment image is different, the size and shape of the set area are predefined, and the position of the set area in the environment image depends on the image center and the preset vehicle pixel size information (including but not limited to: pixel vehicle length, pixel vehicle width, etc.).

[0088] In an optional example, taking the reversing visible image as an example, the relationship between the set area of ​​the reversing visible image and the vehicle position can be as follows: Figure 2 As shown, Figure 2 The red diamond block in the middle represents the image center of the environment image. In the vehicle's reverse direction, the preset pixel distance d from the image center (d represents the preset pixel distance between the front seat and the rear of the vehicle) is the set area with a fixed trapezoidal shape. This example is only an example and is not used as a limitation on the size of the set area corresponding to the environment image and the reverse visible image.

[0089] The image processing method provided by the embodiment of the present invention predicts the output timestamp once every preset period, obtains the environment image with the vehicle position as the image center and records the acquisition timestamp; then calculates the target output field of view at the output timestamp based on the predicted output timestamp, the acquisition timestamp and the real-time posture information of the vehicle; intercepts the regional image in the set area of ​​the environment image, and processes the regional image into the target output image according to the field of view of the regional image and the target output field of view. The environment image of the present invention is obtained by real-time updating based on the collected images of multiple cameras carried by the vehicle, supports outputting any visual image that needs to be perceived during the driving process of the vehicle based on the environment image, realizes the decoupling between the image input of the vehicle camera and various visual output images, and does not need to equip the vehicle with multiple independent visual systems like the prior art, thereby reducing the consumption of hardware resources, and calculates the target output field of view based on the predicted output timestamp and the acquisition timestamp, considering the processing time required for the output field of view image, and ensures the real-time performance of the output field of view image.

[0090] Next, we will introduce the updating process of the environment image.

[0091] See also Figure 3 The image processing method may further include the following steps S101 to S104.

[0092] S101, obtaining images collected by each camera mounted on the vehicle at the current moment and calculated optical flow information.

[0093] In this embodiment, multiple environment perception cameras are installed at different positions of the vehicle. For example, environment perception cameras can be installed at the following positions of the vehicle to ensure that the vehicle perceives the surrounding environment:

[0094] 1. Front-facing camera: installed directly in front of the vehicle to capture the road and traffic conditions ahead.

[0095] 2. Vehicle side: (1) Side view camera: installed on the side of the vehicle body, used to capture the road and obstacles on both sides of the vehicle; (2) Blind spot monitoring camera: installed near the rearview mirror of the vehicle body, used to detect the blind spot areas on both sides of the vehicle;

[0096] 3. Rear of the vehicle: (1) Reversing camera: installed at the rear of the vehicle to provide a clear view of the road and obstacles behind the vehicle when reversing; (2) Side view camera: installed on both sides of the rear of the vehicle to capture the road and obstacles behind the vehicle.

[0097] This example is only an example, and an actual vehicle may be equipped with more or fewer cameras, which is not limited here.

[0098] It is understandable that the acquisition frequency f set by each environment perception camera on the vehicle can be consistent, for example, f = 30Hz or 60Hz. Each environment perception camera will calculate the optical flow information based on the previous frame when taking each frame of the acquisition image, and then send the captured acquisition image and the calculated optical flow information to the vehicle main control unit.

[0099] The optical flow information of the collected image includes the optical flow vector of each pixel in the image, and the optical flow vector represents the moving speed of the pixel. The calculation method of the optical flow vector belongs to the prior art and will not be described in detail here.

[0100] S102 . For each collected image, adjust the position and shape of a moving object in the collected image based on a preset time delay and optical flow information of the collected image to obtain a delayed collected image corresponding to each collected image.

[0101] In this embodiment, the preset delay t=the time t1 required for image transmission between the acquisition camera corresponding to the acquired image and the main control unit+the processing time t2 required to update the environmental image once. The preset delay can be obtained by calibrating the sizes of t1 and t2 in advance.

[0102] Because, taking the camera mounted on the front of the vehicle as an example, when the camera captures the image in front of the vehicle as the acquisition image, a moving object in the acquisition image (assuming it is a car driving in front of the current vehicle on the same lane) is located at position A. After t1 of image transmission, the moving object has actually moved to position B after t1 and the car will become slightly smaller and its shape will change in the camera's field of view (the car suddenly accelerates during t1). However, the moving object in the acquisition image obtained by the main control unit is still located at position A. When the main control unit updates the environmental image, the moving object has actually moved to position C and the car will become smaller and its shape will change in the camera's field of view (the car continues to accelerate during t2). Therefore, the car part in the acquisition image is moved from position A to position C and the shape and size are adjusted in parallel. The position and shape of the car in the delayed acquisition image are consistent with the actual position and shape of the car in the camera's field of view after time t, thereby ensuring the real-time nature of the environmental image subsequently updated using the delayed acquisition image.

[0103] The following takes a captured image as an example to introduce the process of adjusting the position and shape of a moving object.

[0104] Optionally, the process of adjusting the position and shape of the moving object in the captured image based on the preset time delay and the optical flow information of the captured image in step S102 includes the following sub-steps S1021 to S1026.

[0105] S1021. Based on the optical flow information of the collected image, search for the original object contour of at least one moving object in the collected image.

[0106] In this embodiment, for each pixel in the captured image, the optical flow vector of the pixel may be compared with the optical flow vectors of its adjacent pixels to obtain the original object contour of at least one moving object having a sudden change in the optical flow vector in the captured image.

[0107] S1022. For each target pixel point within the outline of any original object, calculate the displacement distance based on a preset time delay and an optical flow vector of the target pixel point.

[0108] S1023. Find out, from the collected image, a pending pixel point that is at a displacement distance from the target pixel point in the direction indicated by its optical flow vector.

[0109] S1024, replacing the pixel value of the undetermined pixel with the pixel value of the target pixel and then setting the pixel value of the target pixel to zero.

[0110] For an optional example, see Figure 4 ,by Figure 4 Taking the target pixel point in the upper left corner of the original object outline as an example, the direction of the optical flow vector of the target pixel point is as follows Figure 4 As shown by the red arrow, the preset time delay t is multiplied by the modulus of the optical flow vector of the target pixel to obtain the displacement distance d, so that the pixel point at a distance d from the target pixel point in the direction indicated by its optical flow vector is the undetermined pixel point, and then the pixel value of the undetermined pixel point is replaced by the pixel value of the target pixel point and the pixel value of the target pixel point is set to zero. Figure 4 As shown.

[0111] S1025. Traverse each target pixel point of each original object contour to obtain an intermediate captured image.

[0112] In this embodiment, for each target pixel point of each original object contour, steps S1022 to S1024 are executed to complete the position adjustment and shape and size adjustment of the moving object. It can be understood that a moving object (surrounding vehicles or pedestrians) is continuously moving during the preset delay period, and the continuous movement will cause size and position changes in the camera field of view. Therefore, the process of adjusting the position and shape of each moving object in the captured image of the present invention is essentially to perform overall translation, rotation and scaling processing on the area where the moving object is located.

[0113] However, after the position adjustment and shape and size adjustment, there are still many pixel points with zero pixel values ​​to be corrected in the intermediate acquisition image. This is because the original object contour before the moving object is adjusted in position and shape and size and the new object contour after the position adjustment and shape and size adjustment are partially overlapped, so there are two non-overlapping areas, and the non-overlapping area in the opposite direction of the overall position adjustment direction of each moving object is the non-overlapping area that needs to be corrected.

[0114] In the optional example, for Figure 4 After each pixel point of the original object contour is processed according to steps S1022 to S1024, the Figure 5 .from Figure 5 It can be seen that in terms of visual effect, the outline of the new object is translated, rotated and scaled based on the outline of the original object, and the outline of the new object partially overlaps with the outline of the original object, and there are multiple black pixels with zero pixel values ​​in the non-overlapping area that needs to be corrected.

[0115] S1026. For each pixel to be corrected whose pixel value is zero in the intermediate collected image, find the non-zero pixel closest to the target pixel in the intermediate collected image and use the non-zero pixel to correct the pixel to be corrected, so as to obtain a delayed collected image.

[0116] In this embodiment, for each pixel to be corrected whose pixel value is zero in the intermediate acquired image, the non-zero pixel closest to the target pixel is searched in the opposite direction of the optical flow vector of the pixel to be corrected, and then the pixel to be corrected is corrected using the non-zero pixel, so that the lagged acquired image can be obtained.

[0117] For optional examples, see Figure 5 , in the non-overlapping area that needs to be corrected, for each small black dot with a pixel value of zero, Figure 5 The nearest non-zero pixel to the black dot can be found in , and then the non-zero pixel found can be used to correct the black dot. Figure 5 For a small black dot A, we can find the nearest non-zero pixel B, and use the pixel value of the non-zero pixel B to correct the small black dot A.

[0118] It should be noted that the above combination Figure 4 and Figure 5 The examples given are merely illustrative and are not intended to be limiting.

[0119] Optionally, after the pixel value of the pixel to be corrected is replaced with the pixel value of the non-zero pixel, the image display may be easily distorted. In order to ensure the image display quality, the real-time pixel value perceived by the camera can be used to correct the pixel to be corrected in one step. Correspondingly, the process of correcting the pixel to be corrected using the non-zero pixel in step S1026 may include:

[0120] (1) Replace the pixel value of the pixel to be corrected with the pixel value of the non-zero pixel;

[0121] (2) obtaining real-time pixel values ​​outside the edge of the moving object perceived by the camera corresponding to the captured image;

[0122] (3) Determine the final pixel value of the pixel to be corrected based on the result of weighted summation of the current pixel value of the pixel to be corrected and the real-time pixel value.

[0123] In this embodiment, after weighted summing up the current pixel value of the pixel to be corrected and the real-time pixel value, Kalman filtering is performed on the weighted summation result to obtain the final pixel value of the pixel to be corrected, which can ensure that no image noise is generated after the correction of each pixel to be corrected is completed.

[0124] By processing each captured image through the above steps S1021 to S1026, the position of each moving object in the captured image can be adjusted to be consistent with the actual position of the moving object in the real world when the environmental image is updated. At the same time, it is ensured that no image noise is generated during the position update process, thereby meeting the real-time requirements for the object position during the environmental image update process.

[0125] S103 , fusing each of the delayed collected images with different regions of the environment image at the previous moment to obtain an initial environment image at the current moment.

[0126] In this embodiment, since there is a difference between the imaging perspective of each lagged acquisition image and the imaging perspective of the environmental image, each lagged acquisition image needs to be deformed and then fused with different areas of the environmental image at the previous moment.

[0127] Optionally, the sub-steps of performing fusion processing in step S103 may include S1031 to S1034:

[0128] S1031, for any delayed acquisition image, calculating a local fusion area of ​​the delayed acquisition image in the environment image based on the camera viewing angle of the delayed acquisition image;

[0129] S1032, based on the difference between the camera perspective of the delayed acquisition image and the imaging perspective of the local fusion area in the environment image, deform the delayed acquisition image to obtain a fusion input image;

[0130] S1033, fusing the fused input image with the image of the local fusion area in the environment image at the previous moment;

[0131] S1034, traverse each delayed acquisition image to obtain the initial environment image at the current moment.

[0132] In this embodiment, for each lagged acquisition image, after determining the local fusion area between it and the environmental image, a deformation matrix can be calculated based on the difference between the camera perspective of the lagged acquisition image and the imaging perspective of the local fusion area in the environmental image. The lagged acquisition image can be deformed using the deformation matrix to obtain a fused input image.

[0133] Optionally, in step S1031, the method for determining the local fusion area of ​​the lagged acquisition image in the environmental image may be: obtaining the local imaging area of ​​the camera of the lagged acquisition image in the three-dimensional space, and the global imaging area of ​​the virtual camera of the environmental image in the three-dimensional space, projecting the local imaging area to the plane where the global imaging area is located to obtain the projection area, and then the overlapping part of the global imaging area and the projection area is the local fusion area of ​​the lagged acquisition image in the environmental image.

[0134] Optionally, in step S1033, the fusion method may be: weighted summing the pixel value of each pixel point of the local input image with the pixel value of the corresponding pixel point in the image of the local fusion area in the environment image at the previous moment, and then performing Kalman filtering. This ensures that image noise does not appear in the fused initial environment image.

[0135] S104: Based on the position information of the vehicle at the current moment and the previous moment, a movement operation is performed on the initial environment image to obtain the environment image at the current moment.

[0136] In this embodiment, the image center in the initial environment image reflects the position of the car at the last moment. In order to make the current position of the car correspond to the image center, the entire image needs to be moved. That is, as the car moves, it can be considered that the car is stationary relative to the environment, but the environment is moving in the opposite direction of the car's travel. Assuming that the car moves forward, by updating the environment image multiple times, the ground environment in front of the car head that was previously collected can be updated to the body area in the environment image, thereby realizing real-time presentation of the ground environment under the car.

[0137] Optionally, the initial environment image may be moved based on the position change of the vehicle between the last moment and the current moment. Correspondingly, the sub-steps of the above step S104 may include S1041 to S1044.

[0138] S1041. Obtain the historical position information of the vehicle at the previous moment and the current position information at the current moment.

[0139] In this embodiment, the position information may include the vehicle's position, speed data (vehicle's speed components on three axes), direction data, acceleration data (vehicle's acceleration components on three axes), and the like.

[0140] S1042. Calculate the posture change information between the historical posture information and the current posture information.

[0141] In this embodiment, the posture change information may include position transformation data (ΔX, ΔY, ΔZ), a yaw acceleration change value (ΔYaw), a pitch angle change value (ΔPitch), and a roll angle change value (ΔTilt).

[0142] S1043. Use the posture change information to construct a mobile transformation matrix.

[0143] In this embodiment, a movement transformation matrix M (ΔX, ΔY, ΔZ, ΔYaw, ΔPitch, ΔTilt) can be constructed using the posture change information.

[0144] S1044: Use the mobile transformation matrix to adjust the initial environment image to the environment image at the current moment.

[0145] In this embodiment, the initial environment image is multiplied by the mobile conversion matrix to obtain the environment image at the current moment. That is, the capacity of the set cache space used to store the environment image in the cache of the main control unit is fixed, and the initial environment image is multiplied by the mobile conversion matrix, which is essentially to move the pixel value of each pixel point in the initial environment image in the storage position in the set cache space according to the mobile conversion matrix. It can be understood that the pixel points whose storage positions overflow the set cache space will be automatically thrown out during the movement process, and after the movement is completed, the set cache space may be partially vacated, and the vacated part can be filled with all 0s.

[0146] For an optional example, see Figure 6 , assuming that the size of the cache space used to store the environment image is Figure 6 As shown in the red box in the figure, the environment image at the previous moment (i.e. Figure 6 Based on the environment image 1), the above steps S101 to S103 and their sub-steps are processed to obtain Figure 6Then, the initial environment image is moved according to step S104 to obtain the current environment image (i.e. Figure 6 Environment image 2).

[0147] from Figure 6 It can be seen that if the vehicle is traveling in an upward direction (assuming that the direction is equivalent to the vehicle traveling forward), the storage position of the pixel values ​​of each pixel in the initial environment image is moved downward (equivalent to the vehicle position remaining unchanged, while the vehicle surroundings move to the rear of the vehicle) to obtain environment image 2. In the process of obtaining environment image 2 from the initial environment image, pixel points whose storage positions overflow the set cache space will be automatically thrown out ( Figure 6 The overflow pixels are framed), and the empty part of the buffer space is set to be filled with all 0s.

[0148] It should be noted that, in actual situations, the vehicle's driving direction also includes but is not limited to driving to the left front, driving to the right front, driving to the rear, etc. For different driving directions, when performing movement calculations on the initial environment image in the set cache space, the changes in the storage position of the pixel value of each pixel point are different. Figure 6 The reference diagram is only provided to facilitate understanding of the scheme and is not intended to be limiting.

[0149] The above description of steps S101 to S104 and their sub-steps is a complete description of the updating process of the environment image.

[0150] Next, the detailed process of outputting visual images is introduced.

[0151] In an optional implementation, for the above step S202, a current output field of view may be determined using the current speed position of the vehicle, and then the current output field of view may be converted into a target output field of view at the output timestamp using the time difference between the output timestamp and the acquisition timestamp and the current vehicle speed. Correspondingly, the sub-steps of step S202 may include S2021 to S2025.

[0152] S2021. Calculate the time difference between the output timestamp and the acquisition timestamp;

[0153] S2022, obtaining the current vehicle speed and the current vehicle position from the real-time position information of the vehicle;

[0154] S2023, determining a current output field of view based on a current vehicle position;

[0155] S2024, calculating the field of view moving distance based on the current vehicle speed and the time difference;

[0156] S2025. Shift the current output field of view in the direction of the current vehicle speed by the field of view movement distance to obtain the target output field of view at the output timestamp.

[0157] In the optional examples, take the target output image as a bird's-eye view panoramic image as an example, see Figure 7 , assuming that the current vehicle is traveling to the left and the current vehicle position is M, then based on point M, the virtual camera position of the current output field of view of the overhead panoramic view can be determined as point E (in the opposite direction of the vehicle's traveling direction, the virtual camera projection point O is a set distance away from point M. Since the imaging area of ​​the current output field of view of the overhead panoramic view is parallel to the ground, a perpendicular line through point O is drawn to the ground, and point E on the perpendicular line at a set height away from point O is the virtual camera position of the current output field of view), and the imaging area of ​​the current output field of view of the overhead panoramic view is the light blue area centered on the car's position, so the current output field of view shown by the blue line can be determined.

[0158] Assume that the time difference between the output timestamp and the acquisition timestamp is Δt, and the current vehicle speed is v. According to the current vehicle speed v, after the time difference Δt, the predicted vehicle position at the predicted output timestamp is N. The vehicle moves, and accordingly, the field of view also needs to move. Then the field of view moves by a distance d = v*Δt, so Figure 7 The field of view movement direction of the current output field of view is the same as the vehicle driving direction. The current output field of view moves to the left by d to obtain the target output field of view at the output timestamp.

[0159] In actual situations, the relationship between the output field of view of different actual images and the position of the car is different. The above examples are only illustrative and are not limiting here.

[0160] In an optional implementation, the process of processing the regional image into the target output image according to the field of view of the regional image and the target output field of view in the above step S203 may include the following sub-steps:

[0161] S2031, calculating the field of view conversion matrix between the field of view of the regional image and the target output field of view;

[0162] S2032. Process the regional image using the field of view conversion matrix to obtain a target output image.

[0163] In this embodiment, the target output image is obtained by multiplying the field of view conversion matrix with the intercepted area image, and the target output image is output to the vehicle display unit at the predicted timestamp.

[0164] In the process of outputting a visual image each time, the present invention first predicts the output timestamp (current timestamp + processing delay) of the visual image based on the pre-calibrated processing delay of outputting a visual image, then obtains the environmental image from the cache and records the acquisition timestamp, and then uses the time difference between the output timestamp and the acquisition timestamp and the current vehicle to translate the current output field of view of the actual image corresponding to the current vehicle position into the target output field of view of the output timestamp, and finally converts the regional image intercepted in the set area of ​​the environmental image into the target output field of view to obtain the visual image to be output, taking into account the processing delay of obtaining the visual image from the environmental image, and ensuring the real-time performance of the output field of view image.

[0165] It should be noted that the execution order of each step in the above method embodiment is not limited to that shown in the drawings, and the execution order of each step shall be based on the actual application situation.

[0166] In order to execute the corresponding steps in the above method embodiment and various possible implementations, an implementation of an image processing device is given below.

[0167] See also Figure 8 , Figure 8 The schematic diagram of the structure of the image processing device provided by the embodiment of the present invention is shown. The image processing device 200 comprises: an output start module 260 , a field of view calculation module 270 , and an image output module 280 .

[0168] The output start module 260 is used to predict the output timestamp once every preset period and then obtain an environmental image with the vehicle position as the image center and record the acquisition timestamp; the environmental image is obtained by real-time updating based on the collected images of multiple cameras carried by the vehicle;

[0169] The field of view calculation module 270 is used to calculate the target output field of view at the output timestamp based on the predicted output timestamp, the acquisition timestamp and the real-time posture information of the vehicle; the target output field of view is the field of view of any visual image among the top view panoramic image of the vehicle, the left rearview mirror image, the right rearview mirror image and the transparent chassis image;

[0170] The image output module 280 is used to capture a regional image in a set area of ​​the environment image, and process the regional image into a target output image according to the field of view of the regional image and the target output field of view.

[0171] Optionally, the image processing device 200 may also be used to update the environment image, so the image processing device 200 may further include:

[0172] The image acquisition module 210 is used to acquire the images acquired by each camera mounted on the vehicle at the current moment and the calculated optical flow information;

[0173] The moving object adjustment module 220 is used to adjust the position and shape of the moving object in each collected image based on the preset time delay and the optical flow information of the collected image to obtain a delayed collected image corresponding to each collected image;

[0174] A fusion module 230 is used to fuse each of the delayed collected images with different regions of the environment image at the previous moment to obtain an initial environment image at the current moment;

[0175] The movement calculation module 240 is used to perform movement calculation on the initial environment image based on the position information of the vehicle at the current moment and the previous moment to obtain the environment image at the current moment.

[0176] Optionally, the moving object adjustment module 220 can be specifically used to: search for the original object contour of at least one moving object in the collected image based on the optical flow information of the collected image; the optical flow information of the collected image includes the optical flow vector of each pixel in the image, and the optical flow vector represents the moving speed of the pixel; for each target pixel in any original object contour, calculate the displacement distance based on the preset time delay and the optical flow vector of the target pixel; from the collected image, find the pending pixel that is separated from the target pixel by the displacement distance in the direction indicated by its optical flow vector; replace the pixel value of the pending pixel with the pixel value of the target pixel and then set the pixel value of the target pixel to zero; traverse each target pixel of each original object contour to obtain an intermediate collected image; for each to-be-corrected pixel whose pixel value in the intermediate collected image is zero, find the non-zero pixel closest to the target pixel in the intermediate collected image and use the non-zero pixel to correct the to-be-corrected pixel to obtain a delayed collected image.

[0177] Optionally, the moving object adjustment module 220 can be specifically used to: replace the pixel value of the pixel to be corrected with the pixel value of a non-zero pixel; obtain the real-time pixel value outside the edge of the moving object perceived by the camera corresponding to the captured image; and determine the final pixel value of the pixel to be corrected based on the result of weighted summation of the current pixel value of the pixel to be corrected and the real-time pixel value.

[0178] Optionally, the fusion module 230 can be specifically used to: for any lagged acquisition image, calculate the local fusion area of ​​the lagged acquisition image in the environmental image based on the camera perspective of the lagged acquisition image; based on the difference between the camera perspective of the lagged acquisition image and the imaging perspective of the local fusion area in the environmental image, deform the lagged acquisition image to obtain a fused input image; fuse the fused input image with the image of the local fusion area in the environmental image at the previous moment; traverse each lagged acquisition image to obtain the initial environmental image at the current moment.

[0179] Optionally, the mobile calculation module 240 can be specifically used to: obtain the historical posture information of the vehicle at the previous moment and the current posture information at the current moment; calculate the posture change information between the historical posture information and the current posture information; use the posture change information to construct a mobile transformation matrix; use the mobile transformation matrix to adjust the initial environment image to the environment image at the current moment.

[0180] Optionally, the field of view calculation module 270 can be specifically used to: calculate the time difference between the output timestamp and the acquisition timestamp; obtain the current vehicle speed and the current vehicle position from the real-time posture information of the vehicle; determine the current output field of view based on the current vehicle position; calculate the field of view movement distance based on the current vehicle speed and the time difference; translate the current output field of view in the direction of the current vehicle speed by the field of view movement distance to obtain the target output field of view at the output timestamp.

[0181] Optionally, the image output module 280 may be specifically used to: calculate a field of view conversion matrix between the field of view of the regional image and the target output field of view; and process the regional image using the field of view conversion matrix to obtain a target output image.

[0182] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the image processing device 200 described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0183] See also Fig. 9 , Fig. 9 The electronic device 300 includes a processor 310 , a memory 320 , and a bus 330 , wherein the processor 310 is connected to the memory 320 via the bus 330 .

[0184] The memory 320 may be used to store software programs, for example, software programs corresponding to the image processing apparatus 200 provided in the embodiment of the present invention. The processor 310 executes various functional applications and data processing to implement the image processing method provided in the embodiment of the present invention by running the software programs stored in the memory 320.

[0185] Among them, the memory 320 can be but is not limited to: RAM (Random Access Memory), ROM (Read Only Memory), FLASH (Flash Memory), PROM (Programmable Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electric Erasable Programmable Read-Only Memory), etc.

[0186] The processor 310 may be an integrated circuit chip with signal processing capability. The processor 310 may be a general-purpose processor, including: CPU (Central Processing Unit), NP (Network Processor), etc.; it may also be: DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0187] Understandably, Fig. 9 The structure shown is for illustration only. The electronic device 300 may also include Fig. 9 More or fewer components as shown, or with Fig. 9 Different configurations are shown. Fig. 9 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0188] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the image processing method disclosed in the above embodiment is implemented. The computer-readable storage medium can be, but is not limited to, various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a PROM, an EPROM, an EEPROM, a FLASH disk, or an optical disk.

[0189] The embodiment of the present invention further provides a program product, which implements the image processing method disclosed in the above embodiment when executed by a processor.

[0190] In summary, the embodiments of the present invention provide an image processing method, device, electronic device and computer-readable storage medium, which obtains an environmental image with the vehicle position as the image center and records the acquisition timestamp after predicting the output timestamp once every preset period; then, based on the predicted output timestamp, the acquisition timestamp and the real-time posture information of the vehicle, the target output field of view at the output timestamp is calculated; the target output field of view is the field of view of any visual image among the panoramic view of the vehicle, the left rearview mirror image, the right rearview mirror image and the transparent chassis image; the regional image is intercepted in the set area of ​​the environmental image, and the regional image is processed into the target output image according to the field of view of the regional image and the target output field of view. In the present invention, the environmental image is obtained by real-time updating based on the collected images of multiple cameras carried by the vehicle, and supports outputting any visual image that needs to be perceived during the driving process of the vehicle based on the environmental image, realizing the decoupling between the image input of the vehicle camera and various visual output images, and there is no need to equip the vehicle with multiple independent visual systems as in the prior art, thereby reducing the consumption of hardware resources.

[0191] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized in that: include: After predicting the output timestamp once every preset period, an environment image with the vehicle position as the image center is acquired and the acquisition timestamp is recorded; The environmental image is obtained by real-time updating based on the collected images of multiple cameras carried by the vehicle; Based on the predicted output timestamp, the acquisition timestamp and the real-time posture information of the vehicle, calculating the target output field of view at the output timestamp; The target output field of view is the field of view of any visual image that needs to be perceived during the vehicle's driving process; A regional image is intercepted in a set area of ​​the environment image, and the regional image is processed into a target output image according to the field of view of the regional image and the target output field of view.

2. The method according to claim 1, characterized in that The updating process of the environment image includes: Obtain the collected images and calculated optical flow information collected by each camera on the vehicle at the current moment; For each of the collected images, adjusting the position and shape of the moving object in the collected image based on a preset time delay and optical flow information of the collected image, so as to obtain a delayed collected image corresponding to each collected image; fusing each of the delayed acquisition images with different regions of the environment image at the previous moment to obtain the initial environment image at the current moment; Based on the position information of the vehicle at the current moment and the previous moment, a movement operation is performed on the initial environment image to obtain the environment image at the current moment.

3. The method according to claim 2, characterized in that The step of adjusting the position and shape of the moving object in the collected image based on the preset time delay and the optical flow information of the collected image includes: Based on the optical flow information of the collected image, searching for the original object contour of at least one moving object in the collected image; the optical flow information of the collected image includes an optical flow vector of each pixel in the image, and the optical flow vector represents the moving speed of the pixel; For each target pixel point within any of the original object contours, calculating a displacement distance based on the preset time delay and an optical flow vector of the target pixel point; From the collected image, find a pixel point to be determined that is spaced from the target pixel point by the displacement distance in the direction indicated by the optical flow vector thereof; After replacing the pixel value of the undetermined pixel point with the pixel value of the target pixel point, the pixel value of the target pixel point is set to zero; Traversing each target pixel point of each original object contour to obtain an intermediate acquisition image; For each pixel to be corrected whose pixel value is zero in the intermediate acquisition image, a non-zero pixel closest to the target pixel is searched in the intermediate acquisition image and the non-zero pixel is used to correct the pixel to be corrected, so as to obtain the delayed acquisition image.

4. The method according to claim 3, characterized in that The step of correcting the pixel to be corrected by using the non-zero pixel comprises: Replacing the pixel value of the pixel to be corrected with the pixel value of the non-zero pixel; Acquire real-time pixel values ​​outside the edge of the moving object perceived by the camera corresponding to the captured image; Based on a result of weighted summation of the current pixel value of the pixel to be corrected and the real-time pixel value, a final pixel value of the pixel to be corrected is determined.

5. The method according to claim 2, characterized in that: The step of fusing each of the delayed acquisition images with different regions of the environment image at the previous moment to obtain the initial environment image at the current moment comprises: For any of the delayed acquisition images, calculating a local fusion area of ​​the delayed acquisition image within the environment image based on the camera viewing angle of the delayed acquisition image; Based on the difference between the camera viewing angle of the delayed acquisition image and the imaging viewing angle of the local fusion area in the environmental image, the delayed acquisition image is deformed to obtain a fusion input image; Fusing the fused input image with the image of the local fusion area in the environment image at the previous moment; Each of the delayed acquisition images is traversed to obtain the initial environment image at the current moment.

6. The method according to claim 2, characterized in that The step of performing a movement operation on the initial environment image based on the respective position information of the vehicle at the current moment and the previous moment to obtain the environment image at the current moment includes: Obtain the vehicle's historical position information at the previous moment and the current position information at the current moment; Calculating the posture change information between the historical posture information and the current posture information; Using the posture change information, constructing a movement transformation matrix; The initial environment image is adjusted to the environment image at the current moment by using the movement transformation matrix.

7. The method according to claim 1, characterized in that The step of calculating the target output field of view at the output timestamp based on the predicted output timestamp, the acquisition timestamp and the real-time posture information of the vehicle comprises: Calculating the time difference between the output timestamp and the acquisition timestamp; Obtain the current vehicle speed and current vehicle position from the vehicle's real-time posture information; Determining a current output field of view based on a current vehicle position; Calculating the field of view movement distance based on the current vehicle speed and the time difference; The current output field of view is translated in the direction of the current vehicle speed by the field of view movement distance to obtain the target output field of view at the output timestamp.

8. An image processing device, characterized in that: include: An output start module, used to predict an output timestamp once every preset period, obtain an environment image with the vehicle position as the image center, and record the acquisition timestamp; The environmental image is obtained by real-time updating based on the collected images of multiple cameras carried by the vehicle; A field of view calculation module, used to calculate the target output field of view at the output timestamp based on the predicted output timestamp, the acquisition timestamp and the real-time posture information of the vehicle; The target output field of view is the field of view of any visual image that needs to be perceived during the vehicle's driving process; The image output module is used to intercept a regional image in a set area of ​​the environmental image, and process the regional image into a target output image according to the field of view of the regional image and the target output field of view.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a software program, and when the electronic device is running, the processor executes the software program to implement the image processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image processing method according to any one of claims 1 to 7 is implemented.

Images