Image display method and device, vehicle image auxiliary display system and vehicle

Abstract

The invention discloses an image display method and device, a vehicle image auxiliary display system and a vehicle, and relates to the technical field of intelligent driving. The image display method comprises the steps of obtaining a to-be-displayed image; adding a pre-calibrated grid to the to-be-displayed image to obtain a grid image, and displaying the grid image; the pre-calibrated grid comprises a plurality of unit grids, the unit grids correspond to actual unit grid sizes, the actual unit grid sizes are the sizes of the unit grids in a vehicle coordinate system, and the vehicle coordinate system is a coordinate system established by taking a specific position or part of a vehicle as an original point. Therefore, the user can judge the distance between the object in the image and the vehicle only from the image, and the user can operate the vehicle conveniently.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and specifically to an image display method, an image display device, a vehicle image auxiliary display system, a vehicle, a machine-readable storage medium, and an electronic device. Background Art

[0002] Vehicle image auxiliary display plays an extremely critical role in the field of modern automotive technology. During driving, the vehicle image auxiliary display system uses cameras and other image acquisition devices to capture the scene around the vehicle in real time. Some vehicles are also equipped with a 360° panoramic imaging system. Multiple cameras are distributed in the front, back, left and right of the vehicle to collect image data from different angles. After processing, a complete bird's-eye view of the vehicle's surroundings is synthesized and displayed on the screen. However, images can only be used to observe the environment, and it is impossible to determine the distance of objects in the image from the vehicle alone.

[0003] For example, for medium-sized and super-large excavators, the cab is located in the front of the upper body, with a large blind spot. It is usually necessary to install front and rear high-definition cameras or wide-angle cameras around to achieve a 360-degree panoramic view. However, whether it is a high-definition camera or a wide-angle camera, the operator cannot determine the distance of the object from the excavator from the image alone. Especially for remote-controlled excavators, the operator can only observe the environment from the image. This will undoubtedly reduce the operator's operating confidence, reduce operating efficiency, and even cause collision accidents.

[0004] Therefore, the existing image-assisted display on the vehicle cannot determine the distance between the object in the image and the vehicle from the image alone. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an image display method, an image display device, a vehicle image auxiliary display system, a vehicle, a machine-readable storage medium and an electronic device to solve the problem of inaccurate lane line detection in the prior art.

[0006] In order to achieve the above-mentioned object, the present application provides, in a first aspect, an image display method, comprising: Get the image to be displayed; Adding a pre-calibrated grid to the image to be displayed to obtain a grid image, and displaying the grid image; The pre-calibrated grid includes a plurality of unit grids, each of which corresponds to an actual unit grid size, which is the size of the unit grid in a vehicle coordinate system. The vehicle coordinate system is a coordinate system established with a specific position or component of the vehicle as the origin.

[0007] In the embodiment of the present application, the step of obtaining the image to be displayed includes: Get the original image; Distortion correction and / or view conversion are performed on the original image to obtain an image to be displayed.

[0008] In the embodiment of the present application, the image to be displayed is a corrected image; The step of adding the pre-calibrated grid to the image to be displayed to obtain a grid image includes: Adding a pre-calibrated grid to the image to be displayed to obtain a distortion-free grid image; The undistorted grid image is converted into a distorted image to obtain a grid image.

[0009] In the embodiment of the present application, the step of adding a pre-calibrated grid to the image to be displayed to obtain a grid image includes: Determining whether the image to be displayed and the image of the pre-calibrated grid have different viewing angles; When it is determined that the image to be displayed and the pre-calibrated grid have different image perspectives, performing view conversion on the image to be displayed or the pre-calibrated grid to obtain the image to be displayed and the grid with the same image perspective; The image to be displayed and the grid having the same image viewing angle are superimposed to obtain a grid image.

[0010] In the embodiment of the present application, there are multiple images to be displayed, and the acquisition positions of the images to be displayed are different; The step of adding the pre-calibrated grid to the image to be displayed to obtain a grid image includes: Adding corresponding pre-calibrated grids to the images to be displayed respectively to obtain a plurality of initial grid images, wherein the image perspective of each initial grid image is a bird's-eye view; The multiple initial grid images are spliced ​​together to obtain a grid image.

[0011] In an embodiment of the present application, the method further includes: The grid image is subjected to view conversion and / or distortion processing to obtain a converted grid image.

[0012] In the embodiment of the present application, the image perspective of the grid image is a top-down perspective, and the method further includes: Obtain driving speed and vehicle body rotation angle in real time; Based on the driving speed and the vehicle body rotation angle, a predicted vehicle path is drawn in the grid image to obtain a path grid top view.

[0013] In the embodiment of the present application, the grid calibration process includes: Acquire an actual calibration size, where the actual calibration size is a size of a first quadrilateral in the vehicle coordinate system, where the first quadrilateral is a rectangle formed by four points selected on the ground in front of the vehicle; Acquire a calibration image, and determine pixel points corresponding to the four points in the calibration image to obtain a second quadrilateral, wherein the calibration image and the image to be displayed are acquired at the same location, and at least one set of opposite sides in the second quadrilateral is parallel to one side of the calibration image; According to a preset segmentation rule, a segmentation line is constructed in the second quadrilateral to obtain a first initial grid; Based on the actual calibrated size, the actual size of the unit grid in the first initial grid is determined to obtain a grid.

[0014] In the embodiment of the present application, constructing a segmentation line in the second quadrilateral according to a preset segmentation rule to obtain a first initial grid includes: According to a preset segmentation rule, uniform segmentation is performed on each side of the second quadrilateral to obtain a plurality of first segmentation points; The corresponding first segmentation points on opposite sides of the second quadrilateral are respectively connected to form segmentation lines to obtain a first initial grid.

[0015] In the embodiment of the present application, the grid calibration process includes: Acquire an actual calibration size, where the actual calibration size is a size of a first quadrilateral in the vehicle coordinate system, where the first quadrilateral is a rectangle formed by four points selected on the ground in front of the vehicle; Acquire a calibration image, and determine pixel points corresponding to the four points in the calibration image to obtain a second quadrilateral, wherein the calibration image and the image to be displayed are acquired at the same location, and at least one set of opposite sides in the second quadrilateral is parallel to one side of the calibration image; Transform the second quadrilateral into a top view to obtain a third quadrilateral; According to a preset segmentation rule, a segmentation line is constructed in the third quadrilateral to obtain a second initial grid; Based on the actual calibrated size, the actual size of the unit grid in the second initial grid is determined to obtain a grid.

[0016] In the embodiment of the present application, constructing a segmentation line in the third quadrilateral according to a preset segmentation rule to obtain a second initial grid includes: According to a preset segmentation rule, uniform segmentation is performed on each side of the third quadrilateral to obtain a plurality of second segmentation points; The corresponding second segmentation points on opposite sides of the third quadrilateral are respectively connected to form segmentation lines to obtain a second initial grid.

[0017] In the embodiment of the present application, after uniform segmentation is performed on each side of the third quadrilateral according to the preset segmentation rule to obtain a plurality of second segmentation points, the method further includes: Perspectively transforming the plurality of second segmentation points into the first quadrilateral to obtain a plurality of third segmentation points; respectively connecting corresponding third segmentation points on opposite sides of the first quadrilateral to form segmentation lines, thereby obtaining a third initial grid; Based on the actual calibrated size, the actual size of the unit grid in the third initial grid is determined to obtain a grid.

[0018] A second aspect of the present application provides an image display device, comprising: An acquisition module, used for acquiring an image to be displayed; A display module is used to add a pre-calibrated grid to the image to be displayed, obtain a grid image, and display the grid image; the pre-calibrated grid includes a plurality of unit grids, each unit grid corresponds to an actual unit grid size, and the actual unit grid size is the size of the unit grid in a vehicle coordinate system, and the vehicle coordinate system is a coordinate system established with a specific position or component of the vehicle as the origin.

[0019] A third aspect of the present application provides a vehicle image auxiliary display system, comprising an image processing device, a display device and at least one image acquisition device arranged on a vehicle; the image acquisition device is used to obtain an image to be displayed, and send the image to be displayed to the image processing device; the image processing device is used to add a pre-calibrated grid to the image to be displayed to obtain a grid image, and send the grid image to the display device for display, the pre-calibrated grid includes a plurality of unit grids, the unit grid corresponds to an actual size of the unit grid, and the actual size of the unit grid is the size of the unit grid in a vehicle coordinate system, and the vehicle coordinate system is a coordinate system established with a specific position or component of the vehicle as the origin.

[0020] A fourth aspect of the present application provides a vehicle, including a vehicle image auxiliary display system, wherein the vehicle image auxiliary display system uses the above-mentioned image display method to display images.

[0021] A fifth aspect of the present application provides an electronic device, the electronic device comprising: at least one processor; a memory connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the above-mentioned image display method by executing the instructions stored in the memory.

[0022] A sixth aspect of the present application provides a machine-readable storage medium having instructions stored thereon, which, when executed by a processor, configure the processor to execute the above-mentioned image display method.

[0023] Through the above technical solution, by acquiring an image to be displayed; adding a pre-calibrated grid to the image to be displayed, obtaining a grid image, and displaying the grid image; the pre-calibrated grid includes a plurality of unit grids, and the unit grid corresponds to an actual size of the unit grid, and the actual size of the unit grid is the size of the unit grid in the vehicle coordinate system, and the vehicle coordinate system is a coordinate system established with a specific position or component of the vehicle as the origin. By adding a pre-calibrated grid to the image to be displayed, the displayed image has a grid, and the actual size of the unit grid in the grid is the size of each unit grid in the vehicle coordinate system. Therefore, according to the number of unit grids between the object and the vehicle in the image, the distance between the object and the vehicle in the image can be quickly determined, so that the user can determine the distance between the object and the vehicle in the image from the image alone, which is helpful for the user to operate the vehicle.

[0024] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings: Figure 1 A schematic diagram of a process of an image display method according to an embodiment of the present application is schematically shown; Figure 2 A schematic diagram of an excavator coordinate system according to an embodiment of the present application is schematically shown; Figure 3 A schematic diagram showing the comparison between a normal image and a barrel distorted image according to an embodiment of the present application is shown; Figure 4 The original viewing angle image according to the embodiment of the present application is schematically shown; Figure 5 Schematically shows a top view with grid lines according to an embodiment of the present application; Figure 6 Schematically shows an original viewing angle image with grid lines according to an embodiment of the present application; Figure 7 Schematically shows a 360° surround view image with grid lines according to an embodiment of the present application; Figure 8 A schematic diagram of a straight line path predicted by a chassis in a top view of a front-view camera according to an embodiment of the present application is shown schematically; Fig. 9 A schematic diagram of a predicted arc path of a chassis displayed in a top view of a front-view camera according to an embodiment of the present application is schematically shown; Fig.10 A schematic diagram of a straight path predicted by a chassis in a 360° surround view according to an embodiment of the present application is schematically shown; Fig.11 A schematic diagram of displaying a predicted arc path of a chassis in a 360° surround view according to an embodiment of the present application is shown schematically; Fig.12 Schematically shows a top view with fan-shaped grid lines according to an embodiment of the present application; Fig.13 The following schematically shows a schematic diagram of the steps of displaying an excavator image according to an embodiment of the present application; Fig.14 The electrical connection diagram of the excavator according to the embodiment of the present application is schematically shown; Fig.15 A schematic diagram of the structure of an image display device according to an embodiment of the present application is shown; Fig.16 The internal structure diagram of a computer device according to an embodiment of the present application is schematically shown.

[0026] Description of Reference Numerals 410 - acquisition module; 420 - display module; A01 - processor; A02 - network interface; A03 - internal memory; A04 - display screen; A05 - input device; A06 - non-volatile storage medium; B01 - operating system; B02 - computer program. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0028] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0029] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0030] Please see Figure 1 , Figure 1 The following is a schematic diagram of a flow chart of an image display method according to an embodiment of the present application. This embodiment provides an image display method, comprising the following steps: Step 210: Obtain an image to be displayed; In this embodiment, the image to be displayed may be acquired by an image acquisition device installed on the vehicle, and the image acquisition device may be a high-definition camera, a wide-angle camera, etc. The image to be displayed may be the original view image acquired by the image acquisition device, or may be the image after view conversion and / or preprocessing of the original view image, and the preprocessing includes denoising, deformity correction, etc. For example, when a standard camera is used, distortion generally rarely occurs, while the image captured by a wide-angle camera has obvious barrel distortion. After the image is captured by the wide-angle camera, the image to be displayed may be obtained by barrel distortion correction.

[0031] In some embodiments, the step of obtaining the image to be displayed comprises the following steps: First, obtain the original image; In this embodiment, the original image may be an image acquired by an image acquisition device, which may be a camera.

[0032] Then, distortion correction and / or view conversion is performed on the original image to obtain an image to be displayed.

[0033] In this embodiment, it is considered that some images taken by cameras may be distorted, such as fisheye cameras, wide-angle cameras, etc. This embodiment is mainly described by taking a wide-angle camera as an example. The images taken by a wide-angle camera obviously have barrel distortion. Please refer to Figure 3 , Figure 3 A schematic diagram of comparing a normal image and a barrel-distorted image according to an embodiment of the present application is schematically shown. Therefore, it is necessary to correct the barrel distortion of the original image captured by the wide-angle camera, and specifically, the barrel distortion can be corrected according to the internal parameters and radial distortion parameters of the wide-angle camera. It should be noted that if the internal parameters and radial distortion parameters of the wide-angle camera are unknown, they can be obtained by camera calibration. Common calibration methods include Tsai two-step method, Zhang calibration method, etc. The specific calibration process belongs to the prior art and will not be repeated here. The above-mentioned view conversion refers to converting the image perspective of the original image imaging into a top-view perspective, and specifically, the perspective of the original image imaging can be converted into a top-view perspective through a homography matrix. The image information displayed by the top-view perspective is more intuitive than the image perspective of the imaging image, so as to facilitate viewing the information in the image. The above-mentioned distortion correction and view conversion can be used separately or together, which is determined according to the actual situation. When used together, the original image can be firstly subjected to distortion correction to obtain a corrected image, and then the corrected image is subjected to view conversion to obtain an image to be displayed.

[0034] The following is a detailed description of the steps for wide-angle camera barrel distortion correction, which specifically includes the following steps: Step 1: Set the pixel coordinates on the pixel plane Convert to image plane ,Right now: , in, is the camera internal parameter.

[0035] Step 2: Perform radial distortion correction on the points on the image plane, specifically: , in, , is the radial distortion parameter, is the point in the image plane after radial distortion correction.

[0036] Step 3: Convert the points on the corrected image plane to the pixel plane through the intrinsic parameters to obtain the pixel coordinates , which is obtained by the following formula: .

[0037] It should be noted that if the undistorted image is converted into a barrel-distorted image, only step 2 is required to be the opposite of the barrel distortion correction, that is, it can be converted by the following formula: .

[0038] The following is a detailed description of view conversion: The coordinates of a pixel point in the image captured by the camera are , which can be obtained by the homography matrix Convert the imaging perspective into a bird's-eye view. The pixel coordinates are: ,Right now: , Because the above formula is scale invariant, for the matrix Multiply by any scaling factor The above formula still holds, so ,matrix There are only 8 degrees of freedom, namely: , Arranging the above formula gives: , Because one set of matching points can get two equations, to solve 8 unknowns, four sets of matching points are needed. Therefore, we can get: , By finding four sets of pixel coordinates and the corresponding pixel coordinates of the top-down perspective, substituting them into the above equations for solution, we can get Matrix. Then the coordinates of all pixels in the original image are The matrix projection is transformed into pixel coordinates of a top-down perspective, thereby obtaining an image to be displayed.

[0039] By performing distortion correction on the original image, the influence of distortion on the image accuracy can be reduced. Performing view conversion on the original image can make the converted image more intuitive and facilitate image display.

[0040] Step 220: Add a pre-calibrated grid to the image to be displayed to obtain a grid image, and display the grid image; the pre-calibrated grid includes a plurality of unit grids, each of which corresponds to an actual unit grid size, and the actual unit grid size is the size of the unit grid in a vehicle coordinate system, and the vehicle coordinate system is a coordinate system established with a specific position or component of the vehicle as the origin.

[0041] In this embodiment, when the pre-calibrated grid is added to the image to be displayed, if the range of the grid is large enough, the pre-calibrated grid can be directly added to the image to be displayed. If the range of the grid is not large enough, the pre-calibrated grid can be superimposed on the image to be displayed, and the grid can be further extended to expand the range of the grid in the image to be displayed. The extension can be to first extend the edge of the grid, and then construct a dividing line according to the unit grid size on the extended part to achieve grid extension. The range size of the grid can be determined by first setting a range threshold, and then comparing the range of the grid with the threshold. The pre-calibrated grid has the actual size of the unit grid, and the position of the object relative to the vehicle in the vehicle coordinate system can be intuitively displayed through the grid. The calibration of the grid can be performed in the image view of the image, or in the top view. The image to be displayed can be a variety of image perspectives, for example, it can be a top view, or it can be the original perspective, that is, the image perspective of the image. When the image perspective of the image to be displayed is a top view, the above-mentioned adding of the pre-calibrated grid to the image to be displayed can realize the display of the grid in the top view; when the image perspective of the image to be displayed is the original perspective, the above-mentioned adding of the pre-calibrated grid to the image to be displayed can realize the display of the grid in the original perspective image.

[0042] The above-mentioned pre-calibrated grid can be a bird's-eye view or an original image view. For the grid of the original image view, it can be calibrated in the calibration image by direct calibration, which is direct and simple; or it can be obtained by first obtaining the segmentation points in the calibration image of the bird's-eye view, and then converting the segmentation points to the calibration image for connection to obtain the grid of the original image view. This calibration method can more intuitively calibrate the segmentation points in the calibration image of the bird's-eye view, which helps to obtain a more accurate grid.

[0043] In some embodiments, when direct calibration is used for calibration, the calibration process of the grid includes the following steps: First, an actual calibration size is obtained, where the actual calibration size is the size of a first quadrilateral in the vehicle coordinate system, where the first quadrilateral is a rectangle formed by four points selected on the ground in front of the vehicle; In this embodiment, four vertices may be selected on the ground in front of the camera, and the connecting line of two of the vertices is ensured to be close to and parallel to the edge of the vehicle body, so that they form four vertices of a rectangle, and the size of each side of the first quadrilateral is determined according to the positions of the four vertices, that is, the actual calibration size is obtained. It should be noted that the larger the above rectangle is, the more accurate the constructed mesh is.

[0044] Then, a calibration image is acquired, and pixel points corresponding to the four points are determined in the calibration image to obtain a second quadrilateral, wherein the calibration image and the image to be displayed are acquired at the same position, and at least one set of opposite sides in the second quadrilateral is parallel to one side of the calibration image; In this embodiment, the image viewing angle of the calibration image is the original viewing angle, that is, the image viewing angle of the imaging. The calibration image may be an image after the image acquisition device acquires the image and performs distortion correction. In the absence of distortion correction, the calibration image is the image acquired by the image acquisition device. The acquisition position of the calibration image and the image to be displayed being the same means that the position and angle of the acquisition device are the same when the two images are acquired. The calibration image may also be the image to be displayed. The second quadrilateral may be a trapezoid, the upper and lower sides of the trapezoid are parallel, and are parallel to one side of the calibration image. It should be noted that either of the upper and lower sides of the second quadrilateral may be on one side of the calibration image.

[0045] Then, according to a preset segmentation rule, a segmentation line is constructed in the second quadrilateral to obtain a first initial grid; In this embodiment, the preset segmentation rule refers to segmentation into a plurality of unit grids. According to the number of unit grids to be segmented, a segmentation line is constructed to obtain a first initial grid.

[0046] In some embodiments, constructing a segmentation line in the second quadrilateral according to a preset segmentation rule to obtain a first initial grid includes: The first step is to perform uniform segmentation on each side of the second quadrilateral according to a preset segmentation rule to obtain a plurality of first segmentation points; In this embodiment, the number of segmentation points on each edge can be determined according to a preset segmentation rule, thereby obtaining a plurality of first segmentation points.

[0047] In the second step, the corresponding first segmentation points on opposite sides of the second quadrilateral are respectively connected to form segmentation lines to obtain a first initial grid.

[0048] In this embodiment, corresponding first segmentation points on opposite sides are respectively connected to form segmentation lines, and the segmentation lines intersect to form a unit grid to obtain a first initial grid.

[0049] Finally, based on the actual calibrated size, the actual size of the unit grid in the first initial grid is determined to obtain a grid.

[0050] In this embodiment, the size of the corresponding side of the unit grid can be determined according to the size of each side in the actual calibrated size, so as to obtain the actual size of the unit grid and calibrate the grid.

[0051] For example, taking the vehicle as an excavator, the original image collected by the camera is as follows: Figure 4 As shown, the excavator coordinate system is Figure 2 As shown. The coordinate system of the excavator can be a coordinate system established with the projection of the rotation center of the excavator body on the ground as the origin, the X-axis as the forward movement direction of the excavator, the Y-axis as perpendicular to the X-axis and pointing to the left, and the Z-axis as perpendicular to the X-axis and pointing upward. Find the corresponding coordinate system in the excavator coordinate system Point, measured The distance between meters, and then find the points in the excavator coordinate system respectively , so that the line segment The vertical YZ plane extends forward and is parallel to the X axis, The distance is the maximum operating radius of the excavator , and then find the corresponding The pixel coordinates of the corresponding pixel points are also known, so the actual calibration size and the size of the second quadrilateral are obtained. , Divide evenly Duan, yes , Divide evenly Segment. Find the edge and edge on ( -1) pixel coordinates of the segmentation points, and and edge on ( -1) pixel coordinates of the segmentation points; connecting edges and edge The corresponding split point on , The corresponding segmentation points on the graph are as follows. Each unit grid represents a square with a side length of 1 meter in the excavator coordinate system, forming a grid.

[0052] By obtaining the actual calibration size, obtaining the calibration image, and determining the pixel points corresponding to the four points in the calibration image, a second quadrilateral is obtained, and according to the preset segmentation rule, a segmentation line is constructed in the second quadrilateral to obtain a first initial grid; based on the actual calibration size, the actual size of the unit grid in the first initial grid is determined, so that the grid can be calibrated quickly and accurately in the imaged image view. The preset segmentation rule can be used to set the number of unit grids as needed to meet different user needs.

[0053] In some embodiments, the grid calibration process includes the following steps: First, an actual calibration size is obtained, where the actual calibration size is the size of a first quadrilateral in the vehicle coordinate system, where the first quadrilateral is a rectangle formed by four points selected on the ground in front of the vehicle; In this embodiment, four vertices may be selected on the ground in front of the camera to form the four vertices of a rectangle, and the sizes of the sides of the first quadrilateral are determined according to the positions of the four vertices, that is, the actual calibration size is obtained. It should be noted that the larger the rectangle, the more accurate the constructed mesh.

[0054] Then, a calibration image is acquired, and pixel points corresponding to the four points are determined in the calibration image to obtain a second quadrilateral, wherein the calibration image and the image to be displayed are acquired at the same position, and at least one set of opposite sides in the second quadrilateral is parallel to one side of the calibration image; In this embodiment, the image viewing angle of the calibration image is the original viewing angle, that is, the image viewing angle of the imaging. The calibration image may be an image after the image acquisition device acquires the image and performs distortion correction. In the absence of distortion correction, the calibration image is the image acquired by the image acquisition device. The acquisition position of the calibration image and the image to be displayed being the same means that the position and angle of the acquisition device are the same when the two images are acquired. The calibration image may also be the image to be displayed. The second quadrilateral may be a trapezoid, the upper and lower sides of the trapezoid are parallel, and are parallel to one side of the calibration image. It should be noted that either of the upper and lower sides of the second quadrilateral may be on one side of the calibration image.

[0055] Then, the second quadrilateral is perspective-transformed into a top view to obtain a third quadrilateral; In this embodiment, the perspective transformation can be to transform the coordinates of all pixels in the second quadrilateral through The matrix projection is transformed into pixel coordinates of a top-down perspective, thereby obtaining a third quadrilateral.

[0056] Then, according to a preset segmentation rule, a segmentation line is constructed in the third quadrilateral to obtain a second initial grid; In this embodiment, the third quadrilateral is an image from a top-down perspective, and the preset segmentation rule is segmentation into a plurality of unit grids. According to the number of unit grids to be segmented, segmentation lines are constructed in the third quadrilateral to obtain a second initial grid.

[0057] In some embodiments, constructing a segmentation line in the third quadrilateral according to a preset segmentation rule to obtain a second initial grid includes: The first step is to perform uniform segmentation on each side of the third quadrilateral according to a preset segmentation rule to obtain a plurality of second segmentation points; In this embodiment, the number of segmentation points on each edge can be determined according to a preset segmentation rule, thereby obtaining a plurality of second segmentation points.

[0058] In the second step, corresponding second segmentation points on opposite sides of the third quadrilateral are respectively connected to form segmentation lines to obtain a second initial grid.

[0059] In this embodiment, corresponding second segmentation points on opposite sides are connected respectively to form segmentation lines, and the segmentation lines intersect to form a unit grid to obtain a second initial grid.

[0060] Finally, based on the actual calibrated size, the actual size of the unit grid in the second initial grid is determined to obtain a grid.

[0061] In this embodiment, the size of the corresponding side of the unit grid can be determined according to the size of each side in the actual calibration size, so as to obtain the actual size of the unit grid and calibrate the grid. The grid is a grid in a bird's-eye view.

[0062] For example, taking the vehicle as an excavator, the original image collected by the camera is as follows: Figure 4 As shown, the excavator coordinate system is Figure 2 The image height is pixels, width Pixels, are the lower left and lower right vertices in the original image, and point The coordinates in the pixel coordinate system are , pixel The coordinates in the pixel coordinate system are The coordinate system of the excavator can be a coordinate system established with the projection of the rotation center of the excavator on the ground as the origin, along the forward movement direction of the excavator, the X axis is horizontal, the Y axis is perpendicular to the X axis and points to the left, and the Z axis is perpendicular to the X axis and points upward. Find the corresponding coordinate system in the excavator coordinate system Point, measured The distance between meters, and then find the points in the excavator coordinate system respectively , so that the line segment The vertical YZ plane extends forward and is parallel to the X axis, The distance is the maximum operating radius of the excavator , and then find the corresponding The pixel coordinates of the corresponding pixel points are also known, so the actual calibration size and the size of the second quadrilateral are obtained. The image size ratio is kept consistent with the rectangular size ratio in the excavator vehicle coordinate system. Perspective transform to a new rectangle . The coordinates in the pixel coordinate system are , The coordinates in the pixel coordinate system are , The coordinates in the pixel coordinate system are , The coordinates in the pixel coordinate system are So we can get four pairs of matching points , , , through these four pairs of matching points, we can also get the homography matrix of perspective transformation According to the division of each unit grid into a square with a side length of 1 meter in the vehicle coordinate system, the image can be evenly divided into L rows and W columns, with a row spacing of Pixels, the distance between each column Pixels. It is easy to find the row dividing line at the edge and edge on ( -1) The pixel coordinates of the intersection points and the dividing line on the edge and edge on ( -1) pixel coordinates of the intersection point. Figure 5 , Figure 5 A top view with grid lines according to an embodiment of the present application is schematically shown.

[0063] It should be noted that the above constructed grid is a rectangular grid. In some embodiments, in addition to the rectangular grid, a fan-shaped grid can also be displayed in the top view. Fig.12 , Fig.12 A schematic top view with fan-shaped grid lines according to an embodiment of the present application is shown. It should be noted that: Fig.12 The center of the circle shown in the figure is the camera. In the specific implementation, other positions can also be selected as the center of the circle. In the above example, the construction of the fan-shaped grid includes the following process: Perspective transform to a new rectangle Then, select the center coordinates of the circle in the top view. ,generally = , that is, the ratio of the number of pixels in the length and width directions of the image to the distance in the corresponding direction in the excavator coordinate system is equal. Then, let the radius of the semicircle be , 3, ..., make a series of concentric circles, and then use the coordinates of the center of the circle Lead out ( ) rays, the angle between two adjacent rays is fixed, dividing the semicircle into A fan-shaped grid is obtained.

[0064] By obtaining the actual calibration size, obtaining the calibration image, and determining the pixel points corresponding to the four points in the calibration image, a second quadrilateral is obtained, and the second quadrilateral is perspective-transformed into a top view to obtain a third quadrilateral; according to a preset segmentation rule, a segmentation line is constructed in the third quadrilateral to obtain a second initial grid; based on the actual calibration size, the actual size of the unit grid in the second initial grid is determined, so that the grid under the top view can be obtained.

[0065] In some embodiments, after uniform segmentation is performed on each side of the third quadrilateral according to a preset segmentation rule to obtain a plurality of second segmentation points, the method further includes the following steps: First, the plurality of second segmentation points are perspectively transformed into the first quadrilateral to obtain a plurality of third segmentation points; In this embodiment, the perspective transformation can be performed by a homography matrix The inverse matrix of is used to perspective transform the original image, that is, transform it into the first quadrilateral, and obtain multiple third segmentation points.

[0066] Then, corresponding third segmentation points on opposite sides of the first quadrilateral are connected respectively to form segmentation lines, thereby obtaining a third initial grid; In this embodiment, by connecting the third segmentation points corresponding to the opposite sides, a mesh can be formed in the first quadrilateral to obtain a third initial mesh.

[0067] Finally, based on the actual calibrated size, the actual size of the unit grid in the third initial grid is determined to obtain a grid.

[0068] In this embodiment, the size of the corresponding side of the unit grid can be determined according to the size of each side in the actual calibrated size, so as to obtain the actual size of the unit grid and calibrate the grid.

[0069] For example: In the above example, find the row dividing line on the edge and edge on ( -1) The pixel coordinates of the intersection points and the dividing line on the edge and edge on ( -1) pixel coordinates of the intersection points. The above calculations are for the intersection points in the top view, which can then be obtained through the homography matrix The inverse matrix of is used to transform the perspective image to the original image, and the intersection points corresponding to the opposite edges are connected to form a grid in the original image. Figure 6 , Figure 6 The original viewing angle image with grid lines according to an embodiment of the present application is schematically shown.

[0070] After determining a plurality of second segmentation points on each side of the third quadrilateral according to a preset segmentation rule, the plurality of second segmentation points are perspectively transformed into the first quadrilateral to obtain a plurality of third segmentation points, and the corresponding third segmentation points on opposite sides of the first quadrilateral are respectively connected to form segmentation lines to obtain a third initial grid. Based on the actual calibrated size, the actual size of the unit grid in the third initial grid is determined, so that the grid under the original viewing angle can be obtained, which is helpful for grid display under the original viewing angle.

[0071] It should be noted that, since the bird's-eye view is more intuitive, in specific implementation, a bird's-eye view grid is preferred. The calibration image and the image to be displayed are acquired at the same location, which means that the calibration image and the image to be displayed are acquired at the same location, for example: the image to be displayed is taken by a camera directly in front of the vehicle, and correspondingly, the calibration image is also taken by a camera directly in front of the vehicle. Thereby ensuring the validity of the grid. The above-mentioned adding of the pre-calibrated grid to the image to be displayed may be superimposing the image of the pre-calibrated grid on the image to be displayed, or may be drawing a corresponding grid on the image to be displayed according to the pixel coordinates of the pre-calibrated grid, thereby obtaining a grid image and displaying it.

[0072] In some embodiments, for a wide-angle camera, due to the presence of distortion, distortion correction may be performed after capturing the image, that is, the image to be displayed is a corrected image; accordingly, adding a pre-calibrated grid to the image to be displayed to obtain a grid image includes: First, a pre-calibrated grid is added to the image to be displayed to obtain a distortion-free grid image; Then, the undistorted grid image is converted into a distorted image to obtain a grid image.

[0073] In this embodiment, for a wide-angle camera, the wide-angle viewing angle can first be corrected into a distortion-free image through barrel distortion, that is, the image to be displayed is a corrected image, and then the grid lines are displayed on the distortion-free image, and then the distortion-free grid image is converted into a barrel-type distortion image, that is, a grid image is obtained, that is, the grid is displayed in the original distorted image, so that the displayed field of view is larger.

[0074] In specific implementation, the image collected by a single high-definition camera has two display modes, one displays grid lines in the original view image, and one displays grid lines in the top view; while the wide-angle camera has three display modes, one displays grid lines in the original barrel distorted image, one displays grid lines in the original view image, and one displays grid lines in the top view. The comparison of different display modes is shown in Table 1 below, and users can choose different display modes according to actual needs.

[0075] Table 1: Comparison of different display modes

[0076] In some embodiments, adding a pre-calibrated grid to the image to be displayed to obtain a grid image includes: First, determining whether the image to be displayed and the image of the pre-calibrated grid have different viewing angles; In this embodiment, the image to be displayed may be a plurality of image viewing angles, for example, it may be an original viewing angle or a top-down viewing angle. Similarly, the image viewing angle of the pre-calibrated grid may also be a plurality of viewing angles. It may be determined whether the image viewing angles of the two are different.

[0077] Then, when it is determined that the image to be displayed and the pre-calibrated grid have different image perspectives, the image to be displayed or the pre-calibrated grid is subjected to view conversion to obtain the image to be displayed and the grid with the same image perspective; In this embodiment, if the image perspectives of the two images are different, the image perspectives of the two images need to be unified, and the view conversion is performed on one of them.

[0078] Finally, the image to be displayed and the grid having the same image viewing angle are superimposed to obtain a grid image.

[0079] In this embodiment, for example, if the image to be displayed is a top view and the pre-calibrated grid is the original viewing angle, the homography matrix can be used. The inverse matrix of is used to convert the image to be displayed to the original viewing angle, and then superimposes it with the pre-calibrated grid to display the grid in the image of the original viewing angle. It can also be done by using the homography matrix The pre-calibrated grid is converted into a top view and then superimposed with the image to be displayed to display the grid in the image from a top-down perspective.

[0080] It should be noted that if the image to be displayed and the image of the pre-calibrated grid have the same viewing angle, the pre-calibrated grid is directly added to the image to be displayed to obtain a grid image.

[0081] By judging whether the image perspectives of the image to be displayed and the pre-calibrated grid are different; when it is determined that the image perspectives of the image to be displayed and the pre-calibrated grid are different, the image to be displayed or the pre-calibrated grid is subjected to view conversion to make the image perspectives consistent, and then the image to be displayed and the grid with consistent image perspectives are superimposed to ensure that the grid in the grid image is accurate and reliable.

[0082] In some embodiments, there are multiple images to be displayed, and the acquisition positions of the images to be displayed are different; adding the pre-calibrated grid to the images to be displayed to obtain the grid image includes: First, corresponding pre-calibrated grids are added to the images to be displayed to obtain a plurality of initial grid images, each of which has an image viewing angle of a bird's-eye view; In this embodiment, considering that there are multiple cameras in the vehicle, multiple images can be obtained at the same time, that is, the number of images to be displayed is multiple, and the acquisition positions of each image to be displayed are different. When performing grid display, a corresponding pre-calibrated grid can be added to each image to be displayed to obtain multiple initial grid images. In order to facilitate image stitching, the image perspective of the initial grid images is a top-down perspective.

[0083] Then, the multiple initial grid images are spliced ​​to obtain a grid image.

[0084] In this embodiment, the stitching can be the stitching of 360° panoramic images. Figure 7 , Figure 7 The 360° surround image with grid lines according to an embodiment of the present application is schematically shown. The above stitching can be achieved by using existing image stitching technology, which will not be described in detail here.

[0085] By joining the top view with grid lines to the 360° surround view image, an auxiliary grid can be formed in the 360° surround view image, and the grid can be displayed in the surround view image, which further helps the user judge the distance of the object.

[0086] In the above implementation process, the image to be displayed is obtained; a pre-calibrated grid is added to the image to be displayed to obtain a grid image, and the grid image is displayed; the pre-calibrated grid includes a plurality of unit grids, and the unit grid corresponds to an actual size of the unit grid, and the actual size of the unit grid is the size of the unit grid in the vehicle coordinate system, and the vehicle coordinate system is a coordinate system established with a specific position or component of the vehicle as the origin. By adding the pre-calibrated grid to the image to be displayed, the displayed image has a grid, and the actual size of the unit grid in the grid is the size of each unit grid in the vehicle coordinate system. Therefore, according to the number of unit grids between the object and the vehicle in the image, the distance between the object and the vehicle in the image can be quickly determined, so that the user can determine the distance between the object and the vehicle in the image from the image alone, which is helpful for the user to operate the vehicle.

[0087] In some embodiments, the method further comprises: performing view conversion and / or distortion processing on the grid image to obtain a converted grid image.

[0088] In this embodiment, since the image view of the grid image can be a top view, the view can also be converted to the original view, that is, by using the homography matrix The inverse matrix of the initial grid image is used to convert the image view of the initial grid image to the original viewing angle, so as to display the grid at the original angle. Furthermore, the grid image can also be distorted, that is, the undistorted image is converted into a barrel-distorted image, and the specific conversion is opposite to the barrel distortion correction. The above-mentioned view conversion and distortion processing can be implemented separately or together, and can be set as needed.

[0089] By performing view conversion and / or distortion processing on the grid image, the grid can be displayed in the original viewing angle and in the original barrel-distorted image, thereby realizing multiple display modes to meet various needs of users.

[0090] In some embodiments, the image perspective of the grid image is a top-down perspective, and the method further includes: First, obtain the driving speed and body rotation angle in real time; In this embodiment, the driving speed can be obtained by detecting the rotation speed of the driving wheel in real time. The body rotation angle refers to the angle of the body relative to the chassis. Taking the vehicle as an excavator as an example, the body rotation angle can be obtained in real time. (the angle at which the excavator body rotates relative to the tracks), and the speed of the left and right tracks , .

[0091] Then, based on the driving speed and the vehicle body rotation angle, a predicted vehicle path is drawn in the grid image to obtain a path grid top view.

[0092] In this embodiment, taking the vehicle as an excavator as an example, the movement path of the chassis mainly includes two types: linear movement and circular movement. and When the speeds of the left and right tracks are equal, the chassis movement path is a straight line. and When they are not equal, the chassis motion path is an arc (circumference), and its arc radius in the vehicle coordinate system is , is the left track speed value, is the right track speed value, is the distance between the left and right tracks.

[0093] When the chassis's motion path is a straight line, the initial point can be determined in the grid image. The initial point is the midpoint of the track center line. Then a straight line can be drawn in the grid image. The straight line passes through the initial point and the angle between the straight line and the vertical direction is the body rotation angle. Thus, the predicted vehicle path can be drawn in the grid image. Please refer to Figure 8 and Fig.10 , Figure 8 A schematic diagram of a straight line path predicted by the chassis in a top view of a front-view camera according to an embodiment of the present application is shown schematically. Fig.10 The schematic diagram of the straight path predicted by the chassis in the 360° panoramic view according to the embodiment of the present application is shown schematically. Figure 8 In the figure, the solid line part is the part displayed in the top view of the front camera, and the dotted line part will not be displayed in the top view of the front camera.

[0094] When the chassis moves in a circular motion, the coordinates of the center of the circle in the image pixel coordinate system can be determined based on the rotation angle of the vehicle body, the radius of the arc, the relative position relationship between the midpoint of the camera and the track center line, and the ratio of the number of pixels in the length and width direction of the image to the distance in the corresponding direction in the excavator coordinate system. Fig. 9 , Fig. 9 The schematic diagram shows a schematic diagram of a predicted arc path of the chassis in a top view of a front-view camera according to an embodiment of the present application. Fig. 9 In the above example, the origin of the image coordinate system is set at the upper left corner of the grid image, the y axis is downward, and the x axis is to the right. Then the coordinates of the center of the circle in the pixel coordinate system ( ) can be expressed as: , in, is the pixel width of the grid image, is the pixel height of the grid image, is the distance corresponding to the pixel width in the excavator coordinate system, is the distance corresponding to the pixel height in the excavator coordinate system, is the body rotation angle, is the arc radius in the vehicle coordinate system. The front-view camera is installed in the front right of the midpoint of the track center line. is the distance between the optical center of the front camera and the midpoint of the line connecting the centers of the crawlers in the Y-axis direction of the excavator coordinate system, It is the distance between the optical center of the front camera and the midpoint of the track center line in the X-axis direction of the excavator coordinate system. After determining the coordinates of the center of the circle, according to the coordinates of the center of the circle in the pixel coordinate system and the pixel distance of the arc radius, an arc can be drawn in the grid image to obtain the vehicle predicted path. Please refer to Fig.11 , Fig.11 A schematic diagram of a predicted arc path of a chassis displayed in a 360° surround view according to an embodiment of the present application is schematically shown.

[0095] It should be noted that, when the grid is a sector-shaped grid, the grid display in the 360° panoramic view may be in the form of concentric circles. Taking the vehicle as an excavator as an example, when the grid is a sector-shaped grid, the grid display in the 360° panoramic view may be in the form of concentric circles with the midpoint of the track center line as the center.

[0096] By acquiring the driving speed and the body rotation angle in real time, a predicted vehicle path is drawn in the grid image based on the driving speed and the body rotation angle. The predicted vehicle path can be added to the grid image to display the predicted vehicle movement path in the image, so that the predicted path can be known from the image alone, which helps users operate the vehicle.

[0097] In some embodiments, the method further includes: performing view conversion and / or distortion processing on the path grid top view to obtain a path grid image.

[0098] In this embodiment, since the image view of the path grid top view is a top view, the view can also be converted to the original view, that is, by using the homography matrix The inverse matrix of the path grid image is used to convert the image view of the path grid image to the original viewing angle, so that the grid and the predicted path can be displayed at the original angle. Furthermore, the path grid top view can also be distorted, that is, the undistorted image is converted into a barrel-distorted image. The specific conversion is the opposite of the barrel distortion correction. The above view conversion and distortion processing can be implemented separately or together, and can be set as needed. In this way, the predicted path of the vehicle can be displayed in the top view first, and then the homography matrix can be used to calculate the predicted path of the vehicle. The inverse matrix of is projected and transformed to the original view image. In addition, if the chassis moves backward, it can be displayed in the image captured by the rear-view camera, which will not be described here.

[0099] By performing view conversion and / or distortion processing on the path grid top view, it is possible to display grid lines and predicted paths in the original viewing angle of the path grid top view and in the original barrel distorted image, thereby achieving multiple display modes to meet various user needs.

[0100] The following is a specific example, please refer to Fig.13 , Fig.13 The schematic diagram of the excavator image display steps according to the embodiment of the present application is schematically shown. Taking the vehicle as an excavator as an example, first, the original image and the data of each encoder are obtained, and then the barrel distortion of the image collected by the wide-angle camera is corrected, and then the image of the original perspective is converted into a top view, and the grid and the predicted chassis trajectory are displayed in the top view, and the top view is spliced ​​into a 360° panoramic view, and then the top view is converted into the original perspective image, and finally the original perspective image is converted into a wide-angle barrel distortion image.

[0101] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0102] The present embodiment provides a vehicle image auxiliary display system, comprising an image processing device, a display device and at least one image acquisition device arranged on a vehicle; the image acquisition device is used to obtain an image to be displayed, and send the image to be displayed to the image processing device; the image processing device is used to add a pre-calibrated grid to the image to be displayed to obtain a grid image, and send the grid image to the display device for display, the pre-calibrated grid includes a plurality of unit grids, the unit grid corresponds to an actual size of the unit grid, and the actual size of the unit grid is the size of the unit grid in a vehicle coordinate system, and the vehicle coordinate system is a coordinate system established with a specific position or component of the vehicle as the origin.

[0103] In this embodiment, the image processing device may be an image processing host, and the image acquisition device includes a camera. Fig.14 , Fig.14 The electrical connection diagram of the excavator according to the embodiment of the present application is schematically shown. The image processing host receives the body rotation angle transmitted by the excavator controller (the angle at which the excavator body rotates relative to the tracks), and the speed of the left and right tracks , and then the predicted movement path of the grid and chassis is displayed in the image through the display screen. Among them, the excavator controller obtains the body rotation angle through the encoder of the body rotation center, and the excavator controller obtains the speed of the left and right crawlers through the encoders of the left and right crawlers.

[0104] In the above implementation process, the image to be displayed is acquired by the image acquisition device, and the image to be displayed is sent to the image processing device. The image processing device adds a pre-calibrated grid to the image to be displayed to obtain a grid image, and sends the grid image to the display device for display. The pre-calibrated grid includes a plurality of unit grids, and the unit grid corresponds to an actual size of the unit grid. The actual size of the unit grid is the size of the unit grid in the vehicle coordinate system. The vehicle coordinate system is a coordinate system established with a specific position or component of the vehicle as the origin. By adding the pre-calibrated grid to the image to be displayed, the displayed image has a grid, and the actual size of the unit grid in the grid is the size of each unit grid in the vehicle coordinate system. Therefore, according to the number of unit grids between the object and the vehicle in the image, the distance between the object and the vehicle in the image can be quickly determined, so that the user can determine the distance between the object and the vehicle in the image from the image alone, which is helpful for the user to operate the vehicle.

[0105] This embodiment provides a vehicle, including a vehicle image auxiliary display system, wherein the vehicle image auxiliary display system uses the above-mentioned image display method to display images.

[0106] In this embodiment, the above-mentioned vehicles are passenger cars, engineering vehicles, such as skid loaders, wheel excavators, etc. The above-mentioned image display method can display a grid in the image, and the actual size of the unit grid in the grid is the size of each unit grid in the vehicle coordinate system. Therefore, according to the number of unit grids between the object and the vehicle in the image, the distance between the object and the vehicle in the image can be quickly determined, so that the user can determine the distance between the object and the vehicle in the image from the image alone, which is helpful for the user to operate the vehicle.

[0107] Please see Fig.15 , Fig.15 The structure diagram of an image display device according to an embodiment of the present application is schematically shown. This embodiment provides an image display device, including an acquisition module 410 and a display module 420, wherein: An acquisition module 410 is used to acquire an image to be displayed; The display module 420 is used to add a pre-calibrated grid to the image to be displayed, obtain a grid image, and display the grid image; the pre-calibrated grid includes a plurality of unit grids, each of which corresponds to an actual size of the unit grid, and the actual size of the unit grid is the size of the unit grid in the vehicle coordinate system, and the vehicle coordinate system is a coordinate system established with a specific position or component of the vehicle as the origin.

[0108] The image display device includes a processor and a memory. The acquisition module 410 and the display module 420 are stored in the memory as program units. The processor executes the program units stored in the memory to implement corresponding functions.

[0109] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and image display is achieved by adjusting kernel parameters.

[0110] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0111] An embodiment of the present invention provides a machine-readable storage medium on which a program is stored. When the program is executed by a processor, the image display method is implemented.

[0112] An embodiment of the present invention provides a processor, which is used to run a program, wherein the image display method is executed when the program is run.

[0113] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.16 As shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05 and a memory (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A06. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor A01, an image display method is implemented. The display screen A04 of the computer device can be a liquid crystal display or an electronic ink display, and the input device A05 of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0114] Those skilled in the art will understand that Fig.16The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0115] In one embodiment, the image display device provided by the present application can be implemented in the form of a computer program. The computer program can be Fig.16 The computer device shown in the figure is run on the computer device. The memory of the computer device can store various program modules constituting the image display device, such as, Fig.15 The acquisition module 410 and the display module 420 are shown. The computer program composed of various program modules enables the processor to execute the steps of the image display method of each embodiment of the present application described in this specification.

[0116] Fig.16 The computer device shown can be Fig.15 The acquisition module 410 in the image display device shown executes step 210. The computer device may execute step 220 through the display module 420.

[0117] An embodiment of the present application provides an electronic device, the electronic device comprising: at least one processor; a memory connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, the at least one processor implements the above-mentioned image display method by executing the instructions stored in the memory, and the processor implements the following steps when executing the instructions: Get the image to be displayed; Adding a pre-calibrated grid to the image to be displayed to obtain a grid image, and displaying the grid image; The pre-calibrated grid includes a plurality of unit grids, each of which corresponds to an actual unit grid size, which is the size of the unit grid in a vehicle coordinate system. The vehicle coordinate system is a coordinate system established with a specific position or component of the vehicle as the origin.

[0118] In one embodiment, obtaining the image to be displayed includes: Get the original image; Distortion correction and / or view conversion are performed on the original image to obtain an image to be displayed.

[0119] In one embodiment, the image to be displayed is a rectified image; The step of adding the pre-calibrated grid to the image to be displayed to obtain a grid image includes: Adding a pre-calibrated grid to the image to be displayed to obtain a distortion-free grid image; The undistorted grid image is converted into a distorted image to obtain a grid image.

[0120] In one embodiment, the adding of the pre-calibrated grid to the image to be displayed to obtain the grid image includes: Determining whether the image to be displayed and the image of the pre-calibrated grid have different viewing angles; When it is determined that the image to be displayed and the pre-calibrated grid have different image perspectives, performing view conversion on the image to be displayed or the pre-calibrated grid to obtain the image to be displayed and the grid with the same image perspective; The image to be displayed and the grid having the same image viewing angle are superimposed to obtain a grid image.

[0121] In one embodiment, the number of the images to be displayed is multiple, and the acquisition positions of the images to be displayed are different; The step of adding the pre-calibrated grid to the image to be displayed to obtain a grid image includes: Adding corresponding pre-calibrated grids to the images to be displayed respectively to obtain a plurality of initial grid images, wherein the image perspective of each initial grid image is a bird's-eye view; The multiple initial grid images are spliced ​​together to obtain a grid image.

[0122] In one embodiment, the method further comprises: The grid image is subjected to view conversion and / or distortion processing to obtain a converted grid image.

[0123] In one embodiment, the image perspective of the grid image is a top-down perspective, and the method further includes: Obtain driving speed and vehicle body rotation angle in real time; Based on the driving speed and the vehicle body rotation angle, a predicted vehicle path is drawn in the grid image to obtain a path grid top view.

[0124] In one embodiment, the grid calibration process includes: Acquire an actual calibration size, where the actual calibration size is a size of a first quadrilateral in the vehicle coordinate system, where the first quadrilateral is a rectangle formed by four points selected on the ground in front of the vehicle; Acquire a calibration image, and determine pixel points corresponding to the four points in the calibration image to obtain a second quadrilateral, wherein the calibration image and the image to be displayed are acquired at the same location, and at least one set of opposite sides in the second quadrilateral is parallel to one side of the calibration image; According to a preset segmentation rule, a segmentation line is constructed in the second quadrilateral to obtain a first initial grid; Based on the actual calibrated size, the actual size of the unit grid in the first initial grid is determined to obtain a grid.

[0125] In one embodiment, constructing a segmentation line in the second quadrilateral according to a preset segmentation rule to obtain a first initial grid includes: According to a preset segmentation rule, uniform segmentation is performed on each side of the second quadrilateral to obtain a plurality of first segmentation points; The corresponding first segmentation points on opposite sides of the second quadrilateral are respectively connected to form segmentation lines to obtain a first initial grid.

[0126] In one embodiment, the grid calibration process includes: Acquire an actual calibration size, where the actual calibration size is a size of a first quadrilateral in the vehicle coordinate system, where the first quadrilateral is a rectangle formed by four points selected on the ground in front of the vehicle; Acquire a calibration image, and determine pixel points corresponding to the four points in the calibration image to obtain a second quadrilateral, wherein the calibration image and the image to be displayed are acquired at the same location, and at least one set of opposite sides in the second quadrilateral is parallel to one side of the calibration image; Transform the second quadrilateral into a top view to obtain a third quadrilateral; According to a preset segmentation rule, a segmentation line is constructed in the third quadrilateral to obtain a second initial grid; Based on the actual calibrated size, the actual size of the unit grid in the second initial grid is determined to obtain a grid.

[0127] In one embodiment, constructing a segmentation line in the third quadrilateral according to a preset segmentation rule to obtain a second initial grid includes: According to a preset segmentation rule, uniform segmentation is performed on each side of the third quadrilateral to obtain a plurality of second segmentation points; The corresponding second segmentation points on opposite sides of the third quadrilateral are respectively connected to form segmentation lines to obtain a second initial grid.

[0128] In one embodiment, after uniform segmentation is performed on each side of the third quadrilateral according to a preset segmentation rule to obtain a plurality of second segmentation points, the method further includes: Perspectively transforming the plurality of second segmentation points into the first quadrilateral to obtain a plurality of third segmentation points; respectively connecting corresponding third segmentation points on opposite sides of the first quadrilateral to form segmentation lines, thereby obtaining a third initial grid; Based on the actual calibrated size, the actual size of the unit grid in the third initial grid is determined to obtain a grid.

[0129] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0130] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0131] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0133] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0134] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0135] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0136] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0137] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. An image display method, characterized in that: include: Get the image to be displayed; Adding a pre-calibrated grid to the image to be displayed to obtain a grid image, and displaying the grid image; The pre-calibrated grid includes a plurality of unit grids, each of which corresponds to an actual unit grid size, which is the size of the unit grid in a vehicle coordinate system. The vehicle coordinate system is a coordinate system established with a specific position or component of the vehicle as the origin.

2. The method according to claim 1, characterized in that The step of obtaining the image to be displayed comprises: Get the original image; Distortion correction and / or view conversion are performed on the original image to obtain an image to be displayed.

3. The method according to claim 1, characterized in that The image to be displayed is a corrected image; The step of adding the pre-calibrated grid to the image to be displayed to obtain a grid image includes: Adding a pre-calibrated grid to the image to be displayed to obtain a distortion-free grid image; The undistorted grid image is converted into a distorted image to obtain a grid image.

4. The method according to claim 1, characterized in that: The step of adding the pre-calibrated grid to the image to be displayed to obtain a grid image includes: Determining whether the image to be displayed and the image of the pre-calibrated grid have different viewing angles; When it is determined that the image to be displayed and the pre-calibrated grid have different image perspectives, performing view conversion on the image to be displayed or the pre-calibrated grid to obtain the image to be displayed and the grid with the same image perspective; The image to be displayed and the grid having the same image viewing angle are superimposed to obtain a grid image.

5. The method according to claim 1, characterized in that There are multiple images to be displayed, and the acquisition positions of the images to be displayed are different; The step of adding the pre-calibrated grid to the image to be displayed to obtain a grid image includes: Adding corresponding pre-calibrated grids to the images to be displayed respectively to obtain a plurality of initial grid images, wherein the image perspective of each initial grid image is a bird's-eye view; The multiple initial grid images are spliced ​​together to obtain a grid image.

6. The method according to claim 1, characterized in that The method further comprises: The grid image is subjected to view conversion and / or distortion processing to obtain a converted grid image.

7. The method according to claim 1, characterized in that The image perspective of the grid image is a top-down perspective, and the method further includes: Obtain driving speed and vehicle body rotation angle in real time; Based on the driving speed and the vehicle body rotation angle, a predicted vehicle path is drawn in the grid image to obtain a path grid top view.

8. The method according to claim 1, characterized in that The grid calibration process includes: Acquire an actual calibration size, where the actual calibration size is a size of a first quadrilateral in the vehicle coordinate system, where the first quadrilateral is a rectangle formed by four points selected on the ground in front of the vehicle; Acquire a calibration image, and determine pixel points corresponding to the four points in the calibration image to obtain a second quadrilateral, wherein the calibration image and the image to be displayed are acquired at the same location, and at least one set of opposite sides in the second quadrilateral is parallel to one side of the calibration image; According to a preset segmentation rule, a segmentation line is constructed in the second quadrilateral to obtain a first initial grid; Based on the actual calibrated size, the actual size of the unit grid in the first initial grid is determined to obtain a grid.

9. The method according to claim 8, characterized in that The step of constructing a segmentation line in the second quadrilateral according to a preset segmentation rule to obtain a first initial grid includes: According to a preset segmentation rule, uniform segmentation is performed on each side of the second quadrilateral to obtain a plurality of first segmentation points; The corresponding first segmentation points on opposite sides of the second quadrilateral are respectively connected to form segmentation lines to obtain a first initial grid.

10. The method according to claim 1, characterized in that The grid calibration process includes: Acquire an actual calibration size, where the actual calibration size is a size of a first quadrilateral in the vehicle coordinate system, where the first quadrilateral is a rectangle formed by four points selected on the ground in front of the vehicle; Acquire a calibration image, and determine pixel points corresponding to the four points in the calibration image to obtain a second quadrilateral, wherein the calibration image and the image to be displayed are acquired at the same location, and at least one set of opposite sides in the second quadrilateral is parallel to one side of the calibration image; Transform the second quadrilateral into a top view to obtain a third quadrilateral; According to a preset segmentation rule, a segmentation line is constructed in the third quadrilateral to obtain a second initial grid; Based on the actual calibrated size, the actual size of the unit grid in the second initial grid is determined to obtain a grid.

11. The method according to claim 10, characterized in that The step of constructing a segmentation line in the third quadrilateral according to a preset segmentation rule to obtain a second initial grid includes: According to a preset segmentation rule, uniform segmentation is performed on each side of the third quadrilateral to obtain a plurality of second segmentation points; The corresponding second segmentation points on opposite sides of the third quadrilateral are respectively connected to form segmentation lines to obtain a second initial grid.

12. The method according to claim 11, characterized in that After uniformly dividing each side of the third quadrilateral according to the preset division rule to obtain a plurality of second division points, the method further includes: Perspectively transforming the plurality of second segmentation points into the first quadrilateral to obtain a plurality of third segmentation points; respectively connecting corresponding third segmentation points on opposite sides of the first quadrilateral to form segmentation lines, thereby obtaining a third initial grid; Based on the actual calibrated size, the actual size of the unit grid in the third initial grid is determined to obtain a grid.

13. An image display device, characterized in that: include: An acquisition module, used for acquiring an image to be displayed; A display module is used to add a pre-calibrated grid to the image to be displayed, obtain a grid image, and display the grid image; the pre-calibrated grid includes a plurality of unit grids, each unit grid corresponds to an actual unit grid size, and the actual unit grid size is the size of the unit grid in a vehicle coordinate system, and the vehicle coordinate system is a coordinate system established with a specific position or component of the vehicle as the origin.

14. A vehicle image auxiliary display system, characterized in that: It includes an image processing device, a display device and at least one image acquisition device arranged on a vehicle; the image acquisition device is used to obtain an image to be displayed and send the image to be displayed to the image processing device; the image processing device is used to add a pre-calibrated grid to the image to be displayed to obtain a grid image, and send the grid image to the display device for display; the pre-calibrated grid includes a plurality of unit grids, the unit grid corresponds to an actual size of the unit grid, and the actual size of the unit grid is the size of the unit grid in the vehicle coordinate system, and the vehicle coordinate system is a coordinate system established with a specific position or component of the vehicle as the origin.

15. A vehicle, characterized in that: It comprises a vehicle image auxiliary display system, and the vehicle image auxiliary display system adopts the image display method according to any one of claims 1 to 12 to display images.

16. An electronic device, characterized in that: The electronic device includes: at least one processor; a memory connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the image display method according to any one of claims 1 to 12 by executing the instructions stored in the memory.

17. A machine-readable storage medium having instructions stored thereon, characterized in that: When the instruction is executed by a processor, the processor is configured to execute the image display method according to any one of claims 1 to 12.

Images