Distance scale display method and device, construction vehicle and storage medium

Abstract

The invention discloses a distance scale display method and device, a construction vehicle and a storage medium, which are applied to the construction vehicle, and a camera device is arranged on a front hood of the construction vehicle. The method comprises the following steps: acquiring current pose information of a construction vehicle and a corresponding current acquisition image; target coordinate information of the distance scale is determined according to the current pose information and internal parameter data and external parameter data of the camera device, and the target coordinate information is used for indicating pixel information of the distance scale in the currently collected image; and according to the target coordinate information, displaying a distance scale on the currently acquired image. According to the method, the distance scale can be continuously adjusted on the acquired image along with the pose of the construction vehicle, so that more accurate distance information in front of the construction vehicle is provided for a driver who performs construction operation in a complex terrain.

Description

Technical Field

[0001] The present invention relates to the field of image technology, and in particular to a display method and device for a distance scale, a construction vehicle and a storage medium. Background Art

[0002] In traditional engineering construction processes, construction vehicles often face complex terrain and environmental conditions, such as cliffs, steep slopes and other terrain features. These features may cause accidents or damage to construction vehicles during operations, and even pose a safety threat to the driver. Therefore, a camera device is installed on the front hood of the construction vehicle to help the operator observe the terrain ahead, thereby assisting construction.

[0003] At present, during the operation of construction vehicles, a distance scale is attached based on the images captured by the camera equipment to help the driver judge the distance between the construction vehicle and the obstacle or work target in front. However, since the distance scale can only ensure that the distance information is relatively accurate when the construction vehicle is in a horizontal state, when the construction vehicle is in a non-horizontal state such as an inclined state, the distance information displayed will have a certain deviation, which will cause the driver to have errors in the distance information judged by the distance scale. Summary of the invention

[0004] The present invention provides a method and device for displaying a distance scale, a construction vehicle and a storage medium, which can realize that the distance scale is continuously adjusted on a captured image following the posture of a construction vehicle, thereby providing more accurate distance information in front of the construction vehicle for drivers performing construction work in complex terrain.

[0005] According to one aspect of the present invention, a method for displaying a distance scale is provided, which is applied to a construction vehicle, wherein a camera device is provided on the front hood of the construction vehicle; the method comprises:

[0006] Obtain the current posture information of the construction vehicle and its corresponding current captured image;

[0007] Determine target coordinate information of the distance scale according to the current posture information, the internal reference data and the external reference data of the camera device, wherein the target coordinate information is used to indicate pixel information of the distance scale in the current captured image;

[0008] According to the target coordinate information, a distance ruler is displayed on the current acquired image.

[0009] According to another aspect of the present invention, a distance scale display device is provided, which is applied to a construction vehicle, and a camera device is arranged on the front hood of the construction vehicle; the device comprises:

[0010] An acquisition module is used to obtain the current posture information of the construction vehicle and its corresponding current captured image;

[0011] A determination module, used to determine target coordinate information of the distance scale according to current posture information, internal reference data and external reference data of the camera device, wherein the target coordinate information is used to indicate pixel information of the distance scale in the current captured image;

[0012] The display module is used to display a distance scale on the current acquired image according to the target coordinate information.

[0013] According to another aspect of the present invention, there is provided a construction vehicle, the construction vehicle comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by at least one processor so that the at least one processor can execute the method for displaying a distance scale according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for displaying a distance scale according to any embodiment of the present invention when executed.

[0018] The method for displaying a distance scale provided by an embodiment of the present invention obtains the current posture information of the construction vehicle and the corresponding current collected image; determines the target coordinate information of the distance scale according to the current posture information, the internal reference data and the external reference data of the camera device; and displays the distance scale on the current collected image according to the target coordinate information. In the above technical scheme, on the one hand, the current collected image corresponding to the current posture information of the construction vehicle is obtained, which provides an accurate image basis for displaying the distance scale and guiding the driver to judge the distance information according to the distance scale. On the other hand, according to the real-time posture information of the construction vehicle, the original coordinate information of the distance scale in the self-vehicle coordinate system is converted and projected to the pixel information of the current collected image to obtain the target coordinate information, which solves the problem that the distance scale cannot change with the posture of the construction vehicle when the construction vehicle is in a non-horizontal state, and realizes that the positioning coordinate information of the distance scale is accurately converted in the process of the continuous change of the real-time posture of the vehicle, ensuring that the positioning coordinate information of the distance scale can be continuously adjusted with the real-time posture of the vehicle. Finally, according to the target coordinate information, a distance ruler is displayed on the current collected image, and the displayed distance ruler is used to guide the driver to drive the construction vehicle to perform construction work, so that the driver can judge the distance information between the front area and the construction vehicle according to the distance ruler. At the same time, this method solves the problem that when the construction vehicle is in an unbalanced state, the existing static ruler displayed on the current collected image will not change with the distortion of the image, and realizes that when the construction vehicle is in an unbalanced state, the current collected image is distorted, and the distance ruler can also be continuously adjusted according to the current posture information of the construction vehicle, so that the distance ruler displayed on the current collected image is more accurate. Further, the display method of the distance ruler of the present invention enables the driver to obtain accurate distance information in any posture of the construction vehicle through the distance ruler that can be continuously adjusted with the posture of the construction vehicle, which greatly improves the operation accuracy of the construction vehicle in complex terrain, reduces operation errors, improves work efficiency, and improves the safety and reliability of remote control.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A flow chart of a method for displaying a distance scale provided by an embodiment of the present invention;

[0022] Figure 2 A flowchart of another method for displaying a distance scale provided by an embodiment of the present invention;

[0023] Figure 3 An example diagram of a distance scale provided by an embodiment of the present invention;

[0024] Figure 4 A schematic structural diagram of a display device for a distance scale provided by an embodiment of the present invention;

[0025] Figure 5 It is a schematic structural diagram of a construction vehicle according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "current", "target", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Figure 1 A flow chart of a method for displaying a distance scale provided in an embodiment of the present invention. This embodiment can be applied to provide a distance scale for a construction vehicle, thereby guiding the driver to determine the distance information based on the distance scale. The method can be executed by a display device for the distance scale, which can be implemented in the form of hardware and / or software, and can be configured in a construction vehicle. In this embodiment, the display device for the distance scale can be configured in the master control of the construction vehicle and presented in the form of an electronic device. Figure 1As shown, the method includes:

[0029] S101, obtaining current posture information of a construction vehicle and its corresponding current captured image.

[0030] Among them, the current posture information is used to indicate the current change of the working posture of the construction vehicle. In this embodiment, the current posture information can be detected by an instrument that can detect posture changes, for example, by using an inclination sensor or an accelerometer. The current captured image is a construction image in front of the construction vehicle captured by a camera device set on the front hood of the construction vehicle when the construction vehicle is in the current posture. In this embodiment, the construction vehicles include construction machinery such as bulldozers, loaders, rollers, graders, excavators, and passenger vehicles that require forward or backward distance scales.

[0031] Specifically, when the driver drives the construction vehicle to perform a construction task, the current position information of the construction vehicle is obtained in real time, and the camera equipment is controlled to shoot the construction area in front to obtain the current captured image.

[0032] For example, in one implementation, if the construction vehicle is on a flat road, the current posture information obtained may be that the construction vehicle is in a balanced state, and the perspective of the area in front of the construction vehicle displayed by the currently acquired image is normal and the image is complete. In another implementation, if the construction vehicle is on a non-flat road, the current posture information of the construction vehicle may be tilted front and back or left and right, and the perspective of the area in front of the construction vehicle displayed by the currently acquired image may produce geometric distortion, incomplete image, or tilt of the road ahead. Therefore, it is necessary to obtain the current acquired image corresponding to the current posture information of the construction vehicle.

[0033] In this embodiment, the current captured image corresponding to the current posture information of the construction vehicle is obtained, which provides an accurate image basis for subsequently displaying a distance scale to guide the driver to judge the distance information according to the distance scale.

[0034] S102: Determine target coordinate information of the distance scale according to current posture information, internal reference data and external reference data of the camera device.

[0035] Among them, the internal parameter data refers to the parameters related to the characteristics of the camera device itself, which describe the internal characteristics of the camera, including its optical characteristics and sensor characteristics. The external parameter data refers to the position and posture of the camera device in the world coordinate system. In this embodiment, the external parameter data refers to the position and posture of the camera device in the self-vehicle coordinate system. The self-vehicle coordinate system is a coordinate system established with the projection position of the geometric center of the construction vehicle on the horizontal plane as the origin, the forward direction of the construction vehicle as the positive direction of the Y axis, and the right direction of the driver when driving the construction vehicle as the positive direction of the X axis. The target coordinate information is used to indicate the pixel information of the distance scale in the current captured image.

[0036] Specifically, the internal parameter data and external parameter data of the camera device can be obtained by calibrating the camera device. For example, the internal parameter data is obtained by calibrating the camera device indoors, and the internal parameter data can be represented by an internal parameter matrix. Also, the external parameter data is obtained by calibrating the camera device on the front hood of a construction vehicle, and the external parameter data can be represented by an external parameter matrix.

[0037] Exemplarily, the camera device uses an internal parameter matrix to represent the internal parameter data, and the internal parameter matrix is ​​in the following form:

[0038]

[0039] Among them, M1 is the internal parameter data (internal parameter matrix), f is the distance from the camera lens to the imaging sensor plane, f / dx is the focal length along the image x-axis, f / dy is the focal length along the image y-axis, μ0 and v0 are the coordinates of the principal point (optical center) in the image coordinate system, and in general, μ0 = image width w / 2, v0 = image height h / 2. In this embodiment, in order to cooperate with the external parameter matrix to solve the subsequent parameters to be solved, it is necessary to expand the original 3*3 matrix to a 3*4 matrix. Therefore, in the actual calculation process, the internal parameter matrix can be transformed as needed.

[0040] And, the camera device uses an external parameter matrix to represent the external parameter data, and the external parameter matrix is ​​in the following form:

[0041]

[0042] Among them, r 11 、r 12 、r 13 、r 21 、r 22 、r 23 、r 31 、r 32 and r 33 are all parameters in R, t x ,t y and t z are all parameters in T. Specifically, r11 、r 12 、r 13 、r 21 、r 22 、r 23 、r 31 、r 32 and r 33 is an element in the 3*3 rotation matrix R. For example, r 11 、r 12 、r 13 Represents the direction cosine of the X-axis in the camera coordinate system in the vehicle coordinate system. 21 、r 22 、r 23 Represents the direction cosine of the Y axis in the camera coordinate system in the vehicle coordinate system. 31 、r 32 、r 31 represents the direction cosine of the Z axis in the camera coordinate system in the vehicle coordinate system. Specifically, t x ,t y and t z is the element in the translation vector T. For example, t x Represents the displacement of the origin of the camera coordinate system relative to the origin of the vehicle coordinate system in the X-axis direction. y Represents the displacement of the origin of the camera coordinate system relative to the origin of the vehicle coordinate system in the Y-axis direction. z Represents the displacement of the origin of the camera coordinate system relative to the origin of the vehicle coordinate system in the Z-axis direction.

[0043] And, after the internal reference data and external reference data of the camera device are determined according to the above method, the target coordinate information can be calculated according to the following formula:

[0044] A′=M1M2M3A;

[0045] Among them, A′ is the target coordinate information, M1 is the internal parameter matrix, M2 is the external parameter matrix, M3 is the current posture information, and A is the original coordinate information corresponding to the target coordinate information. That is, the target coordinate information indicates the pixel information of the distance scale in the current captured image, and the original coordinate information indicates the coordinate information of the distance scale in the vehicle coordinate system.

[0046] Specifically, according to the above calculation method, the coordinate information of the distance scale in the vehicle coordinate system that can reflect its actual position can be converted into pixel information based on the pixels of the current captured image in a manner similar to projection.

[0047] In this embodiment, according to the real-time posture information of the construction vehicle, the original coordinate information of the distance scale in the vehicle coordinate system is converted and projected onto the pixel information of the currently acquired image to obtain the target coordinate information, thereby solving the problem that the distance scale cannot change with the posture of the vehicle when the construction vehicle is in a non-horizontal state. In the process of continuous change of the real-time posture of the vehicle, the positioning coordinate information of the distance scale is accurately converted, thereby ensuring that the positioning coordinate information of the distance scale can be continuously adjusted with the real-time posture of the vehicle.

[0048] S103. Display a distance scale on the current captured image according to the target coordinate information.

[0049] The role of the distance scale is to provide guidance to the driver on the distance relationship between the construction area ahead and the current construction vehicle.

[0050] Specifically, since the target coordinate information indicates the pixel information of the distance scale in the current captured image, the corresponding pixel information in the current captured image can be directly determined based on the target coordinate information, and the pixel information is displayed, so that the driver can view the displayed distance scale on the current captured image.

[0051] In this embodiment, according to the target coordinate information, a distance ruler is displayed on the current collected image, and the displayed distance ruler is used to guide the driver to drive the construction vehicle to perform construction work, so that the driver can judge the distance information between the front area and the construction vehicle according to the distance ruler. At the same time, this method solves the problem that when the construction vehicle is in an unbalanced state, the scale displayed on the current collected image will not change with the distortion of the image, and realizes that when the construction vehicle is in an unbalanced state, the current collected image is distorted, and the distance ruler can also be continuously adjusted according to the current posture information of the construction vehicle, so that the distance ruler displayed on the current collected image is more accurate. Further, the display method of the distance ruler of the present invention enables the driver to obtain accurate distance information in any posture of the construction vehicle through the distance ruler that can be continuously adjusted with the posture of the construction vehicle, which greatly improves the operation accuracy of the construction vehicle in complex terrain, reduces operation errors, improves work efficiency, and improves the safety and reliability of remote control.

[0052] The method for displaying a distance scale provided by an embodiment of the present invention obtains the current posture information of the construction vehicle and the corresponding current collected image; determines the target coordinate information of the distance scale according to the current posture information, the internal reference data and the external reference data of the camera device; and displays the distance scale on the current collected image according to the target coordinate information. In the above technical scheme, on the one hand, the current collected image corresponding to the current posture information of the construction vehicle is obtained, which provides an accurate image basis for displaying the distance scale and guiding the driver to judge the distance information according to the distance scale. On the other hand, according to the real-time posture information of the construction vehicle, the original coordinate information of the distance scale in the self-vehicle coordinate system is converted and projected to the pixel information of the current collected image to obtain the target coordinate information, which solves the problem that the distance scale cannot change with the posture of the construction vehicle when the construction vehicle is in a non-horizontal state, and realizes that the positioning coordinate information of the distance scale is accurately converted in the process of the continuous change of the real-time posture of the vehicle, ensuring that the positioning coordinate information of the distance scale can be continuously adjusted with the real-time posture of the vehicle. Finally, according to the target coordinate information, a distance ruler is displayed on the current collected image, and the displayed distance ruler is used to guide the driver to drive the construction vehicle to perform construction work, so that the driver can judge the distance information between the front area and the construction vehicle according to the distance ruler. At the same time, this method solves the problem that when the construction vehicle is in an unbalanced state, the existing static ruler displayed on the current collected image will not change with the distortion of the image, and realizes that when the construction vehicle is in an unbalanced state, the current collected image is distorted, and the distance ruler can also be continuously adjusted according to the current posture information of the construction vehicle, so that the distance ruler displayed on the current collected image is more accurate. Further, the display method of the distance ruler of the present invention enables the driver to obtain accurate distance information in any posture of the construction vehicle through the distance ruler that can be continuously adjusted with the posture of the construction vehicle, which greatly improves the operation accuracy of the construction vehicle in complex terrain, reduces operation errors, improves work efficiency, and improves the safety and reliability of remote control.

[0053] Figure 2 This is a flowchart of another method for displaying a distance scale provided by an embodiment of the present invention. Based on the above embodiment, this embodiment elaborates on the steps of "determining the target coordinate information of the distance scale according to the current posture information, the internal reference data and the external reference data of the camera device" and "displaying the distance scale on the current captured image according to the target coordinate information". Figure 2 As shown, the method includes:

[0054] S201, obtaining current posture information of a construction vehicle and its corresponding current captured image.

[0055] Specifically, during the operation of the construction vehicle, the video screen of the forward camera device usually only shows the screen of the front area. Some forward camera devices will specifically embed a static distance scale such as a reversing line in the video screen to provide the driver with a forward distance guide, thereby guiding or assisting the driver to judge the distance information between the front and the construction vehicle based on the static distance scale. However, since construction vehicles are often on uneven roads such as construction sites and mining, the posture of the construction vehicles often changes, and the video screen of the forward camera device will also produce a certain degree of distortion. At this time, if a static distance scale is embedded in the video screen of the forward camera device, since the static distance scale will not change, the viewing angle of the picture presented by the camera device will be distorted, and the static distance scale will no longer match the actual distance of the actual ground, resulting in a large error in the distance judgment made by the driver based on the static distance scale. For example, when the construction vehicle leans forward, the viewing angle of the camera device will be downward, resulting in the distance displayed by the distance scale in the current acquisition screen being shorter than the actual distance. When the construction vehicle is tilted backward, the camera's viewing angle will be upward, and the distance shown on the distance scale will be longer than the actual distance. This also applies when the vehicle is tilted left or right, and the actual distance indicated by the distance scale on the left or right direction will also be wrong.

[0056] In order to solve this problem, it is necessary to obtain the current posture information of the construction vehicle in real time when the driver drives the construction vehicle to perform a construction task, and obtain the current captured image corresponding to the current posture information.

[0057] S202: Determine original coordinate information of the distance scale according to a preset positioning distance of the distance scale.

[0058] Among them, the positioning distance is the preset distance of the scale lines. For example, the driver can pre-set the positioning distance of the distance scale with 5m as a scale line. The original coordinate information is the coordinate information of the distance scale in the self-vehicle coordinate system. The self-vehicle coordinate system is a coordinate system established with the projection position of the geometric center of the construction vehicle on the horizontal plane as the origin, the forward direction of the construction vehicle as the positive direction of the Y axis, and the right direction of the driver when driving the construction vehicle as the positive direction of the X axis. In this embodiment, the positive direction of the Z axis of the self-vehicle coordinate system can be perpendicular to the track plane of the construction vehicle and upward.

[0059] Specifically, after obtaining the current posture information of the construction vehicle and its corresponding current captured image, the original coordinate information of the distance scale in the self-vehicle coordinate system when the construction vehicle is in the current posture can be determined according to the positioning distance of the preset distance scale.

[0060] Exemplarily, the original coordinate information may be determined according to the following steps:

[0061] (i) Determine the positioning distance corresponding to each positioning point based on the positioning size of the pre-set distance scale.

[0062] The distance scale includes several fixed points, and the original coordinate information includes the original coordinates of each positioning point. The distance between any two adjacent positioning points is equal to the positioning size.

[0063] For example, Figure 3 This is an example diagram of a distance scale provided by an embodiment of the present invention. Figure 3 As shown, Figure 3 The front of the construction vehicle 20 in the figure shows the change of the distance scale when the construction vehicle is tilted forward. When the construction vehicle is on a flat road, the distance scale is a regular rectangle. When the construction vehicle is tilted forward, for example, when the ground where the front wheels of the construction vehicle are located is lower than the ground where the rear wheels are located, the final display in the current captured image may be as follows: Figure 3 The distance scale shown in FIG. Wherein, the distance scale has scale boundary lines 30 on both sides, and the lateral distance between the scale boundary lines 30 on the left and right sides is generally the lateral size of the construction vehicle 20. Figure 3 The distance scale shown in the figure is for when the construction vehicle is on an uneven road surface. Figure 3 The front end of the distance scale is in a distorted state. And, the distance scale also includes a plurality of transverse engraved lines 40, such as Figure 3 The distance between each horizontal scale line 40 is the positioning distance. And the intersection of each scale line 40 and the scale boundary line 30 is the positioning point. Figure 3 P1, P2, P3, P4, P5, P6, P7 and P8 are all positioning points. Alternatively, the positioning point is not limited to the intersection of each engraved line 40 and the scale boundary line 30, but can also be the midpoint of each engraved line 40 or other places that can be used for positioning.

[0064] It is worth noting that 20 shown in the figure is only an example of the position of the construction vehicle. In actual situations, the distance scale is only an example scale that can be embedded in the image collected by the camera device, and is not a scale that actually exists in front of the construction vehicle.

[0065] Specifically, since the positioning size is preset, and the distance between two adjacent positioning points on the same ruler boundary line is the positioning size, the positioning distance corresponding to each positioning point can be directly determined according to the positioning size.

[0066] (ii) Determine the forward direction of the construction vehicle, and use the projection position of the geometric center of the construction vehicle on the horizontal plane as the coordinate origin. According to the forward direction of the construction vehicle and the positioning distance corresponding to each positioning point, determine the original coordinates of each positioning point in the vehicle coordinate system.

[0067] Specifically, after determining the positioning distance corresponding to each positioning point, according to the current forward direction of the construction vehicle, the projection position of the geometric center of the construction vehicle on the horizontal plane is used as the coordinate origin, and the original coordinates of each positioning point in the vehicle coordinate system can be determined according to the positioning distance corresponding to each positioning point. And, the original coordinates of each positioning point are determined as the original coordinate information.

[0068] In this embodiment, based on the positioning distance of the pre-set distance scale, the original coordinates of each positioning point of the distance scale in the vehicle coordinate system under the current posture of the construction vehicle are calculated to obtain the original coordinate information, thereby realizing direct determination of the actual coordinate information of each positioning point of the distance scale in the vehicle coordinate system according to the current posture of the construction vehicle, and providing accurate real-time coordinate information for subsequently embedding the distance scale reflecting the actual posture of the construction vehicle in the current acquired image.

[0069] S203, determining target coordinate information according to current posture information, internal reference data and external reference data of the camera device, and original coordinate information.

[0070] The current posture information includes forward and backward tilt posture information and left and right posture information, and the target coordinate information includes the target coordinates of each positioning point.

[0071] Specifically, the target coordinate information can be determined according to the following steps:

[0072] (i) Determine the posture matrix based on the front-to-back tilt posture information and the left-to-right posture information.

[0073] Among them, the front-to-back tilt posture information can be the front-to-back tilt angle, and the left-to-right tilt posture information can be the left-to-right tilt angle.

[0074] Specifically, according to the inclination sensor installed on the construction vehicle, the front and rear tilt angle φ and the left and right tilt angle ω of the construction vehicle can be detected. Therefore, the posture matrix of the construction vehicle can be determined based on the detected current posture information. The posture matrix is ​​as follows:

[0075]

[0076] Among them, M3 is the posture matrix of the construction vehicle, φ is the front and rear tilt angle, and ω is the left and right tilt angle.

[0077] (ii) De-distorting the original coordinates of each positioning point according to the intrinsic parameter data, the extrinsic parameter data and the pose matrix to obtain the pixel coordinates of the original coordinates of each positioning point.

[0078] Specifically, for a positioning point, the original coordinates of the positioning point are dedistorted to obtain the pixel coordinates of the original coordinates of the positioning point, including:

[0079] (1) Determine the matrix parameters based on the intrinsic parameter data, extrinsic parameter data, pose matrix and the original coordinates of the positioning points.

[0080] The matrix parameters include a first pixel parameter, a second pixel parameter and a camera coordinate parameter.

[0081] Specifically, the matrix parameters can be determined according to the following formula:

[0082]

[0083] Among them, the matrix parameters are [μZ C V Z C Z C ] T , μZ C is the first pixel parameter, vZ C is the second pixel parameter, Z C is the camera coordinate parameter. W Y W Z W 1] T is the original coordinate information corresponding to the target coordinate information. In this embodiment, [X W Y W Z W 1] T The original coordinates of the positioning point.

[0084] (2) Determine the first pixel coordinates according to the first pixel parameter and the camera coordinate parameter, and determine the second pixel coordinates according to the second pixel parameter and the camera coordinate parameter.

[0085] Specifically, the first pixel parameter μZ can be used C Divide by the camera coordinate parameter Z C , get the first pixel coordinate μ. And, use the second pixel parameter vZ C Divide by the camera coordinate parameter Z C , get the second pixel coordinate v.

[0086] (3) The first pixel coordinate and the second pixel coordinate are used as the pixel coordinates of the original coordinates of the positioning point.

[0087] Specifically, after obtaining the first pixel coordinate μ and the second pixel coordinate v, the first pixel coordinate and the second pixel coordinate can be used as the pixel coordinates of the original coordinates of the positioning point, that is, the pixel coordinates of the positioning point of the distance scale in the current captured image captured by the camera device [μv 1] T .

[0088] In this embodiment, the original coordinates of each positioning point of the distance scale in the vehicle coordinate system are converted to the camera coordinate system of the current acquired image by utilizing the intrinsic parameter matrix, extrinsic parameter matrix of the camera equipment and the posture matrix of the construction vehicle, and the pixel coordinates of each positioning point in the current acquired image are obtained. This realizes the projection of each positioning point of the distance scale from the vehicle coordinate system to the pixel point of the current acquired image, thereby providing the original pixel coordinates of each positioning point for subsequently embedding an accurate distance scale in the current acquired image.

[0089] (iii) obtaining lens distortion parameters of the camera device, and determining the target coordinates of each positioning point based on the lens distortion parameters and the pixel coordinates of the original coordinates of each positioning point.

[0090] Specifically, the current captured image captured by the camera device may not only be distorted due to the different postures of the construction vehicle, but may also be distorted due to the lens of the camera device introducing collective distortion before imaging the imaging area of ​​the camera device, causing the final imaging result to deviate from the imaging result in the actual situation. Therefore, after obtaining the pixel coordinates of each positioning point, it is necessary to consider the impact of lens distortion on the current captured image. Therefore, the determination of the target coordinates of each positioning point includes:

[0091] (1) Determine a radial parameter according to the first pixel coordinate and the second pixel coordinate.

[0092] Among them, the radial parameter is the radial distance.

[0093] Specifically, the calculation formula of the radial parameter is as follows:

[0094]

[0095] Where γ is the radial distance.

[0096] (2) Determine the first target pixel coordinates according to the lens distortion parameter, the radial parameter and the first pixel coordinates, and determine the second target pixel coordinates according to the lens distortion parameter, the radial parameter and the second pixel coordinates.

[0097] The lens distortion parameters include radial distortion coefficients and tangential distortion coefficients. In this embodiment, the radial distortion coefficients include k1, k2, and k3, and the tangential distortion coefficients include p1 and p2.

[0098] Specifically, the first target pixel coordinates can be calculated according to the following formula:

[0099] μ d =μ(1+K1γ 2 +K2γ 4 +K3γ 6 )+2p1μv+p2(γ 2 +2μ 2);

[0100] And, the second target pixel coordinates can be calculated according to the following formula:

[0101] v d =v(1+K1γ 2 +K2γ 4 +K3γ 6 )+2p2μv+p1(γ 2 +2v 2 );

[0102] (3) The first target pixel coordinates and the second target pixel coordinates are determined as the target coordinates of the positioning point.

[0103] Specifically, after the first target pixel coordinates and the second target pixel coordinates are calculated, the target coordinates of the positioning point can be determined. The target coordinates of the positioning point can be expressed as: [μ d v d ] T .

[0104] In this embodiment, the original coordinates of the calculated positioning point are dedistorted according to the lens distortion parameters of the camera device. That is, the distortion effect of the lens of the camera device on the final image is taken into account, ensuring that the target coordinates of the positioning point finally calculated will not be affected by the imaging distortion phenomenon of the lens, and accurately calculating the pixel position of the positioning point mapped from the vehicle coordinate system to the current captured image captured by the camera device.

[0105] S204: After connecting the target coordinates of each positioning point in sequence, a distance ruler is displayed on the current acquired image.

[0106] Specifically, after the point position corresponding to the pixel coordinates on the current captured image is determined according to the target coordinates of each positioning point, the target coordinates of each positioning point can be connected in sequence according to the pre-set connection order of each positioning point, and a distance scale can be displayed on the captured image.

[0107] For example, Figure 3 Take the distance scale shown as an example. After determining the target coordinates of each positioning point, such as determining the target coordinates of P1-P8, the connection order of the positioning points can be determined. For example, the connection order is P4 connected to P8, P3 connected to P7, P2 connected to P6, and P1 connected to P5. At this time, based on the target coordinates of each positioning point on the current acquired image, P4 can be connected to P8, P3 connected to P7, P2 connected to P6, and P1 connected to P5. And, according to another connection order, for example, P1, P2, P3 and P4 are connected in sequence, and P5, P6, P7 and P8 are connected in sequence, the complete distance scale can be displayed on the current acquired image.

[0108] It is worth noting that the above method describes a method for embedding a distance scale based on the current captured image. In practice, since the camera device captures a video, the current captured image in the present invention can be used as a frame image of each frame in the video. That is, when the method is actually used, the distance scale finally embedded is displayed in real time in the video image captured by the camera device.

[0109] The display method of the distance scale provided by the embodiment of the present invention, on the one hand, according to the positioning distance of the distance scale set in advance, the original coordinates of each positioning point of the distance scale in the self-vehicle coordinate system under the current posture state of the construction vehicle are calculated to obtain the original coordinate information, so as to directly determine the actual coordinate information of each positioning point of the distance scale in the self-vehicle coordinate system according to the current posture of the construction vehicle, and provide accurate real-time coordinate information for embedding the distance scale reflecting the actual posture of the construction vehicle in the current acquisition image. On the other hand, by using the internal parameter matrix, external parameter matrix and posture matrix of the camera device, the original coordinates of each positioning point of the distance scale in the self-vehicle coordinate system are converted to the camera coordinate system of the current acquisition image, and the pixel coordinates of each positioning point in the current acquisition image are obtained, so as to realize the projection of each positioning point of the distance scale from the self-vehicle coordinate system to the pixel point of the current acquisition image, and provide the original pixel coordinates of each positioning point for embedding the accurate distance scale in the current acquisition image. And, according to the lens distortion parameters of the camera device, the original coordinates of the calculated positioning points are dedistorted. That is, the distortion effect of the camera lens on the final image is taken into account, ensuring that the target coordinates of the final calculated positioning point will not be affected by the imaging distortion of the lens, and accurately calculating the pixel position of the positioning point mapped from the vehicle coordinate system to the current captured image captured by the camera device.

[0110] Figure 4 A schematic structural diagram of a display device for a distance scale provided by an embodiment of the present invention.

[0111] like Figure 4 As shown, the device comprises:

[0112] The acquisition module 401 is used to acquire the current posture information of the construction vehicle and its corresponding current captured image;

[0113] A determination module 402 is used to determine target coordinate information of the distance scale according to current posture information, internal reference data and external reference data of the camera device, wherein the target coordinate information is used to indicate pixel information of the distance scale in the current captured image;

[0114] The display module 403 is used to display a distance scale on the current captured image according to the target coordinate information.

[0115] Optionally, the determination module 402 is specifically configured to:

[0116] According to the preset positioning distance of the distance scale, the original coordinate information of the distance scale is determined, wherein the original coordinate information is the coordinate information of the distance scale in the self-vehicle coordinate system, and the self-vehicle coordinate system is a coordinate system established with the projection position of the geometric center of the construction vehicle on the horizontal plane as the origin, the forward direction of the construction vehicle as the positive direction of the Y axis, and the right direction of the driver when driving the construction vehicle as the positive direction of the X axis; according to the current posture information, the internal and external parameter data of the camera equipment and the original coordinate information, the target coordinate information is determined.

[0117] Optionally, the distance scale includes a plurality of fixed points, and the original coordinate information includes the original coordinates of each positioning point; the original coordinate information of the distance scale is determined according to the preset positioning distance of the distance scale, and the determination module 402 is specifically used to:

[0118] According to the positioning size of the pre-set distance scale, the positioning distance corresponding to each positioning point is determined, wherein the distance between any two adjacent positioning points is equal to the positioning size; the forward direction of the construction vehicle is determined, and the projection position of the geometric center of the construction vehicle on the horizontal plane is used as the coordinate origin, and according to the forward direction of the construction vehicle and the positioning distance corresponding to each positioning point, the original coordinates of each positioning point in the vehicle coordinate system are determined.

[0119] Optionally, the current posture information includes forward and backward tilt posture information and left and right posture information, and the target coordinate information includes the target coordinates of each positioning point; the target coordinate information is determined according to the current posture information, the internal reference data and the external reference data of the camera device, and the original coordinate information, and the determination module 402 is specifically used for:

[0120] Determine the posture matrix based on the forward and backward tilt posture information and the left and right posture information; dedistort the original coordinates of each positioning point based on the intrinsic parameter data, the extrinsic parameter data and the posture matrix to obtain the pixel coordinates of the original coordinates of each positioning point; obtain the lens distortion parameters of the camera device, and determine the target coordinates of each positioning point based on the lens distortion parameters and the pixel coordinates of the original coordinates of each positioning point.

[0121] Optionally, for a positioning point, the original coordinates of the positioning point are subjected to a dedistortion transformation to obtain pixel coordinates of the original coordinates of the positioning point. The determination module 402 is specifically used for:

[0122] According to the intrinsic parameter data, the extrinsic parameter data, the pose matrix and the original coordinates of the positioning point, the matrix parameters are determined, wherein the matrix parameters include the first pixel parameter, the second pixel parameter and the camera coordinate parameter; the first pixel coordinate is determined according to the first pixel parameter and the camera coordinate parameter, and the second pixel coordinate is determined according to the second pixel parameter and the camera coordinate parameter; the first pixel coordinate and the second pixel coordinate are used as the pixel coordinates of the original coordinates of the positioning point.

[0123] Optionally, for a positioning point, the target coordinates of the positioning point are determined according to the lens distortion parameters and the pixel coordinates of the original coordinates of the positioning point. The determination module 402 is specifically used to:

[0124] Determine radial parameters according to the first pixel coordinates and the second pixel coordinates; determine first target pixel coordinates according to the lens distortion parameter, the radial parameter and the first pixel coordinates, and determine second target pixel coordinates according to the lens distortion parameter, the radial parameter and the second pixel coordinates; determine the first target pixel coordinates and the second target pixel coordinates as target coordinates of the positioning point.

[0125] Optionally, the display module 403 is specifically used for:

[0126] After connecting the target coordinates of each positioning point in sequence, the distance ruler is displayed on the current acquisition image.

[0127] The display device for a distance scale provided in the embodiment of the present invention can execute the display method for a distance scale provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0128] Figure 5 Schematic diagram of the structure of the construction vehicle 10 of an embodiment of the present invention. In this embodiment, the construction vehicle represents various electronic devices, or a construction vehicle loaded with electronic devices. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0129] like Figure 5As shown, the construction vehicle 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the construction vehicle 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0130] A number of components in the construction vehicle 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the construction vehicle 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0131] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a method for displaying a distance scale.

[0132] In some embodiments, the method for displaying a distance scale may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the construction vehicle 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for displaying a distance scale described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the method for displaying a distance scale in any other appropriate manner (e.g., by means of firmware).

[0133] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0134] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0135] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0136] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0137] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0138] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0139] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0140] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for displaying a distance scale, characterized in that: Applied to a construction vehicle, wherein a camera device is arranged on the front hood of the construction vehicle; the method comprises: Obtaining current posture information of the construction vehicle and its corresponding current captured image; Determine target coordinate information of the distance scale according to the current posture information, the internal reference data and the external reference data of the camera device, wherein the target coordinate information is used to indicate pixel information of the distance scale in the current acquired image; The distance scale is displayed on the current acquired image according to the target coordinate information.

2. The method for displaying a distance scale according to claim 1, characterized in that: Determining target coordinate information of the distance scale according to the current posture information, the internal reference data and the external reference data of the camera device includes: Determine the original coordinate information of the distance scale according to the preset positioning distance of the distance scale, wherein the original coordinate information is the coordinate information of the distance scale in the self-vehicle coordinate system, and the self-vehicle coordinate system is a coordinate system established with the projection position of the geometric center of the construction vehicle on the horizontal plane as the origin, the forward direction of the construction vehicle as the positive direction of the Y axis, and the right direction of the driver when driving the construction vehicle as the positive direction of the X axis; The target coordinate information is determined according to the current posture information, the internal reference data and the external reference data of the camera device, and the original coordinate information.

3. The method for displaying a distance scale according to claim 2, characterized in that: The distance scale includes a plurality of fixed points, and the original coordinate information includes the original coordinates of each of the fixed points; and determining the original coordinate information of the distance scale according to the preset fixed distance of the distance scale includes: Determine the positioning distance corresponding to each positioning point according to the preset positioning size of the distance scale, wherein the distance between any two adjacent positioning points is equal to the positioning size; The forward direction of the construction vehicle is determined, and the projection position of the geometric center of the construction vehicle on the horizontal plane is used as the coordinate origin. According to the forward direction of the construction vehicle and the positioning distance corresponding to each positioning point, the original coordinates of each positioning point in the vehicle coordinate system are determined.

4. The method for displaying a distance scale according to claim 3, characterized in that: The current posture information includes forward and backward tilt posture information and left and right posture information, and the target coordinate information includes the target coordinates of each of the positioning points; and determining the target coordinate information according to the current posture information, the internal reference data and the external reference data of the camera device, and the original coordinate information includes: Determine a posture matrix according to the front-to-back tilt posture information and the left-to-right posture information; According to the intrinsic parameter data, the extrinsic parameter data and the pose matrix, the original coordinates of each of the positioning points are dedistorted to obtain the pixel coordinates of the original coordinates of each of the positioning points; The lens distortion parameters of the camera device are obtained, and the target coordinates of each of the positioning points are determined according to the lens distortion parameters and the pixel coordinates of the original coordinates of each of the positioning points.

5. The method for displaying a distance scale according to claim 4, characterized in that: For one of the positioning points, performing a dedistortion transformation on the original coordinates of the positioning point to obtain pixel coordinates of the original coordinates of the positioning point includes: Determine matrix parameters according to the intrinsic parameter data, the extrinsic parameter data, the pose matrix and the original coordinates of the positioning point, wherein the matrix parameters include a first pixel parameter, a second pixel parameter and a camera coordinate parameter; Determine a first pixel coordinate according to the first pixel parameter and the camera coordinate parameter, and determine a second pixel coordinate according to the second pixel parameter and the camera coordinate parameter; The first pixel coordinate and the second pixel coordinate are used as the pixel coordinates of the original coordinates of the positioning point.

6. The method for displaying a distance scale according to claim 5, characterized in that: For one of the positioning points, determining the target coordinates of the positioning point according to the lens distortion parameter and the pixel coordinates of the original coordinates of the positioning point includes: determining a radial parameter according to the first pixel coordinate and the second pixel coordinate; determining a first target pixel coordinate according to the lens distortion parameter, the radial parameter and the first pixel coordinate, and determining a second target pixel coordinate according to the lens distortion parameter, the radial parameter and the second pixel coordinate; The first target pixel coordinates and the second target pixel coordinates are determined as target coordinates of the positioning point.

7. The method for displaying a distance scale according to claim 6, characterized in that: The step of displaying the distance scale on the current acquired image according to the target coordinate information includes: After the target coordinates of each of the positioning points are connected in sequence, the distance scale is displayed on the current acquired image.

8. A distance scale display device, characterized in that: Applied to a construction vehicle, a camera device is provided on the front hood of the construction vehicle; the device comprises: An acquisition module, used to acquire the current posture information of the construction vehicle and its corresponding current captured image; A determination module, used to determine target coordinate information of the distance scale according to the current posture information, the internal reference data and the external reference data of the camera device, wherein the target coordinate information is used to indicate pixel information of the distance scale in the current collected image; A display module is used to display the distance scale on the current acquired image according to the target coordinate information.

9. A construction vehicle, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method for displaying a distance scale according to any one of claims 1 to 7.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for displaying a distance scale according to any one of claims 1 to 7 is implemented.

Images