Tracked vehicle and auxiliary driving method, device and system thereof

Abstract

The invention relates to the field of auxiliary driving, in particular to a tracked vehicle and an auxiliary driving method, device and system thereof, and the method comprises the steps: obtaining a first image of a road shot by a first camera and a second image of the road shot by a second camera; acquiring depth information of the first image according to the second image and the first image; processing the first image to obtain a first group of boundary points and a second group of boundary points; generating a plurality of guide points according to the first group of boundary points and the second group of boundary points; according to the depth information of the first image and the pixel coordinate of each guide point, calculating the world coordinate of each guide point; generating a guide line based on the world coordinate of each guide point; according to the first speed of the first track and the second speed of the second track of the tracked vehicle, generating a predicted path of the tracked vehicle in a period of time in the future; and displaying the guide line and the predicted path so that an operator can control the crawler-type vehicle to run according to the predicted path and the guide line.

Description

Technical Field

[0001] The present disclosure relates to the field of assisted driving, and in particular to a tracked vehicle and an assisted driving method, device and system thereof. Background Art

[0002] There are two main driving control methods for existing tracked vehicles, one is manual control and the other is autonomous control.

[0003] Manual control relies on the operator's visual perception of the relative position of the vehicle and the road, and is often unable to perform more precise operations. Autonomous control solutions use various sensors to replace human perception to achieve measurement and positioning functions, but they require modeling or learning of the vehicle's driving environment. The preliminary preparation work is cumbersome, and the cost is high and difficult to popularize. Summary of the invention

[0004] According to a first aspect of the present disclosure, an assisted driving method for a tracked vehicle is proposed, comprising: acquiring a first image of a road on which the tracked vehicle is traveling, taken by a first camera, and a second image of the road taken by a second camera; acquiring depth information of the first image based on the second image and the first image; processing the first image to obtain a first group of boundary points located at a first boundary of the road and a second group of boundary points located at a second boundary of the road, the first group of boundary points including a plurality of first boundary points, and the second group of boundary points including a plurality of second boundary points; generating a plurality of guide points based on the first group of boundary points and the second group of boundary points, each guide point being located between the corresponding first boundary point and the second boundary point, and different guide points corresponding to different first boundary points. a boundary point and a second boundary point, and the connecting lines of the first boundary point and the second boundary point corresponding to different guide points are all along the first direction, and the pixel coordinates of each guide point are calculated based on the same calculation method for the pixel coordinates of the first boundary point and the second boundary point corresponding to each guide point; the world coordinates of each guide point are calculated according to the depth information of the first image and the pixel coordinates of each guide point; a guide line is generated based on the world coordinates of each guide point; a predicted path of the tracked vehicle in the future is generated according to the first speed of the first track and the second speed of the second track of the tracked vehicle; and the guide line and the predicted path are displayed so that an operator can control the driving of the tracked vehicle according to the predicted path and the guide line.

[0005] According to some embodiments of the present disclosure, the processing includes: performing semantic segmentation on the first image to obtain a first group of initial boundary points located at a first boundary of the road and a second group of initial boundary points located at a second boundary of the road, the first group of initial boundary points including multiple first initial boundary points, and the second group of initial boundary points including multiple second initial boundary points; when the number of the multiple first initial boundary points is less than a first threshold and / or the number of the multiple second initial boundary points is less than a second threshold, adjusting at least one of the position and elevation angle of the first camera so that the number of the multiple first initial boundary points obtained by performing semantic segmentation on the first image captured by the adjusted first camera is greater than or equal to the first threshold, and the number of the multiple second initial boundary points is greater than or equal to the second threshold; using the first group of initial boundary points as the first group of boundary points, and using the second group of initial boundary points as the second group of boundary points.

[0006] According to some embodiments of the present disclosure, the pixel coordinates (u, v) of each guiding point and the pixel coordinates (u1, v1) of the first boundary point and the pixel coordinates (u2, v2) of the second boundary point corresponding to the guiding point satisfy: (u, v) = α(u1, v1) + (1-α)(u2, v2), where α and 1-α are weighted weights of the pixel coordinates of the first boundary point and the second boundary point, respectively, and 0<α<1.

[0007] According to some embodiments of the present disclosure, α=0.5.

[0008] According to some embodiments of the present disclosure, the first direction is a horizontal direction.

[0009] According to some embodiments of the present disclosure, the absolute value of the difference in the horizontal coordinates of the pixel coordinates of the first boundary points corresponding to two adjacent guide points in the vertical direction is less than or equal to a third threshold, and the absolute value of the difference in the horizontal coordinates of the pixel coordinates of the second boundary points corresponding to two adjacent guide points in the vertical direction is less than or equal to a fourth threshold.

[0010] According to some embodiments of the present disclosure, calculating the world coordinates of each guide point based on the depth information of the first image and the pixel coordinates of each guide point includes: acquiring the depth value of each guide point from the depth information of the first image; and obtaining the world coordinates of the guide point based on the intrinsic parameter matrix, extrinsic parameter matrix, the depth value of each guide point and the pixel coordinates of each guide point of the first camera.

[0011] According to some embodiments of the present disclosure, the guide line and the predicted path are displayed in a vehicle body coordinate system.

[0012] According to some embodiments of the present disclosure, the guide line and the predicted path are displayed on a remote controller for controlling the tracked vehicle.

[0013] According to a second aspect of the present disclosure, an auxiliary driving device for a tracked vehicle is proposed, comprising: a first acquisition module, configured to acquire a first image of a road on which the tracked vehicle is traveling, taken by a first camera, and a second image of the road taken by a second camera; a second acquisition module, configured to acquire depth information of the first image based on the second image and the first image; a processing module, configured to process the first image to obtain a first group of boundary points located at a first boundary of the road and a second group of boundary points located at a second boundary of the road, the first group of boundary points including a plurality of first boundary points, and the second group of boundary points including a plurality of second boundary points; a first generation module, configured to generate a plurality of guide points based on the first group of boundary points and the second group of boundary points, each guide point being located between the corresponding first boundary point and the second boundary point, and different guide points corresponding to different a first boundary point and a second boundary point, and the connecting lines of the first boundary points and the second boundary points corresponding to different guide points are all along the first direction, and the pixel coordinates of each guide point are calculated based on the same calculation method for the pixel coordinates of the first boundary point and the second boundary point corresponding to each guide point; a calculation module is configured to calculate the world coordinates of each guide point according to the depth information of the first image and the pixel coordinates of each guide point; a second generation module is configured to generate a guide line based on the world coordinates of each guide point; a third generation module is configured to generate a predicted path of the tracked vehicle in the future according to the first speed of the first track and the second speed of the second track of the tracked vehicle; and a display module is configured to display the guide line and the predicted path so that an operator can control the driving of the tracked vehicle according to the predicted path and the guide line.

[0014] According to a third aspect of the present disclosure, an auxiliary driving device for a tracked vehicle is proposed, comprising: a memory; and a processor coupled to the memory, configured to execute the auxiliary driving method of any one of the above-mentioned embodiments based on instructions stored in the memory.

[0015] According to a fourth aspect of the present disclosure, an auxiliary driving system for a tracked vehicle is proposed, comprising: an auxiliary driving device of any one of the above-mentioned embodiments; the first camera; the second camera; and a controller, configured to control the driving of the tracked vehicle according to a command signal sent by a remote control.

[0016] According to a fifth aspect of the present disclosure, a tracked vehicle is proposed, comprising an auxiliary driving system according to any one of the above embodiments.

[0017] According to a sixth aspect of the present disclosure, a computer-readable storage medium is proposed, comprising computer program instructions, wherein the computer program instructions, when executed by a processor, implement the assisted driving method of any one of the above-mentioned embodiments.

[0018] According to a seventh aspect of the present disclosure, a computer program product is proposed, including a computer program, wherein when the computer program is executed by a processor, the assisted driving method of any one of the above embodiments is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 is a flow chart of an assisted driving method for a tracked vehicle according to some embodiments of the present disclosure.

[0021] Figure 2 is a flowchart of performing processing on a first image according to some embodiments of the present disclosure.

[0022] Figure 3 It is a schematic diagram of the structure of an auxiliary driving device of a tracked vehicle according to some embodiments of the present disclosure.

[0023] Figure 4 It is a schematic structural diagram of an auxiliary driving device for a tracked vehicle according to other embodiments of the present disclosure.

[0024] Figure 5 is a schematic structural diagram of a tracked vehicle according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0026] Unless specifically stated otherwise, the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0027] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0028] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0029] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0030] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0031] The inventors noticed that in certain scenarios, such as when a tracked vehicle is traveling on a road without guide lines, it is difficult for operators to determine the appropriate speed and direction due to limited vision, insufficient operating proficiency, and lack of reference signs. Therefore, it is difficult to use tracked vehicles to meet their own needs, such as firefighting operations, cargo handling, etc.

[0032] In view of this, the present disclosure proposes the following technical solutions to help operators better control the driving of tracked vehicles and improve traffic efficiency and safety.

[0033] Figure 1 is a flow chart of an assisted driving method for a tracked vehicle according to some embodiments of the present disclosure.

[0034] In step S10, a first image of a road on which the tracked vehicle is traveling (hereinafter referred to as the road) taken by a first camera and a second image of the road taken by a second camera are acquired.

[0035] It should be understood that the first camera and the second camera are installed at different positions of the tracked vehicle and move with the tracked vehicle.

[0036] In step S20, depth information of the first image is acquired according to the second image and the first image.

[0037] The depth information of the first image includes a depth value of each pixel in the first image.

[0038] In some embodiments, the disparity map of the first image and the second image can be calculated by correcting the first image and the second image and using a semi-global matching algorithm, and the camera calibration parameters (such as an intrinsic parameter matrix) of the two cameras obtained by calibrating the first camera and the second camera are combined to generate a depth image of the first image, and the depth information of the first image is obtained from the depth image.

[0039] In step S30, the first image is processed to obtain a first group of boundary points located at a first boundary of the road and a second group of boundary points located at a second boundary of the road.

[0040] Here, the first group of boundary points includes a plurality of first boundary points, and the second group of boundary points includes a plurality of second boundary points. For example, the first boundary may be a left boundary of a road, and the second boundary may be a right boundary of the road, or vice versa.

[0041] Step S30 may be executed synchronously with step S20, or may be executed after step S20, or may be executed before step S20.

[0042] In step S40, a plurality of guide points are generated according to the first group of boundary points and the second group of boundary points.

[0043] Each guide point has a corresponding first boundary point and a second boundary point. Each guide point is located between the corresponding first boundary point and the second boundary point, different guide points correspond to different first boundary points and second boundary points, and the lines connecting the first boundary points and the second boundary points corresponding to different guide points are all along the same first direction.

[0044] In other words, a first boundary point and a second boundary point of a line along a first direction can be first found, and a point can be found in a line connecting the first boundary point and the second boundary point as a guide point in a certain way. Subsequently, the above process is repeated to generate multiple guide points.

[0045] In some embodiments, a guide point may be generated by weighted summing the pixel coordinates of the first boundary point and the pixel coordinates of the second boundary point. In this case, the pixel coordinates (u, v) of each guide point and the pixel coordinates (u1, v1) of the first boundary point corresponding to the guide point and the pixel coordinates (u2, v2) of the second boundary point corresponding to the guide point satisfy:

[0046] (u, v) = α (u1, v1) + (1-α) (u2, v2)

[0047] Where 0<α<1, α and (1-α) are weighted weights of the pixel coordinates of the first boundary point and the second boundary point, respectively. The closer α is to 0, the closer the guide point is to the corresponding second boundary point; the closer α is to 1, the closer the guide point is to the corresponding first boundary point.

[0048] As some implementations, α=0.5. At this time, each guide point is the midpoint between the corresponding first boundary point and the second boundary point. In this way, the guide line is also located in the center of the road, which is more convenient for operators to refer to and observe.

[0049] In some embodiments, the first direction is a horizontal direction in the image plane of the first image. In this way, it is not only helpful to find a pair of first boundary points and second boundary points in the first image composed of multiple pixel points, but also helpful to quickly calculate the pixel coordinates (u, v) of the guide point. For example, it is only necessary to calculate the horizontal coordinate u of the guide point, and the vertical coordinate v of the guide point can be the vertical coordinate of the first boundary point or the second boundary point.

[0050] In step S50, the world coordinates (X, Y, Z) of each guide point are calculated based on the depth information of the first image and the pixel coordinates (u, v) of each guide point.

[0051] In some embodiments, the world coordinates (X, Y, Z) of each guidance point may be obtained in the following manner.

[0052] First, the depth value Z of each guide point is obtained from the depth information of the first image. c .

[0053] Secondly, according to the intrinsic parameter matrix I of the first camera, the extrinsic parameter matrix V, the depth value Z of each guide point c And the pixel coordinates (u, v) of each guide point, get the world coordinates (X, Y, Z) of the guide point.

[0054] For example, the world coordinates (X, Y, Z) of the guide point can be obtained by the following formula:

[0055] Z c [u,v,1] T =IV[X,Y,Z,1] T

[0056] Among them, [u,v,1] T The pixel coordinates (u, v) of the guide point are converted into a matrix representation of homogeneous coordinates, [X, Y, Z, 1] T The matrix representation of the world coordinates (X, Y, Z) of the guide point converted into homogeneous coordinates.

[0057] The intrinsic parameter matrix I and the extrinsic parameter matrix V of the first camera can be obtained by calibrating the first camera.

[0058] For example, the intrinsic parameter matrix I of the first camera may have the following form:

[0059]

[0060] In the intrinsic parameter matrix I, f is the focal length of the first camera, s x and y are the sizes of a single pixel of the first camera in the horizontal and vertical directions, respectively; u0 and v0 are the abscissa and ordinate of the pixel coordinates of the principal point of the first camera, respectively.

[0061] For example, the extrinsic parameter matrix V of the first camera can have the following form:

[0062]

[0063] In the external parameter matrix V, R is a 3×3 matrix, which represents the rotation relationship between the camera coordinate system of the first camera and the world coordinate system; T is a 3×1 matrix, which represents the translation relationship between the position of the first camera and the origin of the world coordinate system.

[0064] In step S60, a guide line is generated based on the world coordinates of each guide point.

[0065] In some embodiments, multiple guide points may be connected in sequence to generate a guide line.

[0066] In other embodiments, multiple guide points may be fitted (eg, polynomial fitting) to generate a guide line, which is smoother and more convenient for operators to observe.

[0067] In some embodiments, the plurality of guide points may be projected onto the ground (ie, Y=0) first, and then the plurality of guide points may be connected or fitted in sequence.

[0068] In step S70, a predicted path of the tracked vehicle within a future period of time T is generated according to a first speed v1 of the first track of the tracked vehicle and a second speed v2 of the second track of the tracked vehicle.

[0069] In some embodiments, the linear velocity v and angular velocity w of the tracked vehicle may be calculated based on the kinematic model of the tracked vehicle by the following equations:

[0070]

[0071] Wherein, B is the distance between the first track and the second track of the tracked vehicle.

[0072] Subsequently, the current position (x, z) of the tracked vehicle is obtained, and the motion of the tracked vehicle is discretized to calculate the position of the tracked vehicle at multiple moments in the future period T, thereby generating a predicted path of the tracked vehicle in the future period T (for example, by connecting the positions at multiple moments or fitting such as polynomial fitting).

[0073] For example, the position of the tracked vehicle is calculated once every time interval dt by the following formula until the number of calculations*time interval dt exceeds T:

[0074]

[0075] Wherein, t and t+dt are respectively the previous moment and the next moment of each calculation, and θ is the heading angle of the tracked vehicle. The time interval dt can be set according to requirements, for example, 0.01s, 0.1s or 0.5s.

[0076] In step S80, the guide line and the predicted path are displayed so that the operator can control the tracked vehicle to travel according to the predicted path and the guide line.

[0077] Through the above steps S10-S80, during the driving process of the crawler vehicle, a guide line indicating the extension direction of the road where the crawler vehicle is located can be displayed in real time, and the predicted path of the crawler vehicle can be displayed at the same time. In this way, even on a road that is not convenient for driving without a guide line, the operator can better control the driving of the crawler vehicle and improve the driving efficiency and safety.

[0078] For example, the operator can control the tracked vehicle to travel according to the positional relationship between the predicted path and the guide line. When the predicted path and the guide line have a high degree of overlap, the operator can maintain the driving state of the tracked vehicle unchanged; when the predicted path deviates to one side relative to the guide line, the operator can adjust the driving direction of the tracked vehicle so that the tracked vehicle travels to the other side.

[0079] In some embodiments, the guide line and the predicted path of the tracked vehicle can be displayed in the world coordinate system. Since the starting point of the predicted path is the position of the tracked vehicle, the operator can simultaneously observe the movement of the tracked vehicle.

[0080] In other embodiments, the guide line and the predicted path of the crawler vehicle can be displayed in the vehicle body coordinate system. In this case, the world coordinates (X, Y, Z) of the guide point obtained in step S50 need to be converted to the vehicle body coordinate system. In the above formula for calculating the position of the crawler vehicle, x=0, z=0, θ=π / 2 in the first calculation. At this time, it is more convenient for the operator to observe and judge the positional relationship between the predicted path and the guide line, and it is convenient for the operator to adjust the form of the crawler vehicle.

[0081] In some embodiments, the guide line and the predicted path are displayed on a remote controller for controlling the tracked vehicle. In this way, the operator can observe and determine the positional relationship between the predicted path and the guide line through the remote controller and control the tracked vehicle through the remote controller at the same time, thereby increasing the convenience of operation.

[0082] Figure 2 is a flowchart of performing processing on a first image according to some embodiments of the present disclosure.

[0083] like Figure 2 As shown, step S30 may include steps S301, S302 and S303.

[0084] In step S301, semantic segmentation is performed on the first image to obtain a first group of initial boundary points located at a first boundary of the road and a second group of initial boundary points located at a second boundary of the road. Similarly, the first group of initial boundary points includes a plurality of first initial boundary points, and the second group of initial boundary points includes a plurality of second initial boundary points.

[0085] In step S302, when the number of first initial boundary points in the first group of initial boundary points is less than a first threshold and / or the number of second initial boundary points in the second group of initial boundary points is less than a second threshold, at least one of the position and the elevation angle of the first camera is adjusted so that the number of first initial boundary points obtained by performing semantic segmentation on the first image taken by the adjusted first camera is greater than or equal to the first threshold, and the number of second initial boundary points is greater than or equal to the second threshold.

[0086] In step S303, the first group of initial boundary points is used as the first group of boundary points, and the second group of initial boundary points is used as the second group of boundary points.

[0087] Through steps S301-S303, the first camera can better photograph the road and obtain a sufficient number of first boundary points and second boundary points, which is conducive to obtaining a sufficient number of guide points, so that the subsequently generated guide lines can more accurately reflect the extension direction of the road.

[0088] In some embodiments, at least one of the first threshold and the second threshold can be set to the number of pixels of the first image in the vertical direction. In this case, if the number of the first boundary point or the second boundary point is less than the first threshold, it means that the first boundary or the second boundary of the road does not extend from the bottom of the first image to the top of the first image, for example, from the side to the top or the bottom. By adjusting the first camera, the first camera can capture more parts of the first boundary or the second boundary of the road.

[0089] In other embodiments, at least one of the first threshold and the second threshold may be set to a number less than the number of pixels of the first image in the vertical direction. For example, the first threshold or the second threshold may be 0.9 times or more of the number of pixels of the first image in the vertical direction, for example, 0.95 times, 0.9 times. In this way, even if a portion of the first boundary points or the second boundary points are screened out due to other reasons (see below), it can still be considered that the number of the first boundary points or the second boundary points is sufficient.

[0090] In some embodiments, the first threshold may be equal to the second threshold.

[0091] For example, the first threshold and the second threshold are both set to the number of pixels of the first image in the vertical direction. This is beneficial for the first camera to simultaneously capture more parts of the two boundaries of the road.

[0092] For another example, the first threshold and the second threshold are both set to be smaller than the number of pixels of the first image in the vertical direction (eg, 0.95 times, 0.9 times the number of pixels).

[0093] In some embodiments, in the first image, the pixel distance in the first direction between the first boundary points corresponding to two adjacent guide points in the second direction perpendicular to the first direction is less than or equal to a third threshold, and the pixel distance in the first direction between the second boundary points corresponding to two adjacent guide points in the second direction is less than or equal to a fourth threshold.

[0094] For example, when the first direction is the horizontal direction, in the first image, the absolute value of the difference between the horizontal coordinates u1 of the first boundary points corresponding to two adjacent guide points in the vertical direction is less than or equal to the third threshold, and the absolute value of the difference between the horizontal coordinates u2 of the second boundary points corresponding to each other is less than or equal to the fourth threshold.

[0095] In the above embodiment, the first boundary points and the second boundary points corresponding to adjacent guide points are relatively close, that is, the distribution continuity of the first group of boundary points and the second group of boundary points used to generate the plurality of guide points is relatively good. For example, the first group of boundary points and the second group of boundary points can be screened to retain only those first boundary points and second boundary points that are relatively continuously distributed.

[0096] The following introduces a method for screening the first set of boundary points and the second set of boundary points, taking the first direction as the horizontal direction:

[0097] (1) Construct an empty queue data structure, where each element of the queue includes the pixel coordinates of a first boundary point (u1, v1) and the pixel coordinates of a second boundary point (u2, v2).

[0098] (2) Insert the horizontal coordinate u1 of the first boundary point and the horizontal coordinate u2 of the second boundary point in the first row of pixels of the first image to the end of the queue. The first row of pixels may be the bottom row of pixels or the top row of pixels of the first image.

[0099] (3) Perform the following operations on each row of pixels from the second row of pixels to the last row of pixels of the first image in order from bottom to top or from top to bottom.

[0100] It is determined whether the first difference between the horizontal coordinate u1 of the first boundary point in the row of pixels and the first boundary point in the last element of the queue is less than or equal to the third threshold, and whether the second difference between the horizontal coordinate u2 of the second boundary point in the row of pixels and the second boundary point in the last element of the queue is less than or equal to the fourth threshold. Here, the first difference and the second difference both refer to the absolute values ​​of the differences.

[0101] When the first difference is less than or equal to the third threshold and the second difference is less than or equal to the fourth threshold, the horizontal coordinate u1 of the first boundary point and the horizontal coordinate u2 of the second boundary point in the row of pixels are inserted into the end of the queue as elements; otherwise, the first boundary point and the second boundary point of the row are discarded.

[0102] (4) The first boundary point and the second boundary point of all elements in the queue are respectively used as the first boundary point and the second boundary point after screening.

[0103] The above method uses a queue to ensure that even if some first boundary points and second boundary points are screened out, the first boundary points and second boundary points that pass the screening each time are always closer to the first boundary points and second boundary points that passed the screening last time, that is, the distribution continuity of the first group of boundary points and the second group of boundary points is guaranteed.

[0104] Figure 3 is a schematic structural diagram of an auxiliary driving device 100 for a tracked vehicle according to some embodiments of the present disclosure.

[0105] like Figure 3 As shown, the auxiliary driving device 100 of a tracked vehicle includes: a first acquisition module 101, a second acquisition module 102, a processing module 103, a first generation module 104, a calculation module 105, a second generation module 106, a third generation module 107 and a display module 108.

[0106] The first acquisition module 101 is configured to acquire a first image of the road on which the tracked vehicle travels taken by a first camera and a second image of the road taken by a second camera.

[0107] The second acquisition module 102 is configured to acquire depth information of the first image according to the second image and the first image.

[0108] The processing module 103 is configured to process the first image to obtain a first group of boundary points located at a first boundary of the road and a second group of boundary points located at a second boundary of the road, wherein the first group of boundary points includes multiple first boundary points and the second group of boundary points includes multiple second boundary points.

[0109] The first generating module 104 is configured to generate a plurality of guide points according to the first group of boundary points and the second group of boundary points, each guide point is located between the corresponding first boundary point and the second boundary point, different guide points correspond to different first boundary points and the second boundary points, and the lines connecting the first boundary points and the second boundary points corresponding to different guide points are all along the first direction, and the pixel coordinates of each guide point are calculated based on the same calculation method for the pixel coordinates of the first boundary point and the second boundary point corresponding to each of them.

[0110] The calculation module 105 is configured to calculate the world coordinates of each guiding point according to the depth information of the first image and the pixel coordinates (u, v) of each guiding point.

[0111] The second generating module 106 is configured to generate a guide line based on the world coordinates of each guide point.

[0112] The third generating module 107 is configured to generate a predicted path of the tracked vehicle within a future period of time according to the first speed of the first track and the second speed of the second track of the tracked vehicle.

[0113] The display module 108 is configured to display the guide line and the predicted path so that the operator can control the tracked vehicle to travel according to the predicted path and the guide line.

[0114] In some embodiments, the auxiliary driving device 100 further includes other modules to execute the auxiliary driving method of any one of the above embodiments.

[0115] Figure 4 1 is a schematic structural diagram of an auxiliary driving device 100 ′ for a tracked vehicle according to other embodiments of the present disclosure.

[0116] The auxiliary driving device 100 ′ of a tracked vehicle includes a memory 11 and a processor 12 coupled to the memory 11 .

[0117] The processor 12 is configured to execute the auxiliary driving method of a tracked vehicle of any one of the above-mentioned embodiments based on the instructions stored in the memory 11 .

[0118] The memory 11 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.

[0119] In some embodiments, the auxiliary driving device 100' of the crawler vehicle may further include an input / output interface 13, a network interface 14, a storage interface 15, etc. These interfaces 13, 14, 15, and the memory 11 and the processor 12 may be connected, for example, via a bus 16. The input / output interface 13 provides a connection interface for input / output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 14 provides a connection interface for various networked devices. The storage interface 15 provides a connection interface for external storage devices such as an SD card and a USB flash drive.

[0120] The present disclosure also proposes an auxiliary driving system for a tracked vehicle, the system comprising: an auxiliary driving device for a tracked vehicle according to any one of the above embodiments, a first camera, a second camera, and a controller. The controller is configured to control the tracked vehicle to travel according to a command signal sent by a remote controller.

[0121] The remote controller may, for example, have a first degree of opening of a first joystick (eg, longitudinal degree of opening) associated with a speed v of the tracked vehicle and a second degree of opening of a second joystick (eg, lateral degree of opening) associated with an angular speed w of the tracked vehicle.

[0122] The present disclosure also proposes a tracked vehicle 200, which includes the auxiliary driving system of the tracked vehicle.

[0123] Figure 5 2 is a schematic structural diagram of a tracked vehicle 200 according to some embodiments of the present disclosure, wherein the left side is a front view of the tracked vehicle 200 and the right side is a side view of the tracked vehicle 200 .

[0124] like Figure 5 As shown, the tracked vehicle 200 includes a first camera 201 and a second camera 202, a first track 251 and a second track 252, a controller (not shown), and an auxiliary driving device of the tracked vehicle of any one of the above-mentioned embodiments, such as the auxiliary driving device 100 or the auxiliary driving device 100' (not shown).

[0125] The controller is configured to control the crawler vehicle 200 to travel according to the command signal sent by the remote controller. For example, the controller may be a proportional integral derivative (PID) controller.

[0126] The first camera 201 and the second camera 202 are installed on the tracked vehicle 200. For example, the first camera 201 and the second camera 202 are both facing the forward direction of the tracked vehicle 200.

[0127] In some embodiments, Figure 5As shown, the first camera 201 and the second camera 202 are installed at the front end of the tracked vehicle 200. As some implementations, the first camera 201 and the second camera 202 are installed at both sides of the front end of the tracked vehicle 200, for example, the first camera 201 is installed at the left side of the front end and the second camera 202 is installed at the right side of the front end, or the first camera 201 is installed at the right side of the front end and the second camera 202 is installed at the left side of the front end. In this way, the first image and the second image can better reflect the road conditions.

[0128] In some embodiments, Figure 5 As shown, the tracked vehicle 200 further includes a bracket 203 for supporting the first camera 201 and the second camera 202. The bracket 203, the first camera 201 and the second camera 202 together constitute a binocular system 210.

[0129] In some embodiments, Figure 5 As shown, the tracked vehicle 200 further includes a mobile chassis 220 , a wireless module 230 , and a rotation speed sensor 240 .

[0130] The auxiliary driving device 100 or the auxiliary driving device 100 ′ of the tracked vehicle can be connected to the binocular system 210 , the mobile chassis 220 , the wireless module 230 and the rotation speed sensor 240 respectively through a USB cable or a CAN bus.

[0131] The wireless module 230 is configured to receive a command signal from the remote controller and send the command signal to the controller of the tracked vehicle.

[0132] The rotation speed sensor 240 is configured to monitor the first speed v1 of the first track 251 and the second speed v2 of the second track 252 in real time, and send the first speed v1 and the second speed v2 as feedback signals to the controller so that the controller adjusts the first speed v1 and the second speed v2 to desired values.

[0133] The embodiment of the present disclosure further provides a computer-readable storage medium, including computer program instructions, which, when executed by a processor, implement the auxiliary driving method of a tracked vehicle in any of the above embodiments.

[0134] The embodiments of the present disclosure further provide a computer program product, including a computer program, which, when executed by a processor, implements the auxiliary driving method of a tracked vehicle of any one of the above embodiments.

[0135] So far, various embodiments of the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed here.

[0136] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present disclosure. It should be understood that the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions specified in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0140] Although some specific embodiments of the present disclosure have been described in detail by way of examples, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified or some technical features may be replaced by equivalents without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for assisting driving of a tracked vehicle, comprising: Acquire a first image of a road on which the tracked vehicle is traveling, taken by a first camera, and a second image of the road, taken by a second camera; Acquire depth information of the first image according to the second image and the first image; Processing the first image to obtain a first group of boundary points located at a first boundary of the road and a second group of boundary points located at a second boundary of the road, wherein the first group of boundary points includes a plurality of first boundary points and the second group of boundary points includes a plurality of second boundary points; generating a plurality of guide points according to the first group of boundary points and the second group of boundary points, each guide point being located between a corresponding first boundary point and a corresponding second boundary point, different guide points corresponding to different first boundary points and second boundary points, and lines connecting the first boundary points and the second boundary points corresponding to different guide points are all along a first direction, and pixel coordinates of each guide point are calculated based on the same calculation method for the pixel coordinates of the first boundary point and the second boundary point corresponding to each guide point; Calculate the world coordinates of each guide point according to the depth information of the first image and the pixel coordinates of each guide point; Generate guide lines based on the world coordinates of each guide point; generating a predicted path of the tracked vehicle within a future period of time according to a first speed of a first track and a second speed of a second track of the tracked vehicle; as well as The guide line and the predicted path are displayed so that an operator can control the tracked vehicle to travel according to the predicted path and the guide line.

2. The method according to claim 1, wherein: The processing includes: Performing semantic segmentation on the first image to obtain a first group of initial boundary points located at a first boundary of the road and a second group of initial boundary points located at a second boundary of the road, the first group of initial boundary points including a plurality of first initial boundary points, and the second group of initial boundary points including a plurality of second initial boundary points; When the number of the plurality of first initial boundary points is less than a first threshold and / or the number of the plurality of second initial boundary points is less than a second threshold, adjusting at least one of the position and the elevation angle of the first camera so that the number of the plurality of first initial boundary points obtained by performing semantic segmentation on the first image captured by the adjusted first camera is greater than or equal to the first threshold, and the number of the plurality of second initial boundary points is greater than or equal to the second threshold; The first group of initial boundary points is used as the first group of boundary points, and the second group of initial boundary points is used as the second group of boundary points.

3. The method according to claim 1, wherein: The pixel coordinates (u, v) of each guide point satisfy the following conditions with the pixel coordinates (u1, v1) of the first boundary point and the pixel coordinates (u2, v2) of the second boundary point corresponding to the guide point: (u, v) = α (u1, v1) + (1-α) (u2, v2), Wherein, α and 1-α are weighted weights of the pixel coordinates of the first boundary point and the second boundary point, respectively, and 0<α<1.

4. The method according to claim 3, wherein: α＝0.5。 5. The method according to any one of claims 1 to 4, wherein: The first direction is a horizontal direction.

6. The method according to claim 5, wherein: The absolute value of the difference in the horizontal coordinates of the pixel coordinates of the first boundary points corresponding to two adjacent guide points in the vertical direction is less than or equal to the third threshold, and the absolute value of the difference in the horizontal coordinates of the pixel coordinates of the second boundary points corresponding to each other is less than or equal to the fourth threshold.

7. The method according to any one of claims 1 to 4, wherein: Calculating the world coordinates of each guide point according to the depth information of the first image and the pixel coordinates of each guide point includes: Acquire a depth value of each guide point from the depth information of the first image; The world coordinates of the guide point are obtained according to the intrinsic parameter matrix, the extrinsic parameter matrix, the depth value of each guide point and the pixel coordinates of each guide point of the first camera.

8. The method according to any one of claims 1 to 4, wherein: The guide line and the predicted path are displayed in a vehicle body coordinate system.

9. The method according to any one of claims 1 to 4, wherein: The guide line and the predicted path are displayed on a remote controller for controlling the tracked vehicle.

10. An auxiliary driving device for a tracked vehicle, comprising: A first acquisition module is configured to acquire a first image of a road on which the tracked vehicle is traveling, taken by a first camera, and a second image of the road, taken by a second camera; A second acquisition module is configured to acquire depth information of the first image according to the second image and the first image; a processing module configured to process the first image to obtain a first group of boundary points located at a first boundary of the road and a second group of boundary points located at a second boundary of the road, wherein the first group of boundary points includes a plurality of first boundary points and the second group of boundary points includes a plurality of second boundary points; a first generating module configured to generate a plurality of guide points according to the first group of boundary points and the second group of boundary points, each guide point being located between a corresponding first boundary point and a second boundary point, different guide points corresponding to different first boundary points and second boundary points, and lines connecting the first boundary points and the second boundary points corresponding to different guide points are all along a first direction, and pixel coordinates of each guide point are calculated based on the same calculation method for the pixel coordinates of the first boundary point and the second boundary point corresponding to each of the guide points; A calculation module, configured to calculate the world coordinates of each guide point according to the depth information of the first image and the pixel coordinates of each guide point; A second generating module is configured to generate a guide line based on the world coordinates of each guide point; A third generating module is configured to generate a predicted path of the tracked vehicle within a future period of time according to a first speed of the first track and a second speed of the second track of the tracked vehicle; as well as The display module is configured to display the guide line and the predicted path so that an operator can control the movement of the tracked vehicle according to the predicted path and the guide line.

11. An auxiliary driving device for a tracked vehicle, comprising: Memory; as well as A processor coupled to the memory is configured to execute the method according to any one of claims 1 to 9 based on instructions stored in the memory.

12. An auxiliary driving system for a tracked vehicle, comprising: The auxiliary driving device according to claim 10 or 11; the first camera; the second camera; as well as The controller is configured to control the movement of the tracked vehicle according to a command signal sent by a remote controller.

13. A tracked vehicle, comprising the auxiliary driving system according to claim 12.

14. A computer-readable storage medium comprising computer program instructions, wherein: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

15. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images