Tracked vehicles and their auxiliary driving methods, devices and systems

Abstract

This disclosure relates to the field of assisted driving, and more particularly to a tracked vehicle and its assisted driving method, apparatus, and system. The method includes: acquiring a first image of a road captured by a first camera and a second image of the road captured by a second camera; acquiring depth information of the first image based on the second image and the first image; processing the first image to obtain a first set of boundary points and a second set of boundary points; generating a plurality of guide points based on the first set of boundary points and the second set of boundary points; calculating the world coordinates of each guide point based on the depth information of the first image and the pixel coordinates of each guide point; generating a guide line based on the world coordinates of each guide point; generating a predicted path for the tracked vehicle over a future period of time based on a first speed of the first track and a second speed of the second track of the tracked vehicle; and displaying the guide line and the predicted path so that an operator can control the tracked vehicle to drive according to the predicted path and the guide line.

Description

Technical Field

[0001] This disclosure relates to the field of assisted driving, and more particularly to a tracked vehicle and its assisted driving method, device and system. Background Technology

[0002] There are two main methods for controlling the movement of existing tracked vehicles: manual control and autonomous control.

[0003] Manual control relies on the operator's visual perception of the vehicle's relative position to the road, which often limits the ability to perform precise operations. Autonomous control solutions use various sensors to replace human perception and achieve measurement and positioning functions, but this requires modeling or learning the vehicle's driving environment, which involves cumbersome preparation work, high costs, and is difficult to popularize. Summary of the Invention

[0004] According to a first aspect of this disclosure, an assisted driving method for a tracked vehicle is proposed, comprising: acquiring a first image of a road traveled by the tracked vehicle captured by a first camera and a second image of the road captured by a second camera; acquiring depth information of the first image based on the second image and the first image; performing processing on the first image to obtain a first set of boundary points located at a first boundary of the road and a second set of boundary points located at a second boundary of the road, wherein the first set of boundary points includes a plurality of first boundary points and the second set of boundary points includes a plurality of second boundary points; generating a plurality of guide points based on the first set of boundary points and the second set of boundary points, wherein each guide point is located between a corresponding first boundary point and a second boundary point, and different guide points correspond to different first boundary points and second boundary points. A first boundary point and a second boundary point are defined, and the lines connecting the first boundary point and the second boundary point corresponding to different guide points are all along a first direction. The pixel coordinates of each guide point are calculated based on the same calculation method for the pixel coordinates of their respective first boundary point and second boundary point. The world coordinates of each guide point are calculated based on the depth information of the first image and the pixel coordinates of each guide point. A guide line is generated based on the world coordinates of each guide point. A predicted path for the tracked vehicle in the future is generated based on the first speed of the first track and the second speed of the second track. The guide line and the predicted path are displayed so that the operator can control the tracked vehicle to move according to the predicted path and the guide line.

[0005] According to some embodiments of this disclosure, the process includes: performing semantic segmentation on the first image to obtain a first set of initial boundary points located at a first boundary of the road and a second set of initial boundary points located at a second boundary of the road, wherein the first set of initial boundary points includes a plurality of first initial boundary points and the second set of initial boundary points includes a plurality of second initial boundary points; if the number of the plurality of first initial boundary points is less than a first threshold and / or the number of the plurality of second initial boundary points is less than a second threshold, adjusting at least one of the position and elevation angle of the first camera, such that the number of the plurality of first initial boundary points obtained by performing semantic segmentation on the first image captured by the adjusted first camera is greater than or equal to the first threshold and the number of the plurality of second initial boundary points is greater than or equal to the second threshold; using the first set of initial boundary points as the first set of boundary points and using the second set of initial boundary points as the second set of boundary points.

[0006] According to some embodiments of this disclosure, the pixel coordinates (u, v) of each guide point and the pixel coordinates (u1, v1) of the first boundary point and the pixel coordinates (u2, v2) of the second boundary point corresponding to the guide point satisfy the following relationship: (u, v) = α(u1, v1) + (1-α)(u2, v2), where α and 1-α are the weighted weights of the pixel coordinates of the first boundary point and the second boundary point, respectively, and 0 < α < 1.

[0007] According to some embodiments of this disclosure, α = 0.5.

[0008] According to some embodiments of this disclosure, the first direction is a horizontal direction.

[0009] According to some embodiments of this disclosure, the absolute value of the difference between the abscissas of the pixel coordinates of the first boundary points corresponding to two adjacent guide points in the vertical direction is less than or equal to a third threshold, and the absolute value of the difference between the abscissas of the pixel coordinates of the second boundary points corresponding to them is less than or equal to a fourth threshold.

[0010] According to some embodiments of this disclosure, calculating the world coordinates of each guide point based on the depth information of the first image and the pixel coordinates of each guide point includes: obtaining the depth value of each guide point from the depth information of the first image; and obtaining the world coordinates of the guide point based on the intrinsic parameter matrix, extrinsic parameter matrix of the first camera, the depth value of each guide point, and the pixel coordinates of each guide point.

[0011] According to some embodiments of this disclosure, the guide lines and the predicted path are displayed in a vehicle coordinate system.

[0012] According to some embodiments of this disclosure, the guide lines and the predicted path are displayed on a remote control for controlling the tracked vehicle.

[0013] According to a second aspect of this disclosure, an auxiliary driving device for a tracked vehicle is provided, comprising: a first acquisition module configured to acquire a first image of a road traveled by the tracked vehicle captured by a first camera and a second image of the road captured by a second camera; a second acquisition module configured to acquire depth information of the first image based on the second image and the first image; a processing module configured to perform processing on the first image to obtain a first set of boundary points located at a first boundary of the road and a second set of boundary points located at a second boundary of the road, the first set of boundary points including a plurality of first boundary points and the second set of boundary points including a plurality of second boundary points; and a first generation module configured to generate a plurality of guide points based on the first set of boundary points and the second set of boundary points, each guide point being located between a corresponding first boundary point and a second boundary point, different guide points corresponding to different... A first boundary point and a second boundary point, and the lines connecting the first boundary point and the second boundary point corresponding to different guide points are all along a first direction. The pixel coordinates of each guide point are calculated based on the same calculation method for the pixel coordinates of their respective first boundary point and second boundary point. A calculation module is configured to calculate the world coordinates of each guide point based on the depth information of the first image and the pixel coordinates of each guide point. A second generation module is configured to generate guide lines based on the world coordinates of each guide point. A third generation module is configured to generate a predicted path of the tracked vehicle for a future period of time based on the first speed of the first track and the second speed of the second track. A display module is configured to display the guide lines and the predicted path so that the operator can control the tracked vehicle to move according to the predicted path and the guide lines.

[0014] According to a third aspect of this disclosure, an auxiliary driving device for a tracked vehicle is provided, comprising: a memory; and a processor coupled to the memory, configured to execute the auxiliary driving method of any of the above embodiments based on instructions stored in the memory.

[0015] According to a fourth aspect of this disclosure, an auxiliary driving system for a tracked vehicle is provided, comprising: an auxiliary driving device of any of the above embodiments; a first camera; a second camera; and a controller configured to control the tracked vehicle to drive according to a command signal sent by a remote controller.

[0016] According to a fifth aspect of this disclosure, a tracked vehicle is provided, including an auxiliary driving system of any of the above embodiments.

[0017] According to a sixth aspect of this disclosure, a computer-readable storage medium is provided, including computer program instructions, wherein the computer program instructions, when executed by a processor, implement the assisted driving method of any of the above embodiments.

[0018] According to a seventh aspect of this disclosure, a computer program product is provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the assisted driving method of any of the above embodiments. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of an assisted driving method for a tracked vehicle according to some embodiments of the present disclosure.

[0021] Figure 2 This is a flowchart of processing a first image according to some embodiments of the present disclosure.

[0022] Figure 3 This is a schematic diagram of the structure of an auxiliary driving device for a tracked vehicle according to some embodiments of the present disclosure.

[0023] Figure 4 This is a structural schematic diagram of an auxiliary driving device for a tracked vehicle according to other embodiments of the present disclosure.

[0024] Figure 5 This is a structural schematic diagram of a tracked vehicle according to some embodiments of the present disclosure. Detailed Implementation

[0025] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0026] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0027] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0028] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0029] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0030] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0031] The inventors noted that in certain scenarios, such as when tracked vehicles are traveling on roads without guide lines, operators may find it difficult to judge the appropriate speed and direction due to limited visibility, insufficient operational skills, and lack of reference markers. Therefore, it is difficult to use tracked vehicles to complete their own needs, such as firefighting operations and transporting goods.

[0032] In view of this, the present disclosure proposes the following technical solutions, which help operators better control the movement of tracked vehicles and improve traffic efficiency and safety.

[0033] Figure 1 This is a flowchart of an assisted driving method for a tracked vehicle according to some embodiments of the present disclosure.

[0034] In step S10, a first image of the road (hereinafter referred to as the road) taken by the first camera and a second image of the road taken by the second camera are acquired.

[0035] It should be understood that the first and second cameras are mounted at different locations on the tracked vehicle and move with the tracked vehicle.

[0036] In step S20, the depth information of the first image is obtained based on the second image and the first image.

[0037] The depth information of the first image includes the depth value of each pixel in the first image.

[0038] In some embodiments, a disparity map of the first image and the second image can be calculated by a correction and semi-global matching algorithm, and a depth image of the first image can be generated by combining the camera calibration parameters (e.g., intrinsic parameter matrix) of the two cameras obtained by calibrating the first camera and the second camera, and the depth information of the first image can be obtained from the depth image.

[0039] In step S30, the first image is processed to obtain a first set of boundary points located at the first boundary of the road and a second set of boundary points located at the second boundary of the road.

[0040] Here, the first set of boundary points includes multiple first boundary points, and the second set of boundary points includes multiple second boundary points. For example, the first boundary can be the left boundary of the road, and the second boundary can be the right boundary of the road, or vice versa.

[0041] Step S30 can be executed simultaneously with step S20, or it can be executed after step S20, or it can be executed before step S20.

[0042] In step S40, multiple guide points are generated based on the first set of boundary points and the second set of boundary points.

[0043] Each guide point has a corresponding first boundary point and a second boundary point. Each guide point is located between its corresponding first and second boundary points. Different guide points correspond to different first and second boundary points, and the lines connecting the first and second boundary points of different guide points are all along the same first direction.

[0044] In other words, we can first find a first boundary point and a second boundary point along a first direction, and then find a point as a guide point by some means along the line connecting the first and second boundary points. We can then repeat this process to generate multiple guide points.

[0045] In some embodiments, a guide point can be generated by weighted summing of the pixel coordinates of the first boundary point and the pixel coordinates of the second boundary point. In this case, the pixel coordinates (u, v) of each guide point satisfy the following relationship with the pixel coordinates (u1, v1) of the corresponding first boundary point and the pixel coordinates (u2, v2) of the corresponding second boundary point:

[0046] (u, v) = α (u1, v1) + (1-α) (u2, v2)

[0047] Where 0 < α < 1, and α and (1-α) are the weighted weights of the pixel coordinates of the first boundary point and the second boundary point, respectively. The closer α is to 0, the closer the guiding point is to the corresponding second boundary point; the closer α is to 1, the closer the guiding point is to the corresponding first boundary point.

[0048] As one implementation method, α = 0.5. In this case, each guide point is the midpoint of the corresponding first and second boundary points. Thus, the guide line is also located in the center of the road, making it easier for operators to refer to and observe.

[0049] In some embodiments, the first direction is the horizontal direction in the image plane of the first image. This not only facilitates finding a pair of first and second boundary points in a first image composed of multiple pixels, but also facilitates the rapid calculation of the pixel coordinates (u, v) of the guide point. For example, only the horizontal coordinate u of the guide point needs to be calculated; the vertical coordinate v of the guide point can be taken as the vertical coordinate of either the first or second boundary point.

[0050] In step S50, the world coordinates (X, Y, Z) of each guide point are calculated based on the depth information of the first image and the pixel coordinates (u, v) of each guide point.

[0051] In some embodiments, the world coordinates (X, Y, Z) of each guide point can be obtained in the following manner.

[0052] First, the depth value Z of each guide point is obtained from the depth information of the first image. c .

[0053] Secondly, based on the intrinsic parameter matrix I, extrinsic parameter matrix V, and depth value Z of each guide point of the first camera... c Given the pixel coordinates (u, v) of each guide point, we can obtain the world coordinates (X, Y, Z) of that guide point.

[0054] For example, the world coordinates (X, Y, Z) of the guide point can be obtained using the following formula:

[0055] Z c [u,v,1] T =IV[X,Y,Z,1] T

[0056] Where, [u,v,1] T The matrix representation of the pixel coordinates (u, v) of the guide point after conversion to homogeneous coordinates is [X, Y, Z, 1]. T The matrix representation of the world coordinates (X, Y, Z) of the guiding point after conversion to homogeneous coordinates.

[0057] The intrinsic parameter matrix I and extrinsic parameter matrix V of the first camera can be obtained by calibrating the first camera.

[0058] For example, the intrinsic parameter matrix I of the first camera can have the following form:

[0059]

[0060] In the intrinsic parameter matrix I, f is the focal length of the first camera, and s x and s y Let u0 and v0 be the dimensions of a single pixel of the first camera in the horizontal and vertical directions, respectively, and let u0 and v0 be the x and y coordinates of the pixel coordinates of the principal point of the first camera, respectively.

[0061] For example, the extrinsic parameter matrix V of the first camera can have the following form:

[0062]

[0063] In the extrinsic parameter matrix V, R is a 3×3 matrix representing the rotation relationship between the camera coordinate system of the first camera and the world coordinate system; T is a 3×1 matrix representing the translation relationship between the position of the first camera and the origin of the world coordinate system.

[0064] In step S60, a guide line is generated based on the world coordinates of each guide point.

[0065] In some embodiments, multiple guide points can be connected sequentially to generate guide lines.

[0066] In other embodiments, multiple guide points can be fitted (e.g., polynomial fitting) to generate guide lines. This results in smoother guide lines, which are easier for operators to reference and observe.

[0067] In some embodiments, multiple guide points can be projected onto the ground (i.e., Y=0) first, and then the multiple guide points can be connected sequentially or fitted.

[0068] In step S70, a predicted path for the tracked vehicle over a future time period T is generated based on the first speed v1 of the first track of the tracked vehicle and the second speed v2 of the second track of the tracked vehicle.

[0069] In some embodiments, the linear velocity v and angular velocity w of a tracked vehicle can be calculated based on the kinematic model of the tracked vehicle using the following formula:

[0070]

[0071] Where B is the distance between the first and second tracks of the tracked vehicle.

[0072] Subsequently, the current position (x, z) of the tracked vehicle is obtained, and the motion of the tracked vehicle is discretized to calculate the position of the tracked vehicle at multiple times in the future time period T, thereby generating a predicted path of the tracked vehicle in the future time period T (e.g., by connecting the positions at multiple times or by fitting such as polynomial fitting).

[0073] For example, the position of the tracked vehicle is calculated every time interval dt using the following formula, until the number of calculations multiplied by the time interval dt exceeds T:

[0074]

[0075] Where t and t+dt are the previous and next moments before each calculation, respectively, and θ is the heading angle of the tracked vehicle. The time interval dt can be set according to requirements, such as 0.01s, 0.1s, or 0.5s.

[0076] In step S80, the guide lines and the predicted path are displayed so that the operator can control the movement of the tracked vehicle based on the predicted path and the guide lines.

[0077] Through the steps S10-S80 described above, during the movement of the tracked vehicle, guide lines indicating the direction of the road where the tracked vehicle is located can be displayed in real time, along with the predicted path of the tracked vehicle. Thus, even on roads without guide lines that are difficult to navigate, operators can better control the tracked vehicle, improving traffic efficiency and safety.

[0078] For example, operators can control the movement of tracked vehicles based on the positional relationship between the predicted path and the guide line. When the predicted path and the guide line have a high degree of overlap, the operator can maintain the tracked vehicle's movement unchanged; when the predicted path deviates to one side relative to the guide line, the operator can adjust the tracked vehicle's direction of travel to make it move to the other side.

[0079] In some embodiments, the guide line and the predicted path of the tracked vehicle can be displayed in a world coordinate system. Since the starting point of the predicted path is the position of the tracked vehicle, the operator can simultaneously observe the movement of the tracked vehicle.

[0080] In other embodiments, the guide line and the predicted path of the tracked vehicle can be displayed in the vehicle coordinate system. In this case, it is also necessary to transform the world coordinates (X, Y, Z) of the guide point obtained in step S50 to the vehicle coordinate system. In the formula for calculating the position of the tracked vehicle, x = 0, z = 0, and θ = π / 2 in the first calculation. This makes it easier for the operator to observe and judge the positional relationship between the predicted path and the guide line, and facilitates the operator to adjust the form of the tracked vehicle.

[0081] In some embodiments, the guide line and the predicted path are displayed on a remote control used to control the tracked vehicle. This allows the operator to observe and determine the positional relationship between the predicted path and the guide line using the remote control, while simultaneously controlling the tracked vehicle's movement, increasing operational convenience.

[0082] Figure 2 This is a flowchart of processing a first image according to some embodiments of the present disclosure.

[0083] like Figure 2 As shown, step S30 may include steps S301, S302 and S303.

[0084] In step S301, semantic segmentation is performed on the first image to obtain a first set of initial boundary points located at the first boundary of the road and a second set of initial boundary points located at the second boundary of the road. Similarly, the first set of initial boundary points includes multiple first initial boundary points, and the second set of initial boundary points includes multiple second initial boundary points.

[0085] In step S302, if the number of first initial boundary points in the first group of initial boundary points is less than the first threshold and / or the number of second initial boundary points in the second group of initial boundary points is less than the second threshold, at least one of the position and elevation angle of the first camera is adjusted so that the number of first initial boundary points obtained by performing semantic segmentation on the first image captured by the adjusted first camera is greater than or equal to the first threshold and the number of second initial boundary points is greater than or equal to the second threshold.

[0086] In step S303, the first set of initial boundary points is used as the first set of boundary points, and the second set of initial boundary points is used as the second set of boundary points.

[0087] Through steps S301-S303, the first camera can better capture images of the road and obtain a sufficient number of first and second boundary points, which is beneficial for obtaining a sufficient number of guide points, so that the subsequently generated guide lines can more accurately reflect the extension direction of the road.

[0088] In some embodiments, at least one of the first threshold and the second threshold can be set to the number of pixels in the first image in the vertical direction. In this case, if the number of first boundary points or second boundary points is less than the first threshold, it indicates that the first boundary or second boundary of the road does not extend from the bottom to the top of the first image, for example, from the side to the top or bottom. By adjusting the first camera, it can be made to capture more of the first boundary or second boundary of the road.

[0089] In other embodiments, at least one of the first threshold and the second threshold can be set to a number less than the number of pixels in the first image in the vertical direction. For example, the first threshold or the second threshold can be 0.9 times or more of the number of pixels in the first image in the vertical direction, such as 0.95 times or 0.9 times. Thus, even if some of the first boundary points or the second boundary points are filtered out for other reasons (see below), the number of the first boundary points or the second boundary points can still be considered sufficient.

[0090] In some embodiments, the first threshold may be equal to the second threshold.

[0091] For example, both the first threshold and the second threshold are set to the number of pixels in the vertical direction of the first image. This allows the first camera to capture more of the two boundaries of the road simultaneously.

[0092] For example, both the first threshold and the second threshold are set to be less than the number of pixels in the vertical direction of the first image (e.g., 0.95 times and 0.9 times the number of pixels).

[0093] In some embodiments, in the first image, the pixel distance between the first boundary points corresponding to two adjacent guide points in a second direction perpendicular to the first direction is less than or equal to a third threshold, and the pixel distance between the second boundary points corresponding to each of them in the first direction is less than or equal to a fourth threshold.

[0094] For example, in the case where the first direction is horizontal, in the first image, the absolute value of the difference between the x-coordinates u1 of the first boundary points corresponding to two adjacent guide points in the vertical direction is less than or equal to the third threshold, and the absolute value of the difference between the x-coordinates u2 of the corresponding second boundary points is less than or equal to the fourth threshold.

[0095] In the above embodiments, the first and second boundary points corresponding to adjacent guide points are relatively close, that is, the distribution continuity of the first and second sets of boundary points used to generate multiple guide points is good. For example, the first and second sets of boundary points can be filtered to retain only those first and second boundary points that are distributed more continuously.

[0096] The following describes a method for filtering the first and second sets of boundary points, using the first direction as the horizontal direction:

[0097] (1) Construct an empty queue data structure. Each element of the queue includes the pixel coordinates of a first boundary point (u1, v1) and the pixel coordinates of a second boundary point (u2, v2).

[0098] (2) Insert the x-coordinate u1 of the first boundary point and the x-coordinate u2 of the second boundary point in the first row of pixels of the first image into the end of the queue. The first row of pixels can be the pixels of the bottom row or the top row of the first image.

[0099] (3) Perform the following operations on each row of pixels from the second row to the last row of pixels in the first image in the order from bottom to top or from top to bottom.

[0100] Determine whether the first difference between the x-coordinate u1 of the first boundary point in the row of pixels and the first boundary point in the last element of the queue is less than or equal to the third threshold, and whether the second difference between the x-coordinate u2 of the second boundary point in the row of pixels and the second boundary point in the last element of the queue is less than or equal to the fourth threshold. Here, the first and second differences refer to the absolute values ​​of the differences.

[0101] If the first difference is less than or equal to the third threshold and the second difference is less than or equal to the fourth threshold, insert the x-coordinate u1 of the first boundary point and the x-coordinate u2 of the second boundary point in the row as elements at the end of the queue; otherwise, discard the first boundary point and the second boundary point of the row.

[0102] (4) Take the first boundary point and the second boundary point of all elements in the queue as the first boundary point and the second boundary point after filtering, respectively.

[0103] The above method, by using a queue, ensures that even if some first and second boundary points are filtered out, the first and second boundary points that pass the filtering each time are always close to the first and second boundary points that passed the filtering in the previous time. In other words, it ensures the continuity of the distribution of the first and second set of boundary points.

[0104] Figure 3 This is a schematic diagram of the structure of an auxiliary driving device 100 for a tracked vehicle according to some embodiments of the present disclosure.

[0105] like Figure 3 As shown, the auxiliary driving device 100 for tracked vehicles includes: a first acquisition module 101, a second acquisition module 102, a processing module 103, a first generation module 104, a calculation module 105, a second generation module 106, a third generation module 107, and a display module 108.

[0106] The first acquisition module 101 is configured to acquire a first image of the road traveled by the tracked vehicle captured by a first camera and a second image of the road captured by a second camera.

[0107] The second acquisition module 102 is configured to acquire depth information of the first image based on the second image and the first image.

[0108] The processing module 103 is configured to perform processing on the first image to obtain a first set of boundary points located at a first boundary of the road and a second set of boundary points located at a second boundary of the road. The first set of boundary points includes a plurality of first boundary points, and the second set of boundary points includes a plurality of second boundary points.

[0109] The first generation module 104 is configured to generate multiple guide points based on the first set of boundary points and the second set of boundary points. Each guide point is located between the corresponding first and second boundary points. Different guide points correspond to different first and second boundary points, and the lines connecting the first and second boundary points corresponding to different guide points are all along the first direction. The pixel coordinates of each guide point are calculated based on the same calculation method for the pixel coordinates of their respective first and second boundary points.

[0110] The calculation module 105 is configured to calculate the world coordinates of each guide point based on the depth information of the first image and the pixel coordinates (u, v) of each guide point.

[0111] The second generation module 106 is configured to generate guide lines based on the world coordinates of each guide point.

[0112] The third generation module 107 is configured to generate a predicted path for the tracked vehicle over a future period of time based on the first speed of the first track and the second speed of the second track.

[0113] The display module 108 is configured to display guide lines and predicted paths so that operators can control the movement of the tracked vehicle based on the predicted paths and guide lines.

[0114] In some embodiments, the assisted driving device 100 further includes other modules to perform the assisted driving method of any of the above embodiments.

[0115] Figure 4 This is a schematic diagram of the structure of an auxiliary driving device 100' for a tracked vehicle according to other embodiments of the present disclosure.

[0116] The auxiliary driving device 100' of the tracked vehicle includes a memory 11 and a processor 12 coupled to the memory 11.

[0117] The processor 12 is configured to execute the assisted driving method of the tracked vehicle in any of the above embodiments based on instructions stored in the memory 11.

[0118] The memory 11 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory may store, for example, the operating system, application programs, boot loader, and other programs.

[0119] In some embodiments, the auxiliary driving device 100' of the tracked vehicle may further include an input / output interface 13, a network interface 14, a storage interface 15, etc. These interfaces 13, 14, and 15, as well as the memory 11 and processor 12, can be connected, for example, via a bus 16. The input / output interface 13 provides a connection interface for input / output devices such as displays, mice, keyboards, and touchscreens. The network interface 14 provides a connection interface for various networked devices. The storage interface 15 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0120] This disclosure also proposes an auxiliary driving system for a tracked vehicle, comprising: an auxiliary driving device for the tracked vehicle according to any of the above embodiments, a first camera, a second camera, and a controller. The controller is configured to control the movement of the tracked vehicle according to command signals sent by a remote controller.

[0121] The remote controller may, for example, have a first opening (e.g., longitudinal opening) of a first joystick associated with the speed v of the tracked vehicle and a second opening (e.g., lateral opening) of a second joystick associated with the angular velocity w of the tracked vehicle.

[0122] This disclosure also proposes a tracked vehicle 200, which includes the aforementioned auxiliary driving system for tracked vehicles.

[0123] Figure 5 This is a structural schematic diagram of a tracked vehicle 200 according to some embodiments of the present disclosure, wherein the left side is a front view of the tracked vehicle 200 and the right side is a side view of the tracked vehicle 200.

[0124] like Figure 5 As shown, the tracked vehicle 200 includes a first camera 201 and a second camera 202, a first track 251 and a second track 252, a controller (not shown), and an auxiliary driving device for the tracked vehicle in any of the above embodiments, such as auxiliary driving device 100 or auxiliary driving device 100' (not shown).

[0125] The controller is configured to control the movement of the tracked vehicle 200 according to command signals sent by the remote controller. For example, the controller can be a proportional-integral-derivative (PID) controller.

[0126] The first camera 201 and the second camera 202 are mounted on the tracked vehicle 200. For example, both the first camera 201 and the second camera 202 are facing the direction of travel of the tracked vehicle 200.

[0127] In some embodiments, such as Figure 5As shown, the first camera 201 and the second camera 202 are mounted on the front end of the tracked vehicle 200. In some implementations, the first camera 201 and the second camera 202 are mounted on opposite sides of the front end of the tracked vehicle 200, for example, the first camera 201 is mounted on the left side of the front end and the second camera 202 is mounted on the right side, or the first camera 201 is mounted on the right side of the front end and the second camera 202 is mounted on the left side. In this way, the first and second images can better reflect the road conditions.

[0128] In some embodiments, such as Figure 5 As shown, the tracked vehicle 200 also includes a bracket 203 for supporting the first camera 201 and the second camera 202. The bracket 203, the first camera 201, and the second camera 202 together form a binocular system 210.

[0129] In some embodiments, such as Figure 5 As shown, the tracked vehicle 200 also includes a mobile chassis 220, a wireless module 230, and a speed sensor 240.

[0130] The auxiliary driving device 100 or auxiliary driving device 100' of the tracked vehicle can be connected to the binocular system 210, the mobile chassis 220, the wireless module 230 and the speed sensor 240 respectively via USB cable or CAN bus.

[0131] The wireless module 230 is configured to receive command signals from the remote controller and send the command signals to the controller of the tracked vehicle.

[0132] The speed sensor 240 is configured to monitor the first speed v1 of the first track 251 and the second speed v2 of the second track 252 in real time, and send the first speed v1 and the second speed v2 as feedback signals to the controller so that the controller can adjust the first speed v1 and the second speed v2 to the desired values.

[0133] This disclosure also provides a computer-readable storage medium including computer program instructions that, when executed by a processor, implement the assisted driving method for a tracked vehicle according to any of the above embodiments.

[0134] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the assisted driving method for a tracked vehicle according to any of the above embodiments.

[0135] The embodiments of this disclosure have now been described in detail. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein based on the above description.

[0136] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that the functions specified in one or more flowchart illustrations and / or one or more blocks in a block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate functions for implementing the functions in the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0138] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0139] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0140] While specific embodiments of this disclosure have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments or equivalent substitutions can be made to some technical features without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for assisting the movement of a tracked vehicle, comprising: Acquire a first image of the road traveled by the tracked vehicle captured by a first camera and a second image of the road captured by a second camera; The depth information of the first image is obtained based on the second image and the first image; The first image is processed to obtain a first set of boundary points located at a first boundary of the road and a second set of boundary points located at a second boundary of the road, wherein the first set of boundary points includes a plurality of first boundary points and the second set of boundary points includes a plurality of second boundary points; Multiple guide points are generated based on the first set of boundary points and the second set of boundary points. Each guide point is located between the corresponding first boundary point and second boundary point. Different guide points correspond to different first boundary points and second boundary points. The lines connecting the first boundary points and second boundary points corresponding to different guide points are all along the first direction. The pixel coordinates of each guide point are calculated based on the same calculation method for the pixel coordinates of their respective first boundary points and second boundary points. Based on the depth information of the first image and the pixel coordinates of each guide point, the world coordinates of each guide point are calculated. Generate guide lines based on the world coordinates of each guide point; Based on the first speed of the first track and the second speed of the second track of the tracked vehicle, a predicted path for the tracked vehicle over a future period of time is generated; as well as The guide lines and the predicted path are displayed so that the operator can control the tracked vehicle to move according to the predicted path and the guide lines.

2. The method according to claim 1, wherein, The process includes: Semantic segmentation is performed on the first image to obtain a first set of initial boundary points located at the first boundary of the road and a second set of initial boundary points located at the second boundary of the road. The first set of initial boundary points includes a plurality of first initial boundary points, and the second set of initial boundary points includes a plurality of second initial boundary points. If the number of the plurality of first initial boundary points is less than a first threshold and / or the number of the plurality of second initial boundary points is less than a second threshold, at least one of the position and elevation angle of the first camera is adjusted so that the number of the plurality of first initial boundary points obtained by performing semantic segmentation on the first image captured by the adjusted first camera is greater than or equal to the first threshold and the number of the plurality of second initial boundary points is greater than or equal to the second threshold. The first set of initial boundary points is used as the first set of boundary points, and the second set of initial boundary points is used as the second set of boundary points.

3. The method according to claim 1, wherein, The pixel coordinates (u, v) of each guide point are related to the pixel coordinates (u1, v1) of the first boundary point and the pixel coordinates (u2, v2) of the second boundary point corresponding to that guide point, satisfying the following: (u, v) = α (u1, v1) + (1-α) (u2, v2), Where α and 1-α are the weighted weights of the pixel coordinates of the first boundary point and the second boundary point, respectively, and 0<α<1.

4. The method according to claim 3, wherein, α＝0.5。 5. The method according to any one of claims 1-4, wherein, The first direction is the horizontal direction.

6. The method according to claim 5, wherein, The absolute value of the difference between the x-coordinates of the pixel coordinates of the first boundary points of two adjacent guide points in the vertical direction is less than or equal to the third threshold, and the absolute value of the difference between the x-coordinates of the pixel coordinates of the second boundary points of each of them is less than or equal to the fourth threshold.

7. The method according to any one of claims 1-4, wherein, The world coordinates of each guide point are calculated based on the depth information of the first image and the pixel coordinates of each guide point, including: Obtain the depth value of each guide point from the depth information of the first image; The world coordinates of the guide point are obtained based on the intrinsic and extrinsic parameter matrices of the first camera, the depth value of each guide point, and the pixel coordinates of each guide point.

8. The method according to any one of claims 1-4, wherein, The guide line and the predicted path are displayed in the vehicle coordinate system.

9. The method according to any one of claims 1-4, wherein, The guide line and the predicted path are displayed on a remote control used to control the tracked vehicle.

10. An auxiliary driving device for a tracked vehicle, comprising: The first acquisition module is configured to acquire a first image of the road traveled by the tracked vehicle captured by a first camera and a second image of the road captured by a second camera; The second acquisition module is configured to acquire depth information of the first image based on the second image and the first image; The processing module is configured to perform processing on the first image to obtain a first set of boundary points located at a first boundary of the road and a second set of boundary points located at a second boundary of the road, wherein the first set of boundary points includes a plurality of first boundary points and the second set of boundary points includes a plurality of second boundary points; The first generation module is configured to generate multiple guide points based on the first set of boundary points and the second set of boundary points. Each guide point is located between the corresponding first boundary point and the second boundary point. Different guide points correspond to different first boundary points and second boundary points, and the lines connecting the first boundary points and the second boundary points corresponding to different guide points are all along the first direction. The pixel coordinates of each guide point are calculated based on the same calculation method for the pixel coordinates of their respective first boundary points and second boundary points. The calculation module is configured to calculate the world coordinates of each guide point based on the depth information of the first image and the pixel coordinates of each guide point; The second generation module is configured to generate guide lines based on the world coordinates of each guide point; The third generation module is configured to generate a predicted path for the tracked vehicle over a future period of time based on the first speed of the first track and the second speed of the second track. as well as The display module is configured to display the guide lines and the predicted path so that an operator can control the movement of the tracked vehicle based on the predicted path and the guide lines.

11. An auxiliary driving device for a tracked vehicle, comprising: Memory; as well as A processor coupled to the memory is configured to execute the method of any one of claims 1-9 based on instructions stored in the memory.

12. An auxiliary driving system for a tracked vehicle, comprising: The auxiliary driving device as described in claim 10 or 11; The first camera; The second camera; as well as The controller is configured to control the movement of the tracked vehicle according to instruction signals sent by the remote controller.

13. A tracked vehicle comprising the auxiliary driving system of claim 12.

14. A computer-readable storage medium comprising computer program instructions, wherein, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-9.

15. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the method described in any one of claims 1-9.

Images