Train hinging angle visual estimation method and device based on deep learning

Through a deep learning-based method, the panoramic circumferential viewing system and neural network are used to estimate the articulation angle of the car train, which solves the problem of easy damage to the angle sensor and inconvenient coupling and decoupling, improves the reliability and durability of the system, and reduces cost and maintenance. Complexity.

CN119068047APending Publication Date: 2024-12-03TSINGHUA UNIVERSITY
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411213186.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the prior art, the hinge angle measurement of automobile trains mainly relies on angle sensors, which are susceptible to physical damage, are inconvenient to periodic replacement, and the coupling and decoupling process is heavy, resulting in inconvenient sensors in actual use.

Method used

Using the visual estimation method of articulation angle of automobile trains based on deep learning, the visual image and angle real values ​​are obtained through the panoramic circumference system, the target neural network is constructed, the relative spatial attitude characteristics between vehicle units are extracted, and the time dependence is compensated by ConvLSTM, and the rotation is decomposed. The matrix is ​​Euler angle to estimate the articulation angle.

Benefits of technology

Avoid the risk of physical damage of angle sensors, reduce the risk of failure, provide hot standby redundancy for sensors, improve system reliability and durability, reduce costs, simplify installation and maintenance processes, avoid coupling to car trains and Interference in the decoupling process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068047A_ABST
    Figure CN119068047A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of crossing of vehicles and environment perception, in particular to a deep learning-based train hinging angle visual estimation method and device, and the method comprises the steps: obtaining a visual image captured by a panoramic looking-around system disposed on a train and an angle true value between two vehicle units of the train; constructing a target neural network, calculating a spatial attitude true value of the angle true value at a corresponding time point of the visual image, constructing a target loss function by using the spatial attitude true value, and training the target neural network by using the target loss function; inputting the visual image into the trained neural network, extracting the features of the relative spatial attitude between the two vehicle units, compensating the time dependence of the features through ConvLSTM to obtain encoded features, and decoding the spatial attitude of the target from the encoded features; the rotation matrix of the spatial attitude is decomposed into Euler angles to estimate the vehicle articulation angle of the motor train.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of intersection of vehicle and environment perception, and in particular to a method and device for visually estimating the articulation angle of a vehicle and a train based on deep learning. Background Art

[0002] As a new type of automatic rail transit mode, auto trains have gradually become an important and daily means of transportation in more and more cities in China. Compared with traditional passenger cars, auto trains are composed of multiple vehicle units connected by hinges, so they have advantages in transportation efficiency. As a significantly more complex dynamic system, auto trains are more prone to unstable movements and face more challenges in actual use. Among them, the articulation angle is always a necessary and inevitable variable for the autonomous driving system. The articulation angle constitutes the most significant and decisive difference in the dynamic characteristics of auto trains and passenger cars. The articulation angle plays an important role in determining the movement of auto trains. Therefore, the importance of measuring or estimating the articulation angle is self-evident.

[0003] In the related art, the articulation angle is usually measured using an angle sensor, such as a high-precision measuring device such as a laser rangefinder. The angle sensor is usually installed near the articulation point to achieve the function of measuring the articulation angle in real time.

[0004] However, the related art uses angle sensors to measure articulation angles, which are susceptible to physical damage and are inconvenient to replace regularly. In addition, because the coupling and decoupling processes of automobiles and trains are very cumbersome, the angle sensors are also very inconvenient in actual use due to the inconvenience of coupling and decoupling, which needs to be solved urgently. Summary of the invention

[0005] The present application provides a method and device for visually estimating the articulation angle of a car train based on deep learning, so as to solve the problems in the related art of using angle sensors to measure the articulation angle, the angle sensors are susceptible to physical damage and are inconvenient to replace regularly, and because the coupling and decoupling process of the car train is very cumbersome, the angle sensors are also very inconvenient in actual use due to the inconvenience of coupling and decoupling.

[0006] The first aspect of the present application provides a method for visual estimation of the articulation angle of a car train based on deep learning, comprising the following steps: obtaining a visual image captured by a panoramic surround view system provided on the car train and a true value of an angle between two vehicle units of the car train; constructing a target neural network, calculating a true value of a spatial posture of the true value of the angle at a corresponding time point of the visual image, so as to construct a target loss function using the true value of the spatial posture, and training the target neural network using the target loss function to obtain a trained neural network, wherein the true value of the spatial posture includes a rotation matrix and a translation vector; inputting the visual image into the trained neural network, extracting features of the relative spatial posture between the two vehicle units of the car train, compensating for the time dependence of the features through ConvLSTM, obtaining encoded features, and decoding the spatial posture of the target from the encoded features; decomposing the rotation matrix of the spatial posture into Euler angles to estimate the vehicle articulation angle of the car train.

[0007] Optionally, in one embodiment of the present application, before calculating the true value of the angle and the true value of the spatial posture at the corresponding time point of the visual image, it also includes: obtaining a first angle and a second angle between two vehicle units of the car train, wherein the first angle is a side where the angle between the two vehicle units is less than 180°, and the second angle is a side where the angle between the two vehicle units is greater than 180°; based on the first angle and the second angle, determining the true value of the angle.

[0008] Optionally, in one embodiment of the present application, the calculation of the spatial posture true value of the angle true value at the corresponding time point of the visual image includes: compensating for the installation angle of the panoramic view system so that the acquisition plane of the panoramic view system rotates to the movement plane of the car train, so as to calculate the spatial posture true value of the angle true value at the corresponding time point of the visual image.

[0009] Optionally, in one embodiment of the present application, the rotation matrix in the true value of the spatial posture is:

[0010]

[0011] in, is the rotation matrix of the true value of the spatial posture at time t, represents the rotation matrix of the adjacent camera coordinate system around the x-axis by an angle of π / 2-γ. The rotation matrix represents the rotation of the vehicle's motion plane around the y-axis at time t, which transforms the camera coordinate system to the adjacent camera coordinate system. θ is the articulation angle. x R i(γ) represents the rotation matrix of the camera coordinate system by an angle of π / 2-γ, where

[0012]

[0013]

[0014] Optionally, in one embodiment of the present application, the translation vector in the true value of the spatial posture is:

[0015]

[0016] in, represents the translation vector in the true value of the spatial posture at time t, represents the rotation matrix of the adjacent camera coordinate system around the x-axis by an angle of π / 2-γ. The rotation matrix represents the rotation of the vehicle's motion plane around the y-axis at time t, which transforms the camera coordinate system to the adjacent camera coordinate system. θ is the articulation angle, t w represents the lateral displacement from one side of the same vehicle to the other side, t w =[0, 0, s w ] T ,s w represents the vehicle width, t l represents the longitudinal displacement from the camera installation point to the vehicle hinge point, t l =[s l , 0, 0] T ,s l Indicates the longitudinal lateral distance from the camera mounting point to the vehicle hinge point.

[0017] Optionally, in one embodiment of the present application, the network structure, input and output of the target neural network are respectively:

[0018]

[0019] h (t) =ConvLSTM(h (t-1) , c (t-1) , x (t) ),

[0020]

[0021] in, Camera C represents the adjacent camera coordinate system j The image captured at time t, The camera C represents the camera coordinate system i The image captured at time t, x (t)represents the feature map obtained by the input layer at time t, PEnc represents the posture encoder, ConvLSTM represents a single-layer ConvLSTM of the deep learning model, and h (t) represents the hidden state of a single-layer ConvLSTM at time t, h (t-1) represents the hidden state at the previous time, c (t-1) represents the unit state at the previous time, represents the first spatial pose estimation result from the camera coordinate system to the adjacent camera coordinate system at time t, and PDec represents the pose decoder.

[0022] Optionally, in one embodiment of the present application, the expression for training the target neural network using the target loss function is:

[0023]

[0024]

[0025]

[0026]

[0027] in, represents the transposed result of the first spatial attitude estimation result, x R i (γ) represents the rotation matrix of the camera coordinate system by an angle of π / 2-γ. Represents the transposed result of the rotation matrix rotating around the x-axis of the camera coordinate system by an angle of π / 2-γ. represents the estimated result of the spatial articulated posture, represents the estimated result of the first translation vector, represents the translation vector in the true value of the spatial attitude at time t, represents the rotation matrix of the adjacent camera coordinate system around the x-axis by an angle of π / 2-γ. represents the loss function at time t, Frob represents the Frobenius norm, λ represents the weight coefficient, I is the unit matrix, represents the target loss function, T represents the time length of the image sequence input to the network, represents the probability density corresponding to a certain hinge angle θ, t w =[0, 0, s w ] T ,s w represents the vehicle width, t l represents the longitudinal displacement from the camera installation point to the vehicle hinge point, t l =[s l , 0, 0] T,s l represents the longitudinal lateral distance from the camera installation point to the vehicle hinge point, is the rotation matrix of the true value of the spatial posture at time t.

[0028] The second aspect of the present application provides a deep learning-based visual estimation device for the articulation angle of a car train, comprising: an acquisition module for acquiring a visual image captured by a panoramic surround view system provided on the car train and a true value of an angle between two vehicle units of the car train; a calculation module for constructing a target neural network, calculating a true value of a spatial posture of the true value of the angle at a corresponding time point of the visual image, so as to construct a target loss function using the true value of the spatial posture, and train the target neural network using the target loss function to obtain a trained neural network, wherein the true value of the spatial posture includes a rotation matrix and a translation vector; a decoding module for inputting the visual image into the trained neural network, extracting features of the relative spatial posture between the two vehicle units of the car train, compensating for the time dependence of the features through ConvLSTM, obtaining encoded features, and decoding the spatial posture of the target from the encoded features; an estimation module for decomposing the rotation matrix of the spatial posture into Euler angles to estimate the vehicle articulation angle of the car train.

[0029] Optionally, in one embodiment of the present application, it also includes: an acquisition module, used to obtain a first angle and a second angle between two vehicle units of the car train before calculating the true value of the spatial posture of the true value of the angle at the corresponding time point of the visual image, wherein the first angle is a side where the angle between the two vehicle units is less than 180°, and the second angle is a side where the angle between the two vehicle units is greater than 180°; a determination module, used to determine the true value of the angle based on the first angle and the second angle.

[0030] Optionally, in one embodiment of the present application, the calculation module includes: a compensation unit, used to compensate for the installation angle of the panoramic view system, so that the acquisition plane of the panoramic view system is rotated to the movement plane of the car train, so as to calculate the spatial posture true value of the angle true value at the corresponding time point of the visual image.

[0031] Optionally, in one embodiment of the present application, the rotation matrix in the true value of the spatial posture is:

[0032]

[0033] in, is the rotation matrix of the true value of the spatial posture at time t, represents the rotation matrix of the adjacent camera coordinate system around the x-axis by an angle of π / 2-γ. The rotation matrix represents the rotation of the vehicle's motion plane around the y-axis at time t, which transforms the camera coordinate system to the adjacent camera coordinate system. θ is the articulation angle. x R i (γ) represents the rotation matrix of the camera coordinate system by an angle of π / 2-γ, where

[0034]

[0035]

[0036] Optionally, in one embodiment of the present application, the translation vector in the true value of the spatial posture is:

[0037]

[0038] in, represents the translation vector in the true value of the spatial posture at time t, represents the rotation matrix of the adjacent camera coordinate system around the x-axis by an angle of π / 2-γ. The rotation matrix represents the rotation of the vehicle's motion plane around the y-axis at time t, which transforms the camera coordinate system to the adjacent camera coordinate system. θ is the articulation angle, t w represents the lateral displacement from one side of the same vehicle to the other side, t w =[0, 0, s w ] T ,s w represents the vehicle width, t l represents the longitudinal displacement from the camera installation point to the vehicle hinge point, t l =[s l , 0, 0] T ,s l Indicates the longitudinal lateral distance from the camera mounting point to the vehicle hinge point.

[0039] Optionally, in one embodiment of the present application, the network structure, input and output of the target neural network are respectively:

[0040]

[0041] h (t) =ConvLSTM(h (t-1) , c (t-1) , x (t) ),

[0042]

[0043] in, Camera C represents the adjacent camera coordinate system j The image captured at time t, The camera C represents the camera coordinate system i The image captured at time t, x (t) represents the feature map obtained by the input layer at time t, PEnc represents the posture encoder, ConvLSTM represents a single-layer ConvLSTM of the deep learning model, and h (t) represents the hidden state of a single-layer ConvLSTM at time t, h (t-1) represents the hidden state at the previous time, c (t-1) represents the unit state at the previous time, represents the first spatial pose estimation result from the camera coordinate system to the adjacent camera coordinate system at time t, and PDec represents the pose decoder.

[0044] Optionally, in one embodiment of the present application, the expression for training the target neural network using the target loss function is:

[0045]

[0046]

[0047]

[0048]

[0049] in, represents the transposed result of the first spatial attitude estimation result, x R i (γ) represents the rotation matrix of the camera coordinate system by an angle of π / 2-γ. Represents the transposed result of the rotation matrix rotating around the x-axis of the camera coordinate system by an angle of π / 2-γ. represents the estimated result of the spatial articulated posture, represents the estimated result of the first translation vector, represents the translation vector in the true value of the spatial attitude at time t, represents the rotation matrix of the adjacent camera coordinate system around the x-axis by an angle of π / 2-γ. represents the loss function at time t, Frob represents the Frobenius norm, λ represents the weight coefficient, I is the unit matrix, represents the target loss function, T represents the time length of the image sequence input by the network, represents the probability density corresponding to a certain hinge angle θ, t w =[0, 0, s w ]T ,s w represents the vehicle width, t l represents the longitudinal displacement from the camera installation point to the vehicle hinge point, t l =[s l , 0, 0] T ,s l represents the longitudinal lateral distance from the camera installation point to the vehicle hinge point, is the rotation matrix of the true value of the spatial posture at time t.

[0050] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the deep learning-based visual estimation method for the articulation angle of a vehicle train as described in the above embodiment.

[0051] The fourth aspect embodiment of the present application provides a computer-readable storage medium, which stores a computer program, which, when executed by a processor, implements the above-mentioned deep learning-based visual estimation method for the articulation angle of a car train.

[0052] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned deep learning-based visual estimation method for the articulation angle of a vehicle train.

[0053] The embodiment of the present application can extract the features of the relative spatial posture between vehicle units by acquiring the visual image and the true angle value captured by the panoramic system, and then decode the spatial posture information of the target from the encoded features through the posture decoder, and finally decompose the rotation matrix of the spatial posture into Euler angles to estimate the vehicle articulation angle of the car train. Thus, the vehicle articulation angle is estimated by only using the visual image and the true angle value captured by the panoramic surround system installed on the car train, avoiding the risk of the angle sensor being susceptible to physical damage, reducing the risk of sensor failure caused by physical damage, providing hot standby redundancy for the angle sensor, improving the reliability and durability of the system, and not interfering with the coupling and decoupling process of the car train, reducing costs and simplifying the installation and maintenance process. Thus, the problems of using angle sensors for articulation angle measurement in the related art are solved, the angle sensors are susceptible to physical damage, regular replacement is very inconvenient, and because the coupling and decoupling process of the car train is very cumbersome, the angle sensor is also very inconvenient in actual use due to the inconvenience of coupling and decoupling.

[0054] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0056] Figure 1 A schematic diagram of a network architecture of a method for visually estimating an articulation angle of a vehicle or train according to an embodiment of the present application;

[0057] Figure 2 A flowchart of a method for visually estimating the articulation angle of a vehicle train provided according to an embodiment of the present application;

[0058] Figure 3 A schematic diagram of a framework for visual estimation of the articulation angle of a vehicle-train according to an embodiment of the present application;

[0059] Figure 4 A schematic diagram of camera installation angles according to an embodiment of the present application;

[0060] Figure 5 A schematic diagram of spatial posture transformation between adjacent cameras according to an embodiment of the present application;

[0061] Figure 6 Schematic diagram of the structure of a vehicle train articulation angle visual estimation device according to an embodiment of the present application;

[0062] Figure 7 Schematic diagram of the structure of an electronic device according to an embodiment of the present application.

[0063] Reference numerals:

[0064] 10-Car-train articulation angle visual estimation device: 100-acquisition module, 200-extraction module, 300-decoding module and 400-estimation module; 701-memory, 702-processor and 703-communication interface. DETAILED DESCRIPTION

[0065] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0066] The following describes the method and device for visually estimating the articulation angle of a car train based on deep learning according to an embodiment of the present application with reference to the accompanying drawings. In view of the problem that the angle sensor is used to measure the articulation angle in the related art mentioned in the above background technology, the angle sensor is susceptible to physical damage and is very inconvenient to replace regularly. In addition, because the coupling and decoupling process of the car train is very cumbersome, the angle sensor is also very inconvenient in actual use due to the inconvenience of coupling and decoupling. The present application provides a method for visually estimating the articulation angle of a car train based on deep learning. In this method, the features of the relative spatial posture between vehicle units can be extracted by acquiring the visual image captured by the panoramic system and the true value of the angle, and then the spatial posture information of the target is decoded from the encoded features by the posture decoder, and finally the rotation matrix of the spatial posture is decomposed into Euler angles to estimate the vehicle articulation angle of the car train. As a result, the vehicle articulation angle is estimated by using only the visual images and true angle values ​​captured by the panoramic surround view system installed on the car train, avoiding the risk of the angle sensor being susceptible to physical damage, reducing the risk of sensor failure caused by physical damage, providing hot standby redundancy for the angle sensor, improving the reliability and durability of the system, and not interfering with the coupling and decoupling process of the car train, reducing costs and simplifying the installation and maintenance process. As a result, the problems of using angle sensors for articulation angle measurement in related technologies, the angle sensors being susceptible to physical damage, and being inconvenient to replace regularly, and the angle sensors being inconvenient in actual use due to the inconvenience of coupling and decoupling, are solved.

[0067] Before explaining the method for visually estimating the articulation angle of a vehicle train in the embodiment of the present application, the system framework involved in the method for visually estimating the articulation angle of a vehicle train in the embodiment of the present application is first explained.

[0068] Figure 1 This is a schematic diagram of the network architecture of a method for visually estimating the articulation angle of a vehicle or train according to an embodiment of the present application. Figure 1 As shown, the network architecture of the method for visual estimation of the articulation angle of a car-train in the embodiment of the present application is mainly but not limited to consisting of five parts: network input, attitude encoder, single-layer ConvLSTM, attitude decoder and spatial attitude estimation.

[0069] Specifically, Figure 2 A flowchart of a method for visually estimating the articulation angle of a vehicle-train based on deep learning provided in an embodiment of the present application.

[0070] like Figure 2 As shown, the vehicle-train articulation angle visual estimation method based on deep learning includes the following steps:

[0071] In step S201, a visual image captured by a panoramic surround view system disposed on a train and a true value of an angle between two vehicle units of the train are obtained.

[0072] It can be understood that the automobile train here refers to a train composed of two or more rigid carriages connected by an articulated device, and the carriages are connected by a channel. The true value of the angle here can be understood as the true value of the articulation angle between the two vehicle units of the automobile train obtained by an angle sensor or other precision instruments in the automobile train, that is, the angle between the longitudinal axis of the tractor and the longitudinal axis of the trailer when the tractor and the trailer are connected by an articulated structure. This angle is of great significance in the driving process of the automobile train. It not only directly reflects the lateral stability of the automobile train, but is also closely related to the maneuverability, safety and reversing efficiency of the train.

[0073] In some embodiments, the size and change rate of the articulation angle can directly reflect the lateral stability of the vehicle train. When the articulation angle is too large, it may cause instability such as folding between the tractor and the trailer, thereby endangering driving safety. Under complex driving conditions such as reversing or turning, the change of the articulation angle will directly affect the handling performance of the vehicle train. A reasonable articulation angle helps the driver to better control the vehicle and improve driving stability. By monitoring the changes in the articulation angle in real time during driving, potential safety hazards can be discovered in a timely manner and corresponding measures can be taken to prevent them.

[0074] Here, the panoramic view system can be understood as an advanced driver assistance system that may be equipped on trains. The system can capture visual images through multiple cameras installed around the vehicle and stitch these images into a complete view of the vehicle's surroundings. This function is particularly important for trains because it can help drivers better understand the situation around the vehicle, especially when turning, changing lanes or parking.

[0075] Based on this, when visually estimating the articulation angle of a car train, the embodiment of the present application can first obtain the visual image captured by the panoramic view system installed on the car train and the true value of the angle between the two vehicle units of the car train, so as to utilize the effective information of the visual image and the true value of the angle to realize the visual estimation of the articulation angle of the car train.

[0076] For example, the embodiment of the present application can obtain the visual images captured by two cameras in the space of adjacent vehicles on the same side and the true value of the angle between two adjacent vehicle units on the same side. It should be noted that the visual images here are images captured by spatially adjacent cameras, rather than two temporally adjacent images captured by the same camera.

[0077] Step S202, construct a target neural network, calculate the true value of the spatial posture at the corresponding time point of the visual image, use the true value of the spatial posture to construct a target loss function, and use the target loss function to train the target neural network to obtain a trained neural network, wherein the true value of the spatial posture includes a rotation matrix and a translation vector.

[0078] Among them, the rotation matrix in the true value of the spatial attitude can be expressed as:

[0079]

[0080] in, is the rotation matrix in the true value of the spatial attitude at time t, represents the rotation matrix of the adjacent camera coordinate system around the x-axis by an angle of π / 2-γ. The rotation matrix represents the rotation of the vehicle's motion plane around the y-axis at time t, which transforms the camera coordinate system to the adjacent camera coordinate system. θ is the articulation angle. x R i (γ) represents the rotation matrix of the camera coordinate system by an angle of π / 2-γ, where

[0081]

[0082]

[0083] The translation vector in the true value of the spatial attitude can be expressed as:

[0084]

[0085] in, Represents the translation vector in the true value of the spatial attitude at time t, represents the rotation matrix of the adjacent camera coordinate system around the x-axis by an angle of π / 2-γ. The rotation matrix represents the rotation of the vehicle's motion plane around the y-axis at time t, which transforms the camera coordinate system to the adjacent camera coordinate system. θ is the articulation angle, t w represents the lateral displacement from one side of the same vehicle to the other side, t w =[0, 0, s w ] T ,s w represents the vehicle width, t l represents the longitudinal displacement from the camera installation point to the vehicle hinge point, t l =[s l , 0, 0] T ,s l Indicates the longitudinal lateral distance from the camera mounting point to the vehicle hinge point.

[0086] The network structure, input and output of the target neural network can be expressed as:

[0087]

[0088] h (t) =ConvLSTM(h (t-1) , c (t-1) , x (t) ),

[0089]

[0090] in, Camera C represents the adjacent camera coordinate system j The image captured at time t, The camera C represents the camera coordinate system i The image captured at time t, x (t) represents the feature map obtained by the input layer at time t, PEnc represents the posture encoder, ConvLSTM represents a single-layer ConvLSTM of the deep learning model, and h (t) represents the hidden state of a single-layer ConvLSTM at time t, h (t-1) represents the hidden state at the previous time, c (t-1) represents the unit state at the previous time, represents the first spatial pose estimation result from the camera coordinate system to the adjacent camera coordinate system at time t, and PDec represents the pose decoder.

[0091] The expression for training the target neural network using the target loss function can be expressed as:

[0092]

[0093]

[0094]

[0095]

[0096] in, represents the transposed result of the first spatial attitude estimation result, xRi(γ) represents the rotation matrix of the camera coordinate system rotated by an angle of π / 2-γ, Represents the transposed result of the rotation matrix rotating around the x-axis of the camera coordinate system by an angle of π / 2-γ. represents the estimated result of the spatial articulated posture, represents the estimated result of the first translation vector, represents the translation vector in the true value of the spatial attitude at time t, represents the rotation matrix of the adjacent camera coordinate system around the x-axis by an angle of π / 2-γ. represents the loss function at time t, Frob represents the Frobenius norm, λ represents the weight coefficient, I is the unit matrix, represents the target loss function, T represents the time length of the image sequence input to the network, represents the probability density corresponding to a certain hinge angle θ, t w =[0, 0, s w ] T ,s w represents the vehicle width, t l represents the longitudinal displacement from the camera installation point to the vehicle hinge point, t l =[s l , 0, 0] T ,s l represents the longitudinal lateral distance from the camera installation point to the vehicle hinge point, is the rotation matrix in the true value of the spatial attitude at time t.

[0097] In some embodiments, the present application ultimately estimates the vehicle articulation angle of the vehicle train according to the spatial posture of the vehicle train. In the process of obtaining the spatial posture, a deep learning neural network needs to be used.

[0098] Therefore, the present application can first construct a target neural network, and then construct a target loss function by calculating the true value of the angle at the corresponding time point of the visual image and the true value of the spatial posture, and then use the target loss function to train the target neural network, and then the trained neural network required for the spatial posture acquisition process can be obtained.

[0099] It should be noted that the use of the true value of the spatial posture to construct the target loss function and the use of the target loss function to train the target neural network in the embodiment of the present application is a cyclic process. It can be understood that after the target neural network is constructed, the true value of the spatial posture is input into the target neural network and output, and the loss function is calculated; then the true value of the spatial posture is input into the target neural network and output, and the loss function is calculated again, and the cycle is repeated many times until the training of the target neural network is completed.

[0100] Among them, the rotation matrix in the true value of the spatial attitude can be expressed as:

[0101]

[0102] in, is the rotation matrix in the true value of the spatial attitude at time t, represents the rotation matrix of the adjacent camera coordinate system around the x-axis by an angle of π / 2-γ. The rotation matrix represents the rotation of the vehicle's motion plane around the y-axis at time t, which transforms the camera coordinate system to the adjacent camera coordinate system. θ is the articulation angle. x R i (γ) represents the rotation matrix of the camera coordinate system by an angle of π / 2-γ, where

[0103]

[0104]

[0105] The translation vector in the true value of the spatial attitude can be expressed as:

[0106]

[0107] in, Represents the translation vector in the true value of the spatial attitude at time t, Represents the rotation matrix of the adjacent camera coordinate system rotating around the x-axis by an angle of π / 2-γ. The rotation matrix represents the rotation of the vehicle's motion plane around the y-axis at time t, which transforms the camera coordinate system to the adjacent camera coordinate system. θ is the articulation angle, t w represents the lateral displacement from one side of the same vehicle to the other side, t w =[0, 0, s w ] T ,s w represents the vehicle width, t l represents the longitudinal displacement from the camera installation point to the vehicle hinge point, t l =[s l , 0, 0] T ,s l Indicates the longitudinal lateral distance from the camera mounting point to the vehicle hinge point.

[0108] The network structure, input and output of the target neural network can be expressed as:

[0109]

[0110] h (t) =ConvLSTM(h (t-1) , c (t-1) , x (t) ),

[0111]

[0112] in, Camera C represents the adjacent camera coordinate system j The image captured at time t, The camera C represents the camera coordinate systemi The image captured at time t, x (t) represents the feature map obtained by the input layer at time t, PEnc represents the posture encoder, ConvLSTM represents a single-layer ConvLSTM of the deep learning model, and h (t) represents the hidden state of a single-layer ConvLSTM at time t, h (t-1) represents the hidden state at the previous time, c (t-1) represents the unit state at the previous time, represents the first spatial pose estimation result from the camera coordinate system to the adjacent camera coordinate system at time t, and PDec represents the pose decoder.

[0113] Furthermore, the expression for training the target neural network using the target loss function can be expressed as:

[0114]

[0115]

[0116]

[0117]

[0118] in, represents the transposed result of the first spatial attitude estimation result, xRi(γ) represents the rotation matrix of the camera coordinate system rotated by an angle of π / 2-γ, Represents the transposed result of the rotation matrix rotating around the x-axis of the camera coordinate system by an angle of π / 2-γ. represents the estimated result of the spatial articulated posture, represents the estimated result of the first translation vector, represents the translation vector in the true value of the spatial attitude at time t, represents the rotation matrix of the adjacent camera coordinate system around the x-axis by an angle of π / 2-γ. represents the loss function at time t, Frob represents the Frobenius norm, λ represents the weight coefficient, I is the unit matrix, represents the target loss function, T represents the time length of the image sequence input to the network, represents the probability density corresponding to a certain hinge angle θ, t w =[0, 0, s w ] T ,s w represents the vehicle width, t l represents the longitudinal displacement from the camera installation point to the vehicle hinge point, t l =[s l , 0, 0] T ,sl represents the longitudinal lateral distance from the camera installation point to the vehicle hinge point, is the rotation matrix in the true value of the spatial attitude at time t.

[0119] It should be noted that step S202 in the embodiment of the present application is mainly to train the target neural network, so the first spatial posture estimation result here can be understood as an estimation result of a spatial posture obtained (output) by any round of training in the training process of "target loss function training target neural network".

[0120] It should also be noted that after the target neural network training is completed, the subsequent processing steps are to actually apply the trained target neural network. Given that the training process and the actual application process actually use the same target neural network, the output result here in the subsequent processing process, i.e., the first spatial posture estimation result, is no longer understood as the spatial posture estimation result in the training process, but can be understood as the spatial posture obtained (output) in the actual application process that can be actually used. In short, the input and output in the actual application process are changed accordingly to the input and output in the training process.

[0121] Optionally, in one embodiment of the present application, before calculating the true value of the angle at the spatial posture true value at the corresponding time point of the visual image, it also includes: obtaining a first angle and a second angle between two vehicle units of the car train, wherein the first angle is a side where the angle between the two vehicle units is less than 180°, and the second angle is a side where the angle between the two vehicle units is greater than 180°; based on the first angle and the second angle, determining the true value of the angle.

[0122] It is understandable that cameras are usually provided on the left and right sides of the automobile train, respectively, so there are two left and right camera pairs, the fields of view of the two camera pairs have overlapping areas of different ranges, and both can measure the same articulation angle. In order to ensure that the two camera pairs do not get the same spatial posture annotation, the spatial posture true value of the angle real value at the corresponding time point of the visual image can be effectively calculated. The embodiment of the present application can obtain the first angle and the second angle between the two vehicle units of the automobile train, where the first angle is set to a true value that maintains the original value, and the second angle is set to an angle value with an opposite sign. Among them, the first angle is the side where the angle between the two vehicle units is less than 180°, and the second angle is the side where the angle between the two vehicle units is greater than 180°; based on the first angle and the second angle, the true value of the angle is determined,

[0123] For example, Figure 3 FIG. 1 is a schematic diagram of the spatial posture transformation between adjacent cameras according to an embodiment of the present application. Figure 3As shown, according to the size of the angle between adjacent vehicle units of a multi-articulated vehicle, the present application can assign a positive value to the side where the articulation angle is less than 180°, and assign a negative value to the side where the articulation angle is greater than 180°, thereby ensuring that each camera pair has its own unique labeling, which helps to distinguish and correct the articulation angles estimated by different camera pairs, thereby avoiding confusion and ensuring that the calculations provided by each camera are independent and accurate, thereby improving the accuracy of the articulation angle estimation.

[0124] Optionally, in one embodiment of the present application, the spatial posture true value of the angle true value at the corresponding time point of the visual image is calculated, including: compensating for the installation angle of the panoramic view system, rotating the acquisition plane of the panoramic view system to the movement plane of the car train, so as to calculate the spatial posture true value of the angle true value at the corresponding time point of the visual image.

[0125] In some embodiments, the panoramic surround view system collects video of the vehicle's surroundings by installing multiple wide-angle cameras (e.g., 4 to 8) around the vehicle train. The video images collected by these cameras are processed by distortion correction, perspective transformation, and panoramic stitching and fusion to finally generate a panoramic bird's-eye view of the vehicle's surroundings, which is displayed in real time on the vehicle display.

[0126] For example, the visual images captured by the panoramic view system usually include the following information: in front of the vehicle: showing the road and traffic conditions ahead; behind the vehicle: helping the driver observe the rear environment when reversing; both sides of the vehicle: especially when turning, the side roads and obstacles can be clearly displayed.

[0127] However, in actual scenarios, due to the long body and complex structure of trains, drivers may have large visual blind spots during driving, such as the connection between carriages, the side and rear of the vehicle, and the front of the vehicle. The panoramic surround view system captures images around the vehicle through multiple cameras and stitches them into a panoramic image to eliminate these blind spots. However, if the camera is installed at a wrong angle and does not match the vehicle's motion plane, the panoramic image may be distorted or misaligned, and thus cannot accurately reflect the actual situation around the vehicle.

[0128] Therefore, the embodiment of the present application can make certain compensation for the installation angle of the panoramic view system to make the image information provided by the panoramic view system more accurate and comprehensive, thereby improving the effectiveness of the calculation results of the spatial posture true value of the angle true value at the corresponding time point of the visual image.

[0129] For example, you can first install the cameras in the front, back, left, and right positions of the vehicle according to the requirements of the surround view system. Pay attention to the installation height and angle of the camera to ensure that it can capture the key areas around the vehicle. Then start the surround view system and enter the calibration mode. At this time, a virtual 360-degree surround image is usually displayed on the screen. Follow the system prompts to make preliminary adjustments to the camera installation angle so that it is roughly facing the key areas around the vehicle.

[0130] Next, perform preliminary calibration and compensation; finally, based on the preliminary calibration, perform manual calibration and fine-tuning, and pay attention to the distortion and dislocation in the picture in real time, and make adjustments one by one until the picture is clear and accurate. This completes the installation angle compensation of the panoramic view system.

[0131] The embodiment of the present application can compensate for the installation angle of the panoramic view system so that the acquisition plane of the panoramic view system can be rotated to the movement plane of the car or train, thereby effectively improving the reliability of the visual image captured by the panoramic view system, and further improving the accuracy of the calculation result of the spatial posture true value of the angle true value at the corresponding time point of the visual image.

[0132] Step S203, input the visual image into the trained neural network, extract the features of the relative spatial posture between the two vehicle units of the car train, compensate the time dependency of the features through ConvLSTM, obtain the encoded features, and decode the spatial posture of the target from the encoded features.

[0133] It can be understood by those skilled in the art that time dependence here can be understood as the dynamic changes of the characteristics of the relative spatial posture between the two vehicle units over time.

[0134] As a possible implementation method, the embodiment of the present application can input the obtained visual image into the trained target neural network to extract the features of the relative spatial posture between the two vehicle units of the automobile train. Then, the time dependency of the features is compensated by ConvLSTM to obtain the encoded features, so as to decode the spatial posture of the target from the encoded features for the final calculation of the vehicle articulation angle of the automobile train.

[0135] For example, after obtaining the visual images captured by the panoramic surround view system, such as the images captured by the cameras of two vehicle units on the same side that are adjacent in space, the visual images captured by the adjacent vehicle units are input into the trained neural network, and then the posture encoder is used to extract the features for estimating the relative spatial posture between the two vehicle units of the train. Among them, the features of the relative spatial posture between the two vehicle units may include, but are not limited to, the relative position between the two vehicle units (the straight-line distance and direction between the two vehicle units), the relative direction (the angle between the two vehicle unit heads (or designated reference points)), the relative rotation (the relative rotation between the two vehicle units), etc.

[0136] Furthermore, after obtaining the characteristics of the relative spatial posture between the two vehicle units, considering that the vehicle is in motion, the embodiment of the present application can compensate for the time dependence of the characteristics of the relative spatial posture between the two vehicle units to ensure that two angle values ​​adjacent in time will not change suddenly.

[0137] For example, assume that the vehicle's movement occurs on a two-dimensional plane parallel to the ground, that is, no vertical movement is involved. At this time, the vehicle's z-axis is perpendicular to the ground and points upward, that is, the vehicle's movement is performed on the horizontal plane, and all rotation transformations are performed on this horizontal plane. At this time, the process of compensating for time dependence of a single-layer ConvLSTM can be expressed in the same way as the input formula of the target neural network:

[0138] h (t) =ConvLSTM(h (t-1) , c (t-1) , x (t) )

[0139] Among them, x (t) represents the image data captured by the input layer at time t, h (t) represents the state of a single-layer ConvLSTM at time t. ConvLSTM represents a single-layer ConvLSTM of a deep learning model that combines the characteristics of a convolutional neural network (CNN) and a long short-term memory network (LSTM). h (t-1) Indicates the state at the previous time, c (t-1) Indicates the state of the unit at the previous time.

[0140] It should be noted that the ConvLSTM (convolutional long short-term memory network) in the embodiment of the present application can be understood as a deep learning model that combines the characteristics of a convolutional neural network (CNN) and a long short-term memory network (LSTM), which is suitable for processing image data (image sequences) with time series characteristics, and can learn spatial and temporal dependencies in image sequences. In some complex network architectures, ConvLSTM may be stacked by multiple levels. In the embodiment of the present application, ConvLSTM consists of one layer, i.e., a single-layer ConvLSTM, which means that the data in the embodiment of the present application can be processed by a layer of ConvLSTM units.

[0141] Furthermore, the single-layer ConvLSTM in the embodiment of the present application can also process the input of the current time step while memorizing the previous input time step, thereby updating the hidden state. This process can capture the temporal dependency in the visual image sequence. Then, the PDec attitude decoder can be used to decode the processed features to obtain the spatial attitude of the target. Among them, the target here can be understood as each vehicle unit that needs to perform visual estimation of the articulation angle.

[0142] Step S204 , decomposing the rotation matrix of the spatial posture into Euler angles to estimate the vehicle articulation angle of the vehicle train.

[0143] As a possible implementation method, after obtaining the spatial posture of the vehicle unit that needs to perform visual estimation of the articulation angle, the embodiment of the present application can perform a spatial posture transformation according to the spatial posture, obtain the rotation matrix of the spatial posture, and then decompose the rotation matrix into Euler angles to obtain the articulation angle between the vehicles.

[0144] Specifically, decomposing the rotation matrix of the spatial posture into Euler angles can be understood as expressing the three-dimensional rotation as three rotations about different coordinate axes. The yaw angle along the z-axis in the decomposed Euler angle is the final vehicle articulation angle of the multi-car train.

[0145] It should be noted that in the embodiment of the present application, since the estimated spatial posture transformation is in the form of a rotation matrix (Using the rotation matrix obtained by the trained target neural network) indicates that in order to recover the articulation angle, the rotation matrix needs to be decomposed into Euler angles to obtain the articulation angle, and because the articulation angle of the vehicle is usually between -36° and 36° in the physical constraints of real-time operation, there will be no problem of gimbal lock when the Euler angle is ±90°. Therefore, the general 3×3 rotation matrix can be decomposed into sequential rotations around each axis, and the roll angle φ, yaw angle θ and pitch angle ψ are obtained respectively, where the yaw angle is the estimated articulation angle.

[0146] For example, Figure 4 This is a schematic diagram of a framework for visual estimation of the articulation angle of a vehicle train according to an embodiment of the present application. Figure 5 This is a schematic diagram of the camera installation angle of an embodiment of the present application. Figure 4 and Figure 5 As shown, in the calculation of the articulation angle of the automobile train, the embodiment of the present application can be implemented but is not limited to using eight fisheye cameras with a field of view of 90° (indicated by C0~C7), γ is the camera installation angle, and the arrow indicates the z-axis of the camera.

[0147] Further, in the process of achieving surrounding environment perception, these eight 190° field of view fisheye cameras can create an omnidirectional, multi-view camera array, and the mature application of panoramic camera systems can be found in the panoramic view application of passenger cars to help drivers with near-field blind spot perception.

[0148] Among them, the C0 camera is installed at the front of the vehicle, the C7 camera is installed at the rear of the vehicle, and C1 to C6 are installed on the side of the vehicle to capture the overlapping relationship between adjacent vehicles. The overlap between the cameras provides spatial transformation information, which can further estimate the articulation angle.

[0149] Since the six cameras on the side of the vehicle can form four camera pairs, two articulation angles can be estimated by four camera pairs. For example, θ∈{θ f ,θ r}, when θ=θ f When θ=θ r When , ji∈{C6&C4, C3&C5}. Among them, camera pairs C4&C2 and C1&C3 are responsible for estimating the front articulation angle θ f , while camera pairs C6&C4, C3&C5 are responsible for estimating the rear articulation angle θ f .

[0150] According to the method for visual estimation of the articulation angle of a car train based on deep learning proposed in the embodiment of the present application, the features of the relative spatial posture between the vehicle units can be extracted by acquiring the visual image and the true angle value captured by the panoramic system, and then the spatial posture information of the target is decoded from the encoded features by the posture decoder, and finally the rotation matrix of the spatial posture is decomposed into Euler angles to estimate the vehicle articulation angle of the car train. Thus, the vehicle articulation angle is estimated by only using the visual image and the true angle value captured by the panoramic surround system installed on the car train, avoiding the risk of the angle sensor being susceptible to physical damage, reducing the risk of sensor failure caused by physical damage, providing hot standby redundancy for the angle sensor, improving the reliability and durability of the system, and not interfering with the coupling and decoupling process of the car train, reducing costs and simplifying the installation and maintenance process. Thus, the problems of using angle sensors for articulation angle measurement in the related art, the angle sensors being susceptible to physical damage, and regular replacement being very inconvenient, and the angle sensors being very inconvenient in actual use due to the inconvenience of coupling and decoupling because the coupling and decoupling process of the car train is very cumbersome are solved.

[0151] Next, a vehicle-train articulation angle visual estimation device based on deep learning proposed in accordance with an embodiment of the present application is described with reference to the accompanying drawings.

[0152] Figure 6 It is a structural schematic diagram of a vehicle-train articulation angle visual estimation device based on deep learning in an embodiment of the present application.

[0153] like Figure 6 As shown, the vehicle-train articulation angle visual estimation device 10 based on deep learning includes: an acquisition module 100, a calculation module 200, a processing module 300 and an estimation module 400.

[0154] The acquisition module 100 is used to acquire the visual image captured by the panoramic surround view system installed on the train and the true value of the angle between two vehicle units of the train.

[0155] The calculation module 200 is used to construct a target neural network, calculate the true value of the spatial posture at the corresponding time point of the visual image, use the true value of the spatial posture to construct a target loss function, and use the target loss function to train the target neural network to obtain a trained neural network.

[0156] The processing module 300 is used to input the visual image into the trained neural network, extract the features of the relative spatial posture between the two vehicle units of the car train, compensate the time dependency of the features through ConvLSTM, obtain the encoded features, and decode the spatial posture of the target from the encoded features.

[0157] The estimation module 400 is used to decompose the rotation matrix of the spatial posture into Euler angles to estimate the vehicle articulation angle of the vehicle train.

[0158] Optionally, in one embodiment of the present application, it also includes: an acquisition module and a determination module.

[0159] The acquisition module is used to obtain the first angle and the second angle between the two vehicle units of the vehicle train before calculating the true value of the angle at the time point corresponding to the visual image, wherein the first angle is the side where the angle between the two vehicle units is less than 180°, and the second angle is the side where the angle between the two vehicle units is greater than 180°.

[0160] The determination module is used to determine the true value of the angle based on the first angle and the second angle.

[0161] Optionally, in one embodiment of the present application, the computing module 200 includes: a compensation unit for compensating for the installation angle of the panoramic view system so that the acquisition plane of the panoramic view system rotates to the movement plane of the car train to calculate the spatial posture true value of the angle true value at the corresponding time point of the visual image.

[0162] Optionally, in one embodiment of the present application, the rotation matrix in the true value of the spatial attitude can be expressed as:

[0163]

[0164] in, is the rotation matrix in the true value of the spatial attitude at time t, represents the rotation matrix of the adjacent camera coordinate system around the x-axis by an angle of π / 2-γ. The rotation matrix represents the rotation of the vehicle's motion plane around the y-axis at time t, which transforms the camera coordinate system to the adjacent camera coordinate system. θ is the articulation angle. x R i (γ) represents the rotation matrix of the camera coordinate system by an angle of π / 2-γ, where

[0165]

[0166]

[0167] Optionally, in one embodiment of the present application, the translation vector in the true value of the spatial posture can be expressed as:

[0168]

[0169] in, Represents the translation vector in the true value of the spatial attitude at time t, Represents the rotation matrix of the adjacent camera coordinate system rotating around the x-axis by an angle of π / 2-γ. The rotation matrix represents the rotation of the vehicle's motion plane around the y-axis at time t, which transforms the camera coordinate system to the adjacent camera coordinate system. θ is the articulation angle, t w represents the lateral displacement from one side of the same vehicle to the other side, t w =[0, 0, s w ] T ,s w represents the vehicle width, t l represents the longitudinal displacement from the camera installation point to the vehicle hinge point, t l =[s l , 0, 0] T ,s l Indicates the longitudinal lateral distance from the camera mounting point to the vehicle hinge point.

[0170] Optionally, in one embodiment of the present application, the network structure, input and output of the target neural network can be expressed as:

[0171]

[0172] h (t) =ConvLSTM(h (t-1) , c (t-1) , x (t) ),

[0173]

[0174] in, Camera C represents the adjacent camera coordinate system j The image captured at time t, The camera C represents the camera coordinate system i The image captured at time t, x (t) represents the feature map obtained by the input layer at time t, PEnc represents the posture encoder, ConvLSTM represents a single-layer ConvLSTM of the deep learning model, and h (t) represents the hidden state of a single-layer ConvLSTM at time t, h (t-1) represents the hidden state at the previous time, c (t-1) represents the unit state at the previous time, represents the first spatial pose estimation result from the camera coordinate system to the adjacent camera coordinate system at time t, and PDec represents the pose decoder.

[0175] Optionally, in one embodiment of the present application, the expression for training the target neural network using the target loss function can be expressed as:

[0176]

[0177]

[0178]

[0179]

[0180] in, represents the transposed result of the first spatial attitude estimation result, xRi(γ) represents the rotation matrix of the camera coordinate system rotated by an angle of π / 2-γ, Represents the transposed result of the rotation matrix rotating around the x-axis of the camera coordinate system by an angle of π / 2-γ. represents the estimated result of the spatial articulated posture, represents the estimated result of the first translation vector, represents the translation vector in the true value of the spatial attitude at time t, represents the rotation matrix of the adjacent camera coordinate system around the x-axis by an angle of π / 2-γ. represents the loss function at time t, Frob represents the Frobenius norm, λ represents the weight coefficient, I is the unit matrix, represents the target loss function, T represents the time length of the image sequence input to the network, represents the probability density corresponding to a certain hinge angle θ, t w =[0, 0, s w ] T ,s w represents the vehicle width, t l represents the longitudinal displacement from the camera installation point to the vehicle hinge point, t l =[s l , 0, 0] T ,s l represents the longitudinal lateral distance from the camera installation point to the vehicle hinge point, is the rotation matrix in the true value of the spatial attitude at time t.

[0181] It should be noted that the aforementioned explanation of the embodiment of the method for visual estimation of the articulation angle of a vehicle train based on deep learning is also applicable to the device for visual estimation of the articulation angle of a vehicle train based on deep learning of this embodiment, and will not be repeated here.

[0182] According to the deep learning-based visual estimation device for the articulation angle of a car train proposed in the embodiment of the present application, the features of the relative spatial posture between the vehicle units can be extracted by acquiring the visual image and the true angle value captured by the panoramic system, and then the spatial posture information of the target is decoded from the encoded features by the posture decoder, and finally the rotation matrix of the spatial posture is decomposed into Euler angles to estimate the vehicle articulation angle of the car train. Thus, the vehicle articulation angle is estimated by only using the visual image and the true angle value captured by the panoramic surround system installed on the car train, avoiding the risk of the angle sensor being susceptible to physical damage, reducing the risk of sensor failure caused by physical damage, providing hot standby redundancy for the angle sensor, improving the reliability and durability of the system, and not interfering with the coupling and decoupling process of the car train, reducing costs and simplifying the installation and maintenance process. Thus, the problems of using mechanical angle sensors for articulation angle measurement in the related art, the angle sensors being susceptible to physical damage, and regular replacement being very inconvenient, and the angle sensors being very inconvenient in actual use due to the inconvenience of coupling and decoupling because the coupling and decoupling process of the car train is very cumbersome are solved.

[0183] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0184] A memory 701 , a processor 702 , and a computer program stored in the memory 701 and executable on the processor 702 .

[0185] When the processor 702 executes the program, the vehicle-train articulation angle visual estimation method based on deep learning provided in the above embodiment is implemented.

[0186] Furthermore, the electronic device further comprises:

[0187] The communication interface 703 is used for communication between the memory 701 and the processor 702 .

[0188] The memory 701 is used to store computer programs that can be executed on the processor 702 .

[0189] The memory 701 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0190] If the memory 701, the processor 702 and the communication interface 703 are implemented independently, the communication interface 703, the memory 701 and the processor 702 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0191] Optionally, in a specific implementation, if the memory 701, the processor 702 and the communication interface 703 are integrated on a chip, the memory 701, the processor 702 and the communication interface 703 can communicate with each other through an internal interface.

[0192] The processor 702 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0193] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned deep learning-based visual estimation method for the articulation angle of a vehicle train.

[0194] An embodiment of the present application also provides a computer program product, including a computer program, which can run computer instructions. When the computer instructions are executed by a processor, the deep learning-based visual estimation method for the articulation angle of a vehicle train provided in an embodiment of the present application is implemented.

[0195] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0196] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0197] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0198] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0199] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of multiple of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0200] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0201] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0202] The storage medium mentioned above may be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for visual estimation of articulation angle of automobile and train based on deep learning, characterized in that: The following steps are involved: Acquire a visual image captured by a panoramic surround view system disposed on a motor vehicle train and a true value of an angle between two vehicle units of the motor vehicle train; Constructing a target neural network, calculating a spatial posture true value of the angle true value at a corresponding time point of the visual image, using the spatial posture true value to construct a target loss function, and using the target loss function to train the target neural network to obtain a trained neural network, wherein the spatial posture true value includes a rotation matrix and a translation vector; Inputting the visual image into the trained neural network, extracting features of the relative spatial posture between the two vehicle units of the vehicle train, compensating the time dependency of the features through ConvLSTM, obtaining encoded features, and decoding the spatial posture of the target from the encoded features; The rotation matrix of the spatial posture is decomposed into Euler angles to estimate the vehicle articulation angle of the vehicle train.

2. The method according to claim 1, characterized in that Before calculating the spatial posture true value of the angle true value at the time point corresponding to the visual image, the method further includes: Acquire a first angle and a second angle between two vehicle units of the vehicle train, wherein the first angle is a side where the angle between the two vehicle units is less than 180°, and the second angle is a side where the angle between the two vehicle units is greater than 180°; The true value of the angle is determined based on the first angle and the second angle.

3. The method according to claim 1, characterized in that The step of calculating the spatial posture true value of the angle true value at the corresponding time point of the visual image includes: The installation angle of the panoramic view system is compensated so that the acquisition plane of the panoramic view system is rotated to the movement plane of the vehicle train, so as to calculate the spatial posture true value of the angle true value at the corresponding time point of the visual image.

4. The method according to claim 1, characterized in that The rotation matrix in the true value of the spatial attitude is: in, is the rotation matrix of the true value of the spatial posture at time t, represents the rotation matrix of the adjacent camera coordinate system around the x-axis by an angle of π / 2-γ. The rotation matrix represents the rotation of the vehicle's motion plane around the y-axis at time t, which transforms the camera coordinate system to the adjacent camera coordinate system. θ is the articulation angle. x R i (γ) represents the rotation matrix of the camera coordinate system by an angle of π / 2-γ, where 5. The method according to claim 1, characterized in that The translation vector in the true value of the spatial attitude is: in, represents the translation vector in the true value of the spatial posture at time t, represents the rotation matrix of the adjacent camera coordinate system around the x-axis by an angle of π / 2-γ. The rotation matrix represents the rotation of the vehicle's motion plane around the y-axis at time t, which transforms the camera coordinate system to the adjacent camera coordinate system. θ is the articulation angle, t w represents the lateral displacement from one side of the same vehicle to the other side, t w =[0,0,s w ] T ,s w represents the vehicle width, t l represents the longitudinal displacement from the camera installation point to the vehicle hinge point, t l =[s l ,0,0] T ,s l Indicates the longitudinal lateral distance from the camera mounting point to the vehicle hinge point.

6. According to the method of claim 1, the network structure, input and output of the target neural network are respectively: h (t) =ConvLSTM(h (t-1) ,c (t-1) ,x (t) ), in, Camera C represents the adjacent camera coordinate system j The image captured at time t, The camera C represents the camera coordinate system i The image captured at time t, x (t) represents the feature map obtained by the input layer at time t, PEnc represents the posture encoder, ConvLSTM represents a single-layer ConvLSTM of the deep learning model, and h (t) represents the hidden state of a single-layer ConvLSTM at time t, h (t-1) represents the hidden state at the previous time, c (t-1) represents the unit state at the previous time, represents the first spatial pose estimation result from the camera coordinate system to the adjacent camera coordinate system at time t, and PDec represents the pose decoder.

7. According to the method of claim 6, the expression for training the target neural network using the target loss function is: in, represents the transposed result of the first spatial attitude estimation result, x R i (γ) represents the rotation matrix of the camera coordinate system by an angle of π / 2-γ. Represents the transposed result of the rotation matrix rotating around the x-axis of the camera coordinate system by an angle of π / 2-γ. represents the estimated result of the spatial articulated posture, represents the estimated result of the first translation vector, represents the translation vector in the true value of the spatial attitude at time t, represents the rotation matrix of the adjacent camera coordinate system around the x-axis by an angle of π / 2-γ. represents the loss function at time t, Frob represents the Frobenius norm, λ represents the weight coefficient, I is the unit matrix, represents the target loss function, T represents the time length of the image sequence input by the network, represents the probability density corresponding to a certain hinge angle θ, t w =[0,0,s w ] T ,s w represents the vehicle width, t l represents the longitudinal displacement from the camera installation point to the vehicle hinge point, t l =[s l ,0,0] T ,s l represents the longitudinal lateral distance from the camera installation point to the vehicle hinge point, is the rotation matrix of the true value of the spatial posture at time t.

8. A vehicle-train articulation angle visual estimation device based on deep learning, characterized in that: include: An acquisition module, used to acquire a visual image captured by a panoramic surround view system arranged on a motor train and a true value of an angle between two vehicle units of the motor train; A calculation module, used to construct a target neural network, calculate the spatial posture true value of the angle true value at the corresponding time point of the visual image, use the spatial posture true value to construct a target loss function, and use the target loss function to train the target neural network to obtain a trained neural network; A decoding module, for inputting the visual image into the trained neural network, extracting features of the relative spatial posture between the two vehicle units of the vehicle train, compensating the time dependency of the features through ConvLSTM, obtaining encoded features, and decoding the spatial posture of the target from the encoded features; The estimation module is used to decompose the rotation matrix of the spatial posture into Euler angles to estimate the vehicle articulation angle of the vehicle train.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the deep learning-based visual estimation method for the articulation angle of a vehicle train as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the deep learning-based visual estimation method for the articulation angle of a vehicle-train as described in any one of claims 1-7.

Citation Information

Cited By

  • Hinge headstock hanging included angle visual estimation method and device

    CN121544684A