Vehicle tire status identification method, device, equipment, vehicle, medium and product

Abstract

The embodiments of the present invention provide a vehicle tire status recognition method, device, equipment, vehicle, medium and product, which are applied to the field of vehicle technology. The method includes: obtaining tire sensor data and vehicle external environment data of the current vehicle, and determining the tire status of the current vehicle according to the tire sensor data and vehicle external environment data. Through the embodiments of the present invention, the vehicle tire status recognition is realized by combining tire sensor data and vehicle external environment, which improves accuracy and robustness.

Description

Technical Field

[0001] The present invention relates to the field of vehicle technology, and in particular to a vehicle tire status recognition method, device, equipment, vehicle, medium and product. Background Art

[0002] At present, tire blowouts pose a serious threat to driving safety. Tire blowouts usually occur in a very short time, causing the vehicle to lose balance and possibly yaw or roll over. In order to reduce the incidence of tire blowout accidents, it is necessary to warn and identify vehicle tire blowouts.

[0003] In the prior art, tire pressure and temperature of the wheels are detected, and then tire blowout is identified by setting thresholds. This method has low accuracy and robustness. Summary of the invention

[0004] In view of the above problems, a vehicle tire status identification method, device, equipment, vehicle, medium and product are proposed to overcome the above problems or at least partially solve the above problems, including:

[0005] A vehicle tire status recognition method, the method comprising:

[0006] Obtain the tire sensor data and external environment data of the current vehicle;

[0007] The current tire status of the vehicle is determined according to the tire sensor data and the vehicle exterior environment data.

[0008] Optionally, determining the tire status of the current vehicle according to the tire sensor data and the vehicle exterior environment data includes:

[0009] Converting the tire sensor data into first image data;

[0010] Determine second image data according to the vehicle exterior environment data;

[0011] The tire status of the current vehicle is determined according to the first image data and the second image data.

[0012] Optionally, the first image data is a Gram angle field map, and the converting the tire sensor data into the first image data includes:

[0013] Determining time series data from the tire sensor data;

[0014] Performing coordinate conversion on the time series data to obtain polar coordinate data;

[0015] A Gram angle field diagram is generated according to the polar coordinate data to obtain first image data.

[0016] Optionally, the performing coordinate conversion on the time series data to obtain polar coordinate data includes:

[0017] Encoding the parameter value of the time series data to obtain an angle value on polar coordinates, and encoding the time value of the time series data to obtain a radius value on polar coordinates;

[0018] Polar coordinate data is obtained according to the angle value and the radius value.

[0019] Optionally, before performing coordinate conversion on the time series data to obtain polar coordinate data, the method further includes:

[0020] The time series data is normalized.

[0021] Optionally, after the normalization processing is performed on the time series data, the method further includes:

[0022] When there are multiple channels of time series data, the multiple channels of time series data are merged into one channel of time series data.

[0023] Optionally, determining the tire status of the current vehicle according to the first image data and the second image data includes:

[0024] Obtaining a first bird's-eye view feature map according to the first image data;

[0025] Obtaining a second bird's-eye view feature map according to the second image data;

[0026] Fusing the first bird's-eye view feature map and the second bird's-eye view feature map to obtain a fused bird's-eye view feature map;

[0027] The fused bird's-eye view feature image is used to perform tire blowout identification to obtain the tire status of the current vehicle.

[0028] Optionally, obtaining a first bird's-eye view feature map according to the first image data includes:

[0029] Extracting a first two-dimensional feature map from the first image data, and converting the first two-dimensional feature map into a first bird's-eye view feature map;

[0030] The step of obtaining a second bird's-eye view feature map according to the second image data comprises:

[0031] A second two-dimensional feature map is extracted from the second image data, and the second two-dimensional feature map is converted into a second bird's-eye view feature map.

[0032] Optionally, extracting a first two-dimensional feature map from the first image data comprises:

[0033] extracting a first two-dimensional feature map from the first image data by a first single-mode encoder;

[0034] The step of extracting a second two-dimensional feature map from the second image data comprises:

[0035] A second two-dimensional feature map is extracted from the second image data by a second single-modal encoder.

[0036] Optionally, converting the first two-dimensional feature map into a first bird's-eye view feature map includes:

[0037] According to preset size parameters, converting the first two-dimensional feature map into a first bird's-eye view feature map;

[0038] The converting the second two-dimensional feature map into a second bird's-eye view feature map comprises:

[0039] The external parameters of the camera are obtained, and the second two-dimensional feature map is converted into a second bird's-eye view feature map by combining the external parameters of the camera and preset size parameters.

[0040] Optionally, before converting the second two-dimensional feature map into a second bird's-eye view feature map, the method further includes:

[0041] Determining a distortion parameter of the second two-dimensional feature map;

[0042] Determine the two-dimensional image space coordinates of the lower boundary of the world coordinate space according to the distortion parameters, and determine the upper limit value and the lower limit value of the coordinates in the two-dimensional image space according to the two-dimensional image space coordinates;

[0043] The second two-dimensional feature map is cropped according to the coordinate upper limit value and the coordinate lower limit value in the two-dimensional image space.

[0044] Optionally, fusing the first bird's-eye view feature map and the second bird's-eye view feature map to obtain a fused bird's-eye view feature map includes:

[0045] Splicing the first bird's-eye view feature map and the second bird's-eye view feature map;

[0046] During the stitching process, a target expansion value is obtained, and the first bird's-eye view feature map and the second bird's-eye view feature map are expanded using the target expansion value to obtain a fused bird's-eye view feature map.

[0047] Optionally, the performing tire blowout identification on the fused bird's-eye view feature image to obtain the tire status of the current vehicle includes:

[0048] A two-stage multi-scale classification prediction decoder is used to identify tire blowouts on the fused bird's-eye view feature image to obtain the tire status of the current vehicle.

[0049] Optionally, it also includes:

[0050] The road information is determined according to the vehicle external environment data, and when it is determined according to the road information that there is a risk of tire blowout, a prompt message indicating that the current vehicle has a risk of tire blowout is presented.

[0051] Optionally, it also includes:

[0052] When the first tire blowout warning message sent by other vehicles to the current vehicle is received, a prompt message indicating that the current vehicle has a tire blowout risk is presented.

[0053] Optionally, the vehicle external environment data includes tire surface data of other vehicles, and further includes:

[0054] The tire surface information of other vehicles is determined according to the vehicle external environment data, and when it is determined that there is a risk of tire blowout according to the tire surface information, a second tire blowout warning message is sent to the other vehicles.

[0055] Optionally, it also includes:

[0056] When the tire state is a flat tire state, a prompt message indicating that the current vehicle is in a flat tire state is presented.

[0057] A vehicle tire status recognition device, the device is used to implement the method as described above.

[0058] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method described above when executed by the processor.

[0059] A vehicle, comprising the device as described above, or the electronic device as described above.

[0060] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0061] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the computer program implements the method as described above.

[0062] The embodiments of the present invention have the following advantages:

[0063] In an embodiment of the present invention, by acquiring tire sensor data and external environment data of the current vehicle, the tire status of the current vehicle is determined based on the tire sensor data and external environment data, thereby realizing vehicle tire status identification by combining tire sensor data and external environment, thereby improving accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0065] Figure 1 is a flowchart of a method for identifying a vehicle tire state provided by some embodiments of the present invention;

[0066] Figure 2 is a schematic diagram of a time series data conversion provided by some embodiments of the present invention;

[0067] Figure 3 is a schematic diagram of a network architecture provided by some embodiments of the present invention;

[0068] Figure 4 It is a flowchart of the steps of another vehicle tire status identification control method provided by some embodiments of the present invention. DETAILED DESCRIPTION

[0069] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] Reference Figure 1 , shows a flowchart of a method for identifying a vehicle tire state provided by some embodiments of the present invention, which may specifically include the following steps:

[0071] Step 101, obtaining tire sensor data and external environment data of the current vehicle.

[0072] As an example, tire sensor data may include tire pressure data and tire temperature data, and external vehicle environment data may include image data collected by an on-board camera and point cloud data collected by an on-board laser radar.

[0073] In practical applications, multiple sensors can be deployed inside and outside the vehicle to collect tire sensing data through in-vehicle sensors set for vehicle tires, and to collect external environment data through external sensors.

[0074] Step 102: determining the current tire status of the vehicle according to the tire sensor data and the vehicle exterior environment data.

[0075] In related technologies, tire status recognition only focuses on vehicle sensor information (such as tire sensor data), lacks the acquisition and utilization of environmental information, and has poor robustness in tire blowout recognition under different scenarios.

[0076] After obtaining the external environment data, the external environment data can carry relevant information about the road. There are different types of roads, such as ordinary roads, mud roads, snow roads, sandy roads, etc. Tires traveling on different roads have different driving conditions and tire temperatures, and the tire pressure characteristics will also be different. The tire sensor data and the external environment data can be combined to determine the tire status of the current vehicle, and realize tire blowout recognition in different road scenarios, which can improve the accuracy and robustness of recognition.

[0077] In an embodiment of the present invention, by acquiring tire sensor data and external environment data of the current vehicle, the tire status of the current vehicle is determined based on the tire sensor data and external environment data, thereby realizing vehicle tire status identification by combining tire sensor data and external environment, thereby improving accuracy and robustness.

[0078] Specifically, by combining the data collected by external sensors (such as on-board camera images and lidar point clouds) with the data collected by internal sensors (such as tire pressure and tire temperature), tire blowout identification is performed together. Compared with using only the data collected by internal sensors, the data collected by external sensors contains the results of road information detection, thereby improving the robustness of tire blowout identification.

[0079] In some embodiments of the present invention, determining the tire status of the current vehicle according to the tire sensor data and the vehicle external environment data includes:

[0080] Sub-step 11: converting the tire sensor data into first image data.

[0081] In practical applications, the tire sensor data collected by the in-vehicle sensor is not image data, but a time series signal, while the external environment data collected by the on-board camera that observes the road environment is image data. The data needs to be unified in form for easy fusion, so the tire sensor data can be converted into the first image data.

[0082] In some embodiments of the present invention, the first image data is a Graham angle field diagram, and the converting of the tire sensor data into the first image data includes: determining time series data from the tire sensor data; performing coordinate conversion on the time series data to obtain polar coordinate data; and generating a Graham angle field diagram based on the polar coordinate data to obtain the first image data.

[0083] In practical applications, since the tire sensor data of the in-vehicle sensor is time series data, the GAF (Gramian Angular Field) image coding technology can be used to convert the time series data in the tire sensor data into two-dimensional image data, which can retain the time series characteristics of the given signal to the greatest extent.

[0084] In some examples, two different Gramian angle field diagrams can be generated based on different operators, namely Gramian Summation Angular Field (GASF) and Gramian Difference Angular Field (GADF), such as the following formula:

[0085]

[0086] Among them, i and j represent the sequential index values ​​of the sequence points in the time series data, θ is the angle value in polar coordinates, and A GASF is the value of the Gram angle and field, A GADF is the value of the Gram angle difference field.

[0087] From the above formula, we can see that as time increases, the position moves from the upper left corner to the lower right corner, so the time dimension is encoded into the geometric structure of the matrix through GAF, thereby achieving high-precision conversion from one-dimensional time series data to two-dimensional color images.

[0088] In some examples, since there is no essential difference between GASF and GADF, embodiments of the present invention may use only GASF to convert time series data.

[0089] like Figure 2 , extract one-dimensional time series data from the tire sensor data. For example, in a rectangular coordinate system, the horizontal axis of the time series data represents the time value, and the vertical axis represents the parameter value measured by the sensor at each time point. The time series data is converted into polar coordinate data. After obtaining the polar coordinate data, the polar coordinate data can be converted into a Gram angle field diagram.

[0090] In some embodiments of the present invention, the coordinate conversion of the time series data to obtain polar coordinate data includes: encoding the parameter value of the time series data to obtain the angle value on the polar coordinate, and encoding the time value of the time series data to obtain the radius value on the polar coordinate; and obtaining the polar coordinate data based on the angle value and the radius value.

[0091] In some embodiments of the present invention, before performing coordinate conversion on the time series data to obtain polar coordinate data, the method further includes: performing normalization processing on the time series data.

[0092] Assume that the time series data collected by a certain in-vehicle sensor is , it can be normalized and scaled to the value range of [-1, 1] according to the following formula.

[0093]

[0094] In practical applications, time series data can be parameter values ​​and the values ​​corresponding to the parameter values. For example, if the parameter values ​​are tire pressure and tire temperature, they can be encoded in polar coordinates. The parameter values ​​can be encoded with the cosine of the angle. The encoded angle value is between [0, π], and the timestamp corresponding to the parameter value can be encoded as a radius value.

[0095] In some examples, the following formula may be used:

[0096]

[0097] Among them, θ is the time series data x i The corresponding angle value, t i is the time series data x i The corresponding time value, M is the normalization factor.

[0098] In some embodiments of the present invention, after the normalization processing is performed on the time series data, the method further includes: if there are multiple channels of time series data, merging the multiple channels of time series data into one channel of time series data.

[0099] In practical applications, various time series data collected by in-vehicle sensors, such as tire temperature, tire pressure, speed, acceleration, etc., can be normalized first, then directly superimposed to synthesize one channel of time series data, and then the time series data can be converted.

[0100] Sub-step 12, determining second image data according to the vehicle exterior environment data.

[0101] As for the vehicle exterior environment data, it includes the second image data collected by the vehicle-mounted camera and does not need to be converted.

[0102] Sub-step 13, determining the tire status of the current vehicle according to the first image data and the second image data.

[0103] After the first image data and the second image data are obtained, the tire status can be identified using a pre-trained neural network model to obtain the tire status of the current vehicle.

[0104] In the embodiment of the present invention, the tire sensing data collected by the in-vehicle sensor is converted into image data, and combined with the image data collected by the in-vehicle camera outside the vehicle to detect the vehicle tire status, so as to adapt to different vehicle external environment scenes, thereby improving the accuracy and robustness of the vehicle tire status detection. Figure 3 A neural network architecture, such as the ResNeXt-50 network, is used to fuse in-vehicle sensor data (i.e., tire sensor data) and on-board camera data (i.e., external environment data) for tire blowout recognition. To train this network, it is necessary to collect sensor data of tire blowouts and non-flattened tires under various road conditions when collecting data.

[0105] In the neural network model, two separate unimodal encoders are used to extract features from each modality independently, and the obtained unimodal features are converted into BEV (Bird's Eye View) space. Subsequently, the CamFuse module (a module for content-aware expansion and multimodal feature fusion) is used for feature fusion, and the obtained multimodal BEV features are processed by a two-stage deep classification decoder.

[0106] In some embodiments of the present invention, determining the tire status of the current vehicle according to the first image data and the second image data includes:

[0107] Sub-step 131, obtaining a first bird's-eye view feature map according to the first image data.

[0108] In some embodiments of the present invention, obtaining a first bird's-eye view feature map based on the first image data includes: extracting a first two-dimensional feature map from the first image data, and converting the first two-dimensional feature map into a first bird's-eye view feature map.

[0109] In some embodiments of the present invention, extracting the first two-dimensional feature map from the first image data includes: extracting the first two-dimensional feature map from the first image data through a first single-modal encoder.

[0110] In some embodiments of the present invention, converting the first two-dimensional feature map into a first bird's-eye view feature map includes: converting the first two-dimensional feature map into a first bird's-eye view feature map according to preset size parameters.

[0111] In order to extract features from each modality independently, two separate unimodal encoders are used and the resulting unimodal features are transformed into the BEV space to obtain a bird’s-eye view feature map.

[0112] Since the first image data is a Gram angular field map obtained by converting time series data, the first single-mode encoder in the ResNeXt-50 network can be used , which is used to learn the Gram angular field map features, that is, the first two-dimensional feature map. After obtaining the first two-dimensional feature map, the first two-dimensional feature map can be converted into a first bird's-eye view feature map by using a preset size parameter.

[0113] Sub-step 132, obtaining a second bird's-eye view feature map according to the second image data.

[0114] In some embodiments of the present invention, obtaining a second bird's-eye view feature map based on the second image data includes: extracting a second two-dimensional feature map from the second image data, and converting the second two-dimensional feature map into a second bird's-eye view feature map.

[0115] In some embodiments of the present invention, extracting a second two-dimensional feature map from the second image data includes: extracting a second two-dimensional feature map from the second image data by using a second single-modal encoder.

[0116] In some embodiments of the present invention, converting the second two-dimensional feature map into a second bird's-eye view feature map includes: obtaining external parameters of the camera, and converting the second two-dimensional feature map into a second bird's-eye view feature map by combining the external parameters of the camera and preset size parameters.

[0117] Since the second image data is a visible light image captured by the on-board camera, the second single-mode encoder in the ResNeXt-50 network can be used , which is used to extract features from the visible spectrum, namely, the second two-dimensional feature map. After obtaining the second two-dimensional feature map, the second two-dimensional feature map can be converted into a second bird's-eye view feature map by combining the external parameters of the camera and the preset size parameters.

[0118] In some examples, the second image data acquired by the vehicle-mounted camera is not directly sent to the network. It needs to calculate the optical flow map with the image of the previous frame. The optical flow map contains more motion information in the image field of view and can better reflect the event of tire blowout with vibration information.

[0119] The second unimodal encoder and the first unimodal encoder may be two separate unimodal encoders, and the second unimodal encoder and the first unimodal encoder have different structures, such as the second unimodal encoder includes a pyramid block of multi-scale features.

[0120] like Figure 3 , a pyramid block is represented by e, which is composed of the second unimodal encoder Create and set a series of such pyramid blocks , N is the number of pyramid blocks representing each multi-scale feature, and each pyramid block extracts and fine-tunes features at multiple stages by using a set of residual blocks.

[0121] In some examples, the lower resolution pyramid block e can be firsti Upsample and then combine with the previous pyramid block e i-1 This helps regularize the network and effectively learn contextual features in multiple spatial dimensions.

[0122] In some embodiments of the present invention, before converting the second two-dimensional feature map into a second bird's-eye view feature map, it also includes: determining the distortion parameters of the second two-dimensional feature map; determining the two-dimensional image space coordinates of the lower boundary of the world coordinate space according to the distortion parameters, and determining the upper limit value and lower limit value of the coordinates in the two-dimensional image space; and cropping the second two-dimensional feature map according to the upper limit value and lower limit value of the coordinates in the two-dimensional image space.

[0123] In BEV projection, the 2D image features need to be projected into the 3D BEV space, which requires ensuring the accurate representation of specific grid areas in the world coordinate space.

[0124] Specifically, the boundary of the required image in the world coordinate space can be set. For example, with the vehicle outline as a reference, images within 5 meters in front, behind, left and right of the vehicle are collected. Since the second two-dimensional feature map is the image data of visible light collected by the on-board camera, the 2D image features themselves may have parts that exceed the boundary. This part of the area needs to be cropped to remove the features in the area. Therefore, the feature map from each pyramid block (i.e., the second two-dimensional feature map) can be subjected to a cropping process to ensure that the 2D image features correspond to the boundary in the world coordinate space.

[0125] In practice, cameras following the Kannala-Brandt (a fisheye camera model) camera model are used. Instead of characterizing radial distortion based on the distance to the image center (radius), the Kannala-Brandt model considers distortion as a function of the angle of incidence of the light rays passing through the lens. This approach was chosen because the distortion function is smoother when parameterized with respect to this angle, which makes it easier to model as a power series.

[0126] In order to crop a region from the feature map, the following formulas 2 to 4 can be used to determine the distortion coefficient, and the two-dimensional image space coordinates (u, v) representing the boundary in the world coordinate space can be obtained using formula 1, the corresponding focal length, distortion parameters, principal point, and height and depth of the world coordinate space.

[0127]

[0128] Among them, π() represents the distortion function based on the Kannala-Brandt model, x and y are the pixel coordinates in the two-dimensional image space, r is the distance between the image point and the principal point, and f x 、fy is the focal length, d(θ) is the polynomial equation describing the internal distortion parameter of the fisheye camera, θ is the angle between the main axis and the incident light, C x and C y represents the principal point along the x-axis and y-axis, K i is the distortion parameter of the i-th order in the polynomial equation d(θ), and z is the distance between the image point and the camera.

[0129] Through this transformation, the two-dimensional image space coordinates (u, v) of the boundary in the world coordinate space are obtained, and then the upper and lower limits of the coordinates in the two-dimensional image space can be determined. Then, the second two-dimensional feature map is cropped with reference to the upper and lower limits of the coordinates in the two-dimensional image space, and the minimum and maximum depths (z min , z max )、height(y min ,y max ) projected into the two-dimensional image space, the upper and lower limits of the coordinates (common width u, maximum value v max、 Minimum value v min ) within the range of .

[0130] In an embodiment of the present invention, by cropping a two-dimensional feature map obtained from image data collected by a vehicle-mounted camera, the accuracy of the two-dimensional feature map relative to the world coordinate space can be improved, thereby improving the accuracy of subsequent tire status recognition.

[0131] Sub-step 133: fusing the first bird's-eye view feature map and the second bird's-eye view feature map to obtain a fused bird's-eye view feature map.

[0132] After obtaining the first bird's-eye view feature map and the second bird's-eye view feature map of two single modalities, the first bird's-eye view feature map and the second bird's-eye view feature map may be fused to obtain a multi-modal fused bird's-eye view feature map.

[0133] For example, using CaMFuse, using and The spatially extracted unimodal BEV features (i.e., the first bird's-eye view feature map and the second bird's-eye view feature map) are synthesized into a fused bird's-eye view feature map.

[0134] Due to the different representations of the environment by sensors, the camera captures intricate semantic details in pixel format, which contains geometric distortions, while the in-vehicle sensors complete the acquisition of the in-vehicle state by recording time series data. This difference in sensing modalities increases the complexity of achieving seamless fusion to effectively present the environment by minimizing the domain gap during feature fusion.

[0135] In an embodiment of the present invention, image data obtained by converting tire sensor data and image data collected by a vehicle-mounted camera are converted into feature maps in a bird's-eye view space, and then feature fusion is performed on the two feature maps in the bird's-eye view space. The fused bird's-eye view feature map can reduce the data difference between two different sensors, and can be combined with the conditions of the external environment on the basis of the tire sensor data, thereby better identifying the vehicle tire status and improving the accuracy and robustness of vehicle tire status identification.

[0136] In some embodiments of the present invention, the fusing of the first bird's-eye view feature map and the second bird's-eye view feature map to obtain a fused bird's-eye view feature map includes: splicing the first bird's-eye view feature map and the second bird's-eye view feature map; during the splicing process, obtaining a target dilation value, and using the target dilation value to dilate the first bird's-eye view feature map and the second bird's-eye view feature map to obtain a fused bird's-eye view feature map.

[0137] Since sensors may be misaligned, if this misalignment is not corrected, it may affect the accuracy of the prediction. If you want to ensure the reliability and precision of the fused features in the final prediction process, it is crucial to solve and correct this misalignment. The CaMFuse module handles the above problem by adjusting the dilated convolution and using content-aware dilution in the BEV feature space of the in-vehicle sensor data to account for its sparsity.

[0138] The convolution kernel is allowed to adjust its dilation value (i.e., target dilation value) according to different contents at the pixel level, and then the first bird's-eye view feature map and the second bird's-eye view feature map are dilated using the target dilation value to obtain a fused bird's-eye view feature map.

[0139] Assume that It consists of l convolutional layers. The output of the lth layer using the dilated convolution kernel K at position (i, j) is represented by λ, which is defined as follows:

[0140]

[0141] in, is the input of the (L-1)th layer, , w, h are related to the input The corresponding width and height, Represents a real matrix with width w and height h, m and n are the indices of the convolution kernel, and δ and w are the expansion value and weight of the learnable kernel K.

[0142] The basic concept of adaptive dilated convolution consists in estimating δ at position (i, j) using a function ∆i,j that takes into account the input, represented as a random variable , where ∆i,j is taken from softmax sampling , using the smoothing parameter τ to control the output to be close to the true classification distribution.

[0143]

[0144] in, represents the set of potential position-specific hidden priors, , , D represents the number of valid options for the expansion value.

[0145] The hidden priors at the lth level are estimated by considering β hierarchical layers and then using a recurrent aggregation method defined as follows.

[0146]

[0147] in, Represents the weight matrix associated with the previous layer of hidden priors, denoted as ; The weight matrix corresponding to the input of the lth layer; f( ) represents the nonlinear activation function.

[0148] As l goes deeper, the equation provided helps to aggregate information across layers, thus promoting interdependence between layers. However, as an extreme case of recursive aggregation, Y Markov aggregation, which takes the kernel weights of the previous hidden prior This means that similar to the Markov model, only the last layer is sufficient to estimate the hidden prior of the content and learn the weights of the dilated convolution kernels.

[0149]

[0150] Data from in-vehicle sensors The BEV feature maps are subjected to content-aware dilated convolution. Subsequently, these processed BEV feature maps are concatenated and combined.

[0151] In an embodiment of the present invention, by adopting the dilated convolution method to perform fusion in the process of fusion of bird's-eye view feature maps, the gap between data collected by different sensors in the feature fusion process is narrowed, and the reliability and accuracy of the fused features in the prediction process are improved.

[0152] Sub-step 134, performing tire blowout identification on the fused bird's-eye view feature image to obtain the tire status of the current vehicle.

[0153] In some embodiments of the present invention, the step of identifying a tire blowout on the fused bird's-eye view feature image to obtain a tire status of the current vehicle includes: using a two-stage multi-scale classification prediction decoder to identify a tire blowout on the fused bird's-eye view feature image to obtain a tire status of the current vehicle.

[0154] In practical applications, a two-stage multi-scale classification prediction decoder DSeg(.) can be used, which takes the BEV feature map generated by CaMFuse as input and generates vehicle tire burst predictions.

[0155] Specifically, a top-down approach is adopted, aiming to predict the probability of vehicle tire blowout from a set of BEV feature maps. The decoder consists of two consecutive residual blocks. The main goal of the initial block is to avoid compromising spatial features within the same resolution. On the other hand, the second block operates on a higher dimension to exploit road condition information.

[0156] The initial block consists of a set of consecutive convolutional layers followed by a concatenation operation. It is worth noting that the input of this block is 16 feature maps and the output is 32 feature maps. The next residual block starts with two parallel transposed convolutional layers.

[0157] One layer has a stride of 2 and a kernel size of 4×4, while the other layer has a stride of 2 and a kernel size of 2×2. However, both layers produce an output of 16 feature maps each. The output feature map of the transposed layer with a stride of 1 is input to a convolutional layer with a stride of 1 and a kernel size of 3×3, and produces the same number of features as the input. Then, the output of this layer and the output of the transposed layer with a kernel size of 2×2 are concatenated and passed through two consecutive convolutional layers with the same input and output channels, a stride of 1, and a kernel size of 3×3. The resulting feature map is further concatenated with the feature map obtained in the previous convolution operation. The feature map undergoes a finalization phase, passing through two convolutional layers Conv1×1. Finally, the weight parameters of the network are updated using the cross entropy function as the loss function.

[0158] The following combination Figure 3 The present invention is exemplified as follows:

[0159] 1. Obtain tire sensor data, convert the tire sensor data into a Gram angle field map, obtain a first two-dimensional feature map through a first single-mode encoder, and convert the first two-dimensional feature map into a first bird's-eye view feature map.

[0160] 2. Obtain the pictures collected by the on-board camera (i.e., the external environment data of the vehicle), and use the images at adjacent moments (such as Figure 3 The optical flow features are obtained by combining the image at time t-1 and the image at time t in the image, and the optical flow graph is obtained. The second single-mode encoder is combined with multiple convolutional layers (such as Figure 3 The convolutional layer 1 and the convolutional layer 2 are operated to obtain the first two-dimensional feature map, and after the feature map is cropped, the two-dimensional feature map is converted into a second bird's-eye view feature map.

[0161] 3. Through the multimodal feature fusion module, the first bird's-eye view feature map and the second bird's-eye view feature map are fused to obtain a fused bird's-eye view feature map. The fused bird's-eye view feature map is used to identify tire blowouts through a two-stage multi-scale classification prediction decoder to obtain the tire status of the current vehicle, that is, the tire blowout state and the tire non-blowout state.

[0162] In some embodiments of the present invention, it also includes:

[0163] When the tire state is a flat tire state, a prompt message indicating that the current vehicle is in a flat tire state is presented.

[0164] After identifying the tire status of the current vehicle, if the tire status is a flat tire state, a prompt message indicating that the current vehicle is in a flat tire state may be presented, i.e., a prompt is given to the user that the vehicle has a flat tire. If the tire status is not a flat tire state, no prompt is given.

[0165] In some embodiments of the present invention, it also includes:

[0166] The road information is determined according to the vehicle external environment data, and when it is determined according to the road information that there is a risk of tire blowout, a prompt message indicating that the current vehicle has a risk of tire blowout is presented.

[0167] In practical applications, road information can be determined from the external environment data, such as the type of road, the location and depth of potholes on the road, the location of sharp objects on the road that can easily puncture tires, and other information. Then, it can be directly judged whether the current vehicle has a risk of tire blowout based on the road information. If a sharp object is detected on the road, it is determined that there is a risk of tire blowout on the road.

[0168] If it is determined based on road information that the current vehicle is at risk of a tire blowout, a prompt message indicating the risk of a tire blowout may be presented, i.e., a warning prompt of the risk of a tire blowout may be given to the user; if it is determined based on road information that the current vehicle is at risk of a tire blowout, no prompt may be given.

[0169] In some embodiments of the present invention, it also includes:

[0170] When the first tire blowout warning message sent by other vehicles to the current vehicle is received, a prompt message indicating that the current vehicle has a tire blowout risk is presented.

[0171] In actual applications, multiple vehicles can communicate with each other through the network, such as using V2X (Vehicle to X, vehicle-to-external information exchange) technology.

[0172] When other vehicles detect that the current vehicle has a tire blowout risk, they can send a first tire blowout warning message to the current vehicle. When receiving the first tire blowout warning message, they can present a prompt message that the current vehicle has a tire blowout risk, that is, warn the user of the tire blowout risk.

[0173] In some examples, by collecting external environmental data, the external environmental data may include tire surface data of other vehicles, such as bulges, cracks, insufficient tread depth, etc. on the tire surface. These features can reflect whether there is a risk of tire blowout on the wheel. When it is determined that there is a risk of tire blowout based on the tire surface information, other vehicles send a first tire blowout warning message to the current vehicle to prompt the current vehicle that there is a risk of tire blowout.

[0174] In some embodiments of the present invention, it also includes:

[0175] The tire surface information of other vehicles is determined according to the vehicle external environment data, and when it is determined that there is a risk of tire blowout according to the tire surface information, a second tire blowout warning message is sent to the other vehicles.

[0176] In actual applications, the external environment data may include tire surface data of other vehicles, such as bulges, cracks, insufficient tread depth, etc. These features can reflect whether there is a risk of tire blowout. When the risk of tire blowout is determined based on the tire surface information, the current vehicle sends a second tire blowout warning message to other vehicles to alert them that there is a risk of tire blowout.

[0177] In an embodiment of the present invention, vehicle-mounted sensors are used to detect tire blowout signs such as bulges, cracks, and tread depth of surrounding vehicle tires, and a tire blowout warning is issued to the target vehicle through V2X. By introducing the detection of wheel appearance information, the tire blowout warning capability is further enhanced.

[0178] The following combination Figure 4 The present invention is exemplified as follows:

[0179] 1. Start the entire process of tire blowout identification and warning and go to step 2;

[0180] 2. Perform data collection to obtain the required data. First, obtain sensor data inside the vehicle, such as tire pressure data and tire temperature data; second, obtain sensor data outside the vehicle, such as on-board camera images and lidar point cloud data; third, receive tire blowout warning information from surrounding vehicles, which can be obtained through V2X means; perform steps 3, 4, and 5 simultaneously;

[0181] 3. Determine whether the tire blowout warning information sent by the surrounding vehicles is received. If so, execute step 8, if not, execute step 3. The surrounding vehicles will use the image or point cloud of the vehicle to detect the tires of the vehicle, such as detecting bulges, cracks, insufficient tread depth and other features on the tire surface. These features can reflect the risk of tire blowout. The information can be sent to the target vehicle through V2X. This process is also the specific execution logic in step 4;

[0182] 4. By analyzing the on-board camera image and the LiDAR point cloud, determine whether there is a risk of tire blowout in the surrounding vehicles. If yes, proceed to step 8; if no, proceed to step 2;

[0183] 5. Identify road information by analyzing the on-board camera image and the LiDAR point cloud, and simultaneously execute steps 6 and 7. The image and point cloud can reflect the type of road, the location and depth of potholes on the road, and the location of sharp objects on the road that can easily puncture tires.

[0184] 6. Determine whether there is a risk of tire blowout on the road. If so, go to step 8. If not, go to step 2. If there are sharp objects on the road, then there is a risk of tire blowout on the road;

[0185] 7. Determine whether the vehicle has a tire blowout based on the in-vehicle sensor data and the acquired road information. If so, execute step 9; if not, execute step 2. There are different types of roads, such as ordinary roads, mud roads, snow roads, sandy roads, etc. The tires on different roads have different driving conditions and tire temperatures, and the tire pressure characteristics will also be different. Performing tire blowout identification in the context of road scene information can improve the robustness of identification.

[0186] 8. If a tire blowout risk exists, the user will be warned;

[0187] 9. A reminder is given that the vehicle has a flat tire, and the process goes to step 10;

[0188] 10. End the process.

[0189] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0190] Some embodiments of the present invention provide a vehicle tire status recognition device, which is used to implement the method described above.

[0191] Some embodiments of the present invention further provide an electronic device, comprising a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the above method is implemented when the computer program is executed by the processor.

[0192] Some embodiments of the present invention further provide a vehicle, comprising the apparatus as described above, or the electronic device as described above.

[0193] Some embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, and the computer program implements the above method when executed by a processor.

[0194] Some embodiments of the present invention further provide a computer program product, including a computer program, which implements the above method when executed by a processor.

[0195] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0196] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0197] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0198] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0199] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0200] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0201] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0202] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0203] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.

[0204] The vehicle tire status recognition method, device, equipment, vehicle, medium and product provided above are introduced in detail. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A vehicle tire status recognition method, characterized in that: The method comprises: Obtain the tire sensor data and external environment data of the current vehicle; Determining the tire status of the current vehicle according to the tire sensor data and the vehicle external environment data; Determining the tire status of the current vehicle according to the tire sensor data and the vehicle exterior environment data includes: Converting the tire sensor data into first image data; Determine second image data according to the vehicle exterior environment data; Determine the tire status of the current vehicle according to the first image data and the second image data; Determining the tire status of the current vehicle according to the first image data and the second image data includes: Obtaining a first bird's-eye view feature map according to the first image data; Obtaining a second bird's-eye view feature map according to the second image data; Fusing the first bird's-eye view feature map and the second bird's-eye view feature map to obtain a fused bird's-eye view feature map; The fused bird's-eye view feature image is used to perform tire blowout identification to obtain the tire status of the current vehicle.

2. The method according to claim 1, characterized in that The first image data is a Gram angle field map, and the converting the tire sensor data into the first image data includes: Determining time series data from the tire sensor data; Performing coordinate conversion on the time series data to obtain polar coordinate data; A Gram angle field diagram is generated according to the polar coordinate data to obtain first image data.

3. The method according to claim 2, characterized in that The step of performing coordinate conversion on the time series data to obtain polar coordinate data includes: Encoding the parameter value of the time series data to obtain an angle value on polar coordinates, and encoding the time value of the time series data to obtain a radius value on polar coordinates; Polar coordinate data is obtained according to the angle value and the radius value.

4. The method according to claim 2, characterized in that: Before performing coordinate conversion on the time series data to obtain polar coordinate data, the method further includes: The time series data is normalized.

5. The method according to claim 4, characterized in that After the time series data is normalized, the method further includes: When there are multiple channels of time series data, the multiple channels of time series data are merged into one channel of time series data.

6. The method according to claim 1, characterized in that The step of obtaining a first bird's-eye view feature map according to the first image data includes: Extracting a first two-dimensional feature map from the first image data, and converting the first two-dimensional feature map into a first bird's-eye view feature map; The step of obtaining a second bird's-eye view feature map according to the second image data comprises: A second two-dimensional feature map is extracted from the second image data, and the second two-dimensional feature map is converted into a second bird's-eye view feature map.

7. The method according to claim 6, characterized in that The step of extracting a first two-dimensional feature map from the first image data comprises: extracting a first two-dimensional feature map from the first image data by a first single-mode encoder; The step of extracting a second two-dimensional feature map from the second image data comprises: A second two-dimensional feature map is extracted from the second image data by a second single-modal encoder.

8. The method according to claim 6, characterized in that The converting the first two-dimensional feature map into a first bird's-eye view feature map comprises: According to preset size parameters, converting the first two-dimensional feature map into a first bird's-eye view feature map; The converting the second two-dimensional feature map into a second bird's-eye view feature map comprises: The external parameters of the camera are obtained, and the second two-dimensional feature map is converted into a second bird's-eye view feature map by combining the external parameters of the camera and preset size parameters.

9. The method according to claim 6, characterized in that Before converting the second two-dimensional feature map into a second bird's-eye view feature map, the method further includes: Determining a distortion parameter of the second two-dimensional feature map; Determine the two-dimensional image space coordinates of the lower boundary of the world coordinate space according to the distortion parameters, and determine the upper limit value and the lower limit value of the coordinates in the two-dimensional image space according to the two-dimensional image space coordinates; The second two-dimensional feature map is cropped according to the coordinate upper limit value and the coordinate lower limit value in the two-dimensional image space.

10. The method according to claim 1, characterized in that The fusing the first bird's-eye view feature map and the second bird's-eye view feature map to obtain a fused bird's-eye view feature map includes: Splicing the first bird's-eye view feature map and the second bird's-eye view feature map; During the stitching process, a target expansion value is obtained, and the first bird's-eye view feature map and the second bird's-eye view feature map are expanded using the target expansion value to obtain a fused bird's-eye view feature map.

11. The method according to any one of claims 7 to 10, characterized in that: The step of performing tire blowout identification on the fused bird's-eye view feature image to obtain the tire status of the current vehicle includes: A two-stage multi-scale classification prediction decoder is used to identify tire blowouts on the fused bird's-eye view feature image to obtain the tire status of the current vehicle.

12. The method according to any one of claims 1 to 10, characterized in that: Also includes: The road information is determined according to the vehicle external environment data, and when it is determined according to the road information that there is a risk of tire blowout, a prompt message indicating that the current vehicle has a risk of tire blowout is presented.

13. The method according to any one of claims 1 to 10, characterized in that: Also includes: When the first tire blowout warning message sent by other vehicles to the current vehicle is received, a prompt message indicating that the current vehicle has a tire blowout risk is presented.

14. The method according to any one of claims 1 to 10, characterized in that: The vehicle external environment data includes tire surface data of other vehicles, and also includes: The tire surface information of other vehicles is determined according to the vehicle external environment data, and when it is determined that there is a risk of tire blowout according to the tire surface information, a second tire blowout warning message is sent to the other vehicles.

15. The method according to any one of claims 1 to 10, characterized in that: Also includes: When the tire state is a flat tire state, a prompt message indicating that the current vehicle is in a flat tire state is presented.

16. A vehicle tire status recognition device, characterized in that: The device is used to implement the method according to any one of claims 1 to 15.

17. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method according to any one of claims 1 to 15 when executed by the processor.

18. A vehicle, characterized in that: The vehicle comprises the apparatus as claimed in claim 16, or the electronic device as claimed in claim 17.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 15 is implemented.

20. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 15.

Images