Method, device and processor for determining load status of vehicle

Abstract

Embodiments of the present application provide a method, device and processor for determining the load state of a vehicle. The method further comprises: obtaining two-dimensional feature information, three-dimensional feature information and sound feature information of a target vehicle carrying a target object, wherein the two-dimensional feature information is used to represent at least the features of the target vehicle and the target object in a two-dimensional space, and the three-dimensional feature information is used to represent at least the features of the target vehicle and the target object in a three-dimensional space; performing feature fusion on the two-dimensional feature information, the three-dimensional feature information and the sound feature information to obtain comprehensive feature information; performing load prediction on the target vehicle based on the comprehensive feature information using a prediction model to obtain a load prediction result; and determining the load state of the target vehicle based on the load prediction result, wherein the load state is used to represent whether the target vehicle is in an overload state or a non-overload state. Through the present application, the technical problem of low flexibility in determining the load state of a vehicle is solved.

Description

Technical Field

[0001] This invention relates to the field of vehicle overload detection technology, and more specifically, to a method, apparatus, and processor for determining the load status of a vehicle. Background Technology

[0002] Currently, a massive number of vehicles (such as transport vehicles and buses) operate on the roads. Overloading not only poses a threat to road safety but also significantly impacts the lifespan of roads, bridges, and tunnels. Therefore, effectively detecting and penalizing vehicle load conditions is crucial for ensuring road safety and achieving the rational use of road resources.

[0003] In related technologies, a contact-based detection method has been proposed, which involves setting up a weighing platform or weighing equipment on the road to detect vehicle speed, number of axles, weight, and other information to determine if a vehicle is overloaded. However, installing such weighing equipment is costly, bulky, space-consuming, difficult to install, and poses certain safety hazards, making it unsuitable for areas with high vehicle density. Furthermore, vehicles need to slow down when passing the weighing platform, which can easily lead to errors in detecting speed, number of axles, and axle weight. Therefore, the technical problem of low flexibility in determining the vehicle's load status remains.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a method, apparatus, and processor for determining the load status of a vehicle, to at least address the technical problem of low flexibility in determining the load status of a vehicle in related technologies.

[0006] According to an embodiment of the present invention, a method for determining the load status of a vehicle is provided, comprising: acquiring two-dimensional feature information, three-dimensional feature information, and sound feature information of a target vehicle carrying a target object, wherein the two-dimensional feature information is used to at least represent the features of the target vehicle and the target object in two-dimensional space, the three-dimensional feature information is used to at least represent the features of the target vehicle and the target object in three-dimensional space, and the sound feature information is used to represent the features of the sound generated by the target vehicle during driving; performing feature fusion on the two-dimensional feature information, the three-dimensional feature information, and the sound feature information to obtain comprehensive feature information; based on the comprehensive feature information, using a prediction model to predict the load of the target vehicle to obtain a load prediction result; and determining the load status of the target vehicle based on the load prediction result, wherein the load status is used to indicate whether the target vehicle is in a non-overloaded state or an overloaded state.

[0007] In an exemplary embodiment, feature fusion is performed on two-dimensional feature information, three-dimensional feature information, and sound feature information to obtain comprehensive feature information. This includes: obtaining target feature vectors corresponding to the two-dimensional feature information, three-dimensional feature information, and sound feature information respectively, and mapping the target feature vectors into a target feature space, wherein the target feature vectors are obtained by normalizing and standardizing the initial feature vectors corresponding to the two-dimensional feature information, three-dimensional feature information, and sound feature information respectively; and fusing the target feature vectors corresponding to the two-dimensional feature information, three-dimensional feature information, and sound feature information in the target feature space to obtain a comprehensive feature vector corresponding to the comprehensive feature information.

[0008] In an exemplary embodiment, in the target feature space, the target feature vectors corresponding to two-dimensional feature information, three-dimensional feature information, and sound feature information are fused to obtain a comprehensive feature vector corresponding to comprehensive feature information. This includes: in the target feature space, using a self-attention mechanism, determining the correlation between the two-dimensional feature information, three-dimensional feature information, and sound feature information based on the target feature vector; determining the weights corresponding to the two-dimensional feature information, three-dimensional feature information, and sound feature information respectively based on the correlation; and fusing the target feature vectors based on the weights to obtain a comprehensive feature vector.

[0009] In an exemplary embodiment, the two-dimensional feature information includes density information of the target object, vehicle model information of the target vehicle, and tire deformation information of the target vehicle. Obtaining density information includes: identifying the stacking height and coverage area information of the target object from a vehicle image of the target vehicle, and determining density information based on the stacking height and coverage area information. The image content of the vehicle image includes at least the target vehicle and the target object, and the coverage area information represents the proportion of the target object occupying the mounting area of ​​the target vehicle. Obtaining vehicle model information includes: identifying the appearance features of the target vehicle from a vehicle image, and determining vehicle model information based on the appearance features. Obtaining tire deformation information includes: identifying the upper radius and lower radius of the tires of the target vehicle from a vehicle image, and determining tire deformation information based on the difference between the upper and lower radii. The upper radius is the radius of the tire above the axle, and the lower radius is the radius of the tire below the axle. The tire deformation information represents the degree of deformation of the tire. The method further includes: vectorizing the density information, vehicle model information, and tire deformation information to obtain an initial feature vector corresponding to the two-dimensional feature information.

[0010] In an exemplary embodiment, the three-dimensional feature information includes the outline data of the target vehicle and the speed information of the target vehicle. Acquiring the outline data includes: scanning the target vehicle using a laser scanning device to acquire point cloud data of the target vehicle; determining the outline data based on the point cloud data; acquiring the speed information includes: determining the speed information based on the point cloud data; the method further includes: vectorizing the outline data and the speed information to obtain an initial feature vector corresponding to the three-dimensional feature information.

[0011] In an exemplary embodiment, acquiring the sound feature information of a target vehicle carrying a target object includes: acquiring a sound signal using a sound sensor during the vehicle's operation; performing time-frequency transformation on the sound signal to obtain an audio frequency domain signal; determining the peak frequency and peak amplitude based on the audio frequency domain signal, and processing the audio frequency domain signal using a Mel frequency cepstral filter to obtain the corresponding Mel frequency cepstral coefficients; the method further includes: vectorizing the peak frequency, peak amplitude, and Mel frequency cepstral coefficients to obtain an initial feature vector corresponding to the sound feature information.

[0012] In an exemplary embodiment, determining the load status of a target vehicle based on a load prediction result includes: determining a target load threshold corresponding to the target vehicle based on the vehicle model information; comparing the load prediction result with the target load threshold; determining the load status as an overloaded state in response to an overload prediction result being greater than the target load threshold; and determining the load status as a non-overloaded state in response to a load prediction result being less than or equal to the target load threshold.

[0013] According to another embodiment of the present invention, a device for determining the load state of a vehicle is provided, comprising: an acquisition unit, configured to acquire two-dimensional feature information, three-dimensional feature information, and sound feature information of a target vehicle carrying a target object, wherein the two-dimensional feature information is used to at least represent the features of the target vehicle and the target object in two-dimensional space, the three-dimensional feature information is used to at least represent the features of the target vehicle and the target object in three-dimensional space, and the sound feature information is used to represent the features of the sound generated by the target vehicle during its operation; a fusion unit, configured to perform feature fusion on the two-dimensional feature information, the three-dimensional feature information, and the sound feature information to obtain comprehensive feature information; a prediction unit, configured to predict the load of the target vehicle based on the comprehensive feature information using a prediction model to obtain a load prediction result; and a determination unit, configured to determine the load state of the target vehicle based on the load prediction result, wherein the load state indicates whether the target vehicle is in a non-overloaded state or an overloaded state.

[0014] In one exemplary embodiment, the apparatus further includes: an image acquisition device for acquiring a vehicle image of the target vehicle while the target vehicle is in motion; a laser scanning device for acquiring point cloud data of the target vehicle while the target vehicle is in motion; a sound acquisition device for acquiring sound signals of the target vehicle while the target vehicle is in motion; and a data processing unit for determining two-dimensional feature information based on the vehicle image, determining three-dimensional feature information based on the point cloud data, and determining sound feature information based on the sound signal in response to receiving the vehicle image, point cloud data, and sound signal.

[0015] According to yet another embodiment of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium storing a plurality of instructions adapted to be loaded by a processor and executed the steps in any of the above method embodiments.

[0016] According to yet another embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0017] According to yet another embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0018] In this embodiment of the invention, if it is necessary to determine the load status of a target vehicle carrying a target object, by comprehensively considering the features of the target object carrying the target vehicle in different dimensions (e.g., two-dimensional space, three-dimensional space, and sound dimension), the two-dimensional feature information, three-dimensional feature information, and sound feature information of the target vehicle in the corresponding dimensions can be obtained. Feature fusion can be performed on the two-dimensional feature information, three-dimensional feature information, and sound feature information to obtain comprehensive feature information. This comprehensive feature information can be input into a pre-trained prediction model. Using the comprehensive feature information, the prediction model can predict the load of the target vehicle, obtaining a load prediction result. That is, the prediction model can predict the current weight of the target object and the target vehicle. Based on the load prediction result, it can be determined whether the current target vehicle is in an overloaded or not overloaded state. In this embodiment, by fusing multimodal data (i.e., two-dimensional feature information, three-dimensional feature information, and sound feature information), the features reflected in the load status can be complementaryly captured from multiple dimensions, thereby improving the accuracy and comprehensiveness of the judgment. Furthermore, since the prediction model is trained on a large number of samples, it can learn the complex intrinsic relationships between various features. Even small changes in features can accurately reflect the vehicle's load status, thus improving prediction accuracy. In summary, by combining multimodal data fusion and a deep learning prediction model, flexible and accurate judgment of vehicle load status is achieved in various complex scenarios. The above method effectively overcomes the limitations of related contact detection methods, thereby improving the technical effect of increasing the flexibility of determining the vehicle's load status and solving the technical problem of low flexibility in determining the vehicle's load status. Attached Figure Description

[0019] Figure 1 This is a hardware structure block diagram of a mobile terminal for a method of determining the load status of a vehicle according to an embodiment of the present invention.

[0020] Figure 2 This is a flowchart of a method for determining the load status of a vehicle according to an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of the upper and lower radii of a tire according to an embodiment of the present invention;

[0022] Figure 4 This is a structural block diagram of a vehicle load state determination device according to an embodiment of the present invention.

[0023] Figure 5 This is a schematic diagram of an electronic device for determining the load status of a vehicle according to an embodiment of the present invention. Detailed Implementation

[0024] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0026] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a method of determining the load status of a vehicle according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0027] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the method for determining the load state of a vehicle in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0028] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0029] Those skilled in the art will understand that Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0030] In one possible implementation, embodiments of this disclosure provide a method for determining the load status of a vehicle. Figure 2 This is a flowchart of a method for determining the load state of a vehicle according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0031] Step S202: Obtain the two-dimensional feature information, three-dimensional feature information, and sound feature information of the target vehicle carrying the target object.

[0032] In the technical solution provided in step S202 of the present invention, the target object can be the goods or people carried on the target vehicle, corresponding to the type / model of the target vehicle (e.g., truck, bus). The target vehicle can be a vehicle whose load status is to be determined. Two-dimensional feature information can be used to at least represent the characteristics of the target vehicle and the target object in two-dimensional space, such as the distribution of goods in two-dimensional space, the type of the target vehicle, whether the tires are deformed, etc. The two-dimensional feature information can be image data of the target vehicle in two-dimensional space. Three-dimensional feature information can be used to at least represent the characteristics of the target vehicle and the target object in three-dimensional space, such as the volume, size, speed, etc. of the target vehicle. The three-dimensional feature information can be the three-dimensional contour information of the target vehicle in three-dimensional space, also known as laser data. Sound feature information can be used to represent the characteristics of the sound generated by the target vehicle during its operation, such as the frequency and volume of the sound, etc., and can be sound data of the target vehicle during its operation.

[0033] It should be noted that the features included in the above-mentioned two-dimensional feature information, three-dimensional feature information and sound feature information are only illustrative examples and are not specifically limited here. As long as the features in the above three dimensions can reflect the load status of the target vehicle, they are all within the protection scope of the embodiments of the present invention.

[0034] In this embodiment, if it is necessary to determine whether the target vehicle is overloaded, the two-dimensional feature information, three-dimensional feature information, and sound feature information of the target vehicle can be obtained.

[0035] Optionally, by acquiring multimodal feature information of the target vehicle, including two-dimensional feature information, three-dimensional feature information, and acoustic feature information, the load status of the target vehicle can be comprehensively evaluated. The above method fully demonstrates that the present invention utilizes multi-source information (multimodal data) to improve the flexibility and accuracy of overload detection.

[0036] For example, a two-dimensional image containing the target vehicle and the target object can be acquired using an image acquisition device (such as a camera). The characteristics of the target vehicle and the target object in the two-dimensional image (such as the distribution of the target object, the type of the target vehicle, the deformation of the wheels, etc.) can be analyzed to obtain two-dimensional feature information.

[0037] For another example, laser data (point cloud data) can be acquired through laser scanning equipment (such as lidar). The characteristics of target vehicles and objects in the laser data (such as the speed of the target vehicle, the outline of the target vehicle and the target object, etc.) can be analyzed to obtain three-dimensional feature information.

[0038] As an alternative example, sound signals generated by the target vehicle while it is in motion can be acquired using a sound sensor. By processing the sound signals, relevant features (such as frequency) can be extracted to obtain sound feature information.

[0039] It should be noted that the methods and processes for obtaining feature information in the above three dimensions are merely illustrative examples and are not specifically limited here. Any process or method that can comprehensively consider the load status of the target vehicle through feature information in the above three dimensions is within the protection scope of this invention.

[0040] In this embodiment of the invention, two-dimensional feature information, three-dimensional feature information, and acoustic feature information reflect the load status of the target vehicle from different perspectives. This allows for the utilization of the complementarity of multimodal data to comprehensively assess the load from multiple dimensions, thereby improving the accuracy of overload judgment. Furthermore, the detection method for the load status from the above three perspectives is non-contact, meaning that no direct contact sensors need to be installed on the vehicle or the ground, reducing deployment costs and maintenance work. Simultaneously, the above method is applicable to various types of vehicles, enhancing the flexibility and wide applicability of the detection.

[0041] In summary, the above-mentioned method, through the comprehensive collection and analysis of multimodal feature information of the target vehicle, enables flexible, accurate, and non-contact detection of the target vehicle's load status. It effectively overcomes the limitations of traditional contact detection methods and improves the technical effectiveness of overload detection, including accuracy, flexibility, real-time performance, adaptability, and robustness. This provides strong technical support for traffic safety management and vehicle overload control.

[0042] Step S204: Perform feature fusion on the two-dimensional feature information, three-dimensional feature information, and sound feature information to obtain comprehensive feature information.

[0043] In the technical solution provided by step S204 of the present invention, the comprehensive feature information can also be called comprehensive feature or fusion feature.

[0044] In this embodiment, after acquiring the two-dimensional feature information, three-dimensional feature information, and sound feature information of the target vehicle, feature fusion can be performed on the two-dimensional feature information, three-dimensional feature information, and sound feature information to obtain comprehensive feature information.

[0045] Optionally, feature fusion is performed on two-dimensional feature information, three-dimensional feature information, and sound feature information to obtain comprehensive feature information. This is a key part of realizing non-contact vehicle overload judgment based on multimodal data comprehensive calculation in this embodiment of the invention. The purpose of feature fusion is to integrate feature information obtained from different modalities (image, laser, sound) into a unified, high-dimensional feature representation, so that subsequent load prediction and overload judgment can be based on more comprehensive information, thereby improving the accuracy and robustness of the judgment.

[0046] In this embodiment of the invention, by comprehensively considering multimodal data, the prediction model can capture a more comprehensive and detailed vehicle load status, avoiding information loss or noise effects that may result from single-modal data, and significantly improving the accuracy of load prediction. Even if the data quality of a certain modality is poor or the information is incomplete, the data from other modalities can provide effective supplementation, making the performance of the prediction model more stable in complex environments and improving the robustness and reliability of overload judgment. In summary, by fusing two-dimensional feature information, three-dimensional feature information, and acoustic feature information to generate comprehensive feature information, not only is the effective integration of multimodal data achieved, but the performance of non-contact vehicle overload judgment is also improved, including prediction accuracy, robustness, and interpretability.

[0047] Step S206: Based on comprehensive feature information, use a prediction model to predict the load of the target vehicle and obtain the load prediction result.

[0048] In the technical solution provided in step S206 of the present invention, the prediction model can be obtained by using a deep learning model to predict using comprehensive feature information samples and load prediction result samples. The comprehensive feature information samples are obtained by fusing two-dimensional feature information samples, three-dimensional feature information samples, and sound feature information samples acquired during the driving process of a vehicle carrying the target object sample. The load prediction result can be the total weight of the target object and the target vehicle predicted by the comprehensive feature information, or it can be a binary classification result indicating whether it is overloaded, depending on the design and training objectives of the deep learning model. In this embodiment of the invention, the total weight of the target object and the target vehicle as the load prediction result can be used as an example for description, but it is not limited to this.

[0049] In this embodiment, after fusing two-dimensional feature information, three-dimensional feature information, and sound feature information to obtain comprehensive feature information, the comprehensive feature information can be input into the prediction model. The prediction model is then used to predict the load of the target vehicle corresponding to the comprehensive feature information, and the load prediction result is obtained.

[0050] Optionally, based on comprehensive feature information, a prediction model is used to predict the load of the target vehicle and obtain the load prediction result. This is one of the key steps in the embodiments of the present invention. It is responsible for converting the comprehensive feature information after the fusion of multimodal data into the load prediction result, providing a direct basis for subsequent overload judgment.

[0051] Optionally, the prediction model in this embodiment of the invention is based on deep learning technology, particularly using a multi-layer perceptron (MLP) or convolutional neural network (CNN) with an attention mechanism. During the model building phase, the network architecture is designed to accommodate multimodal feature inputs, which may include multiple input branches corresponding to image features, laser features, and sound features, respectively, as well as one or more fusion layers for feature fusion processing, and finally an output layer for predicting the load result.

[0052] Optionally, during the training phase, a large dataset containing comprehensive feature information samples and load prediction result samples (actual load value samples) can be used. Comprehensive feature information samples refer to sample data obtained by fusing two-dimensional feature information samples, three-dimensional feature information samples, and sound feature information samples acquired during the driving process of a vehicle carrying the target object sample. Load prediction result samples are the actual load values ​​corresponding to the aforementioned comprehensive feature information samples, used to train the model to learn the mapping relationship from comprehensive feature information to load prediction results.

[0053] Optionally, comprehensive feature information of the target vehicle is acquired and used as input to a pre-trained prediction model. The prediction model utilizes the learned mapping relationship between features and load to output a load prediction result. By leveraging deep learning models, particularly the integration of attention mechanisms, this embodiment of the invention can more effectively process and fuse multimodal feature information, avoiding feature redundancy and information loss, and significantly improving the accuracy of load prediction. Because the comprehensive feature information includes multi-dimensional information from images, lasers, and sound, the prediction model can better adapt to various vehicle types and load conditions, maintaining high prediction performance even under poor lighting conditions, high laser data noise, or complex sound backgrounds.

[0054] In this embodiment of the invention, a deep learning prediction model is used to predict the load based on comprehensive feature information. This effectively combines the advantages of multimodal data, improves the accuracy of load prediction and the adaptability of the model, and provides an efficient and flexible solution for non-contact vehicle overload detection.

[0055] Step S208: Based on the load prediction results, determine the load status of the target vehicle.

[0056] In the technical solution provided by step S208 of the present invention, the load status can be used to indicate whether the target vehicle is in a non-overloaded state or an overloaded state.

[0057] In this embodiment, after predicting the load of the target vehicle using a prediction model based on comprehensive feature information and obtaining the load prediction result, the load status of the target vehicle can be determined based on the load prediction result.

[0058] Optionally, the load prediction results output by the prediction model can be converted into a clear judgment of the load status of the target vehicle, that is, to determine whether the vehicle is overloaded.

[0059] Optionally, the determination of load status can be based on comparing the predicted load result with the maximum permissible load value of the target vehicle. For example, based on the vehicle model, the maximum permissible load value for that model can be obtained from a vehicle load standard database. Vehicle load standard databases typically contain legal load limits for different vehicle models, and this information is crucial for subsequent overload determination. If the predicted load result exceeds the maximum permissible load value, it indicates that the target vehicle is overloaded; conversely, it indicates that the target vehicle is not overloaded.

[0060] It should be noted that the above-described process and method for determining the load status are merely illustrative examples and are not intended to impose specific limitations. Any process or method that can determine whether the total weight of the target object and the target vehicle, as reflected in the load prediction result, is overweight, and thus determines the load status of the target vehicle, is within the protection scope of this invention.

[0061] In this embodiment of the invention, since the load prediction result is calculated based on the comprehensive feature information of multimodal data, its accuracy is higher than that of single-modal data prediction. Therefore, the load status judgment based on this result is also more reliable. The load status judgment method based on multimodal data fusion can adapt to different types of vehicles and accurately judge the load status regardless of different road environments or changing cargo conditions. Load status judgment does not require direct contact with the target vehicle, avoiding the deployment and maintenance costs of contact detection equipment in related technologies, and also reducing interference with normal vehicle operation. By comparing the load prediction result with the vehicle's maximum permissible load value, accurate judgment of the target vehicle's load status is achieved, providing a direct basis for subsequent overload handling and traffic safety management. The above technical solution fully utilizes the advantages of multimodal data fusion, improving the performance of the non-contact vehicle overload detection system, including accuracy, real-time processing capability, and wide adaptability.

[0062] Through steps S202 to S208, if it is necessary to determine the load status of a target vehicle carrying a target object, two-dimensional feature information, three-dimensional feature information, and sound feature information of the target vehicle in the corresponding dimensions can be obtained. Feature fusion can be performed on the two-dimensional feature information, three-dimensional feature information, and sound feature information to obtain comprehensive feature information. This comprehensive feature information can be input into a pre-trained prediction model. Using the comprehensive feature information, the prediction model can predict the load of the target vehicle, obtaining the load prediction result. That is, the prediction model can predict the weight of the current target object and the target vehicle. Based on the load prediction result, it can be determined whether the current target vehicle is overloaded or not. In this embodiment, by combining multimodal data fusion and a deep learning prediction model, flexible and accurate judgment of vehicle load status is achieved in various complex scenarios. The above method can effectively overcome the limitations of related contact detection methods, thereby achieving the technical effect of improving the flexibility of determining the vehicle load status and solving the technical problem of low flexibility in determining the vehicle load status.

[0063] The embodiments of the present invention will now be described in detail with reference to the steps described above.

[0064] As an optional embodiment, step S204 involves fusing the two-dimensional feature information, three-dimensional feature information, and sound feature information to obtain comprehensive feature information. This includes: acquiring the target feature vectors corresponding to the two-dimensional feature information, three-dimensional feature information, and sound feature information respectively, and mapping the target feature vectors into a target feature space. The target feature vectors are obtained by normalizing and standardizing the initial feature vectors corresponding to the two-dimensional feature information, three-dimensional feature information, and sound feature information respectively. In the target feature space, the target feature vectors corresponding to the two-dimensional feature information, three-dimensional feature information, and sound feature information are fused to obtain the comprehensive feature vector corresponding to the comprehensive feature information.

[0065] In this embodiment, during the feature fusion process of two-dimensional feature information, three-dimensional feature information, and sound feature information, target feature vectors corresponding to the two-dimensional feature information, three-dimensional feature information, and sound feature information can be obtained respectively. These target feature vectors can be mapped to a target feature space. In the target feature space, the target feature vectors corresponding to the two-dimensional feature information, three-dimensional feature information, and sound feature information can be fused to obtain a comprehensive feature vector corresponding to the comprehensive feature information. The target feature vector is obtained by normalizing and standardizing the initial feature vectors corresponding to the two-dimensional feature information, three-dimensional feature information, and sound feature information respectively. The target feature space can be a unified feature space to which the mapping is performed.

[0066] Optionally, feature fusion is performed on two-dimensional feature information, three-dimensional feature information, and acoustic feature information to obtain comprehensive feature information. This is specifically detailed in the following steps, which aim to ensure the effective fusion and unified representation of multimodal features to improve the accuracy of subsequent load prediction.

[0067] Optionally, initial feature vectors are extracted from the two-dimensional feature information, three-dimensional feature information, and acoustic feature information, respectively. These initial feature vectors contain key information obtained from the analysis of their respective modal data, such as the distribution of goods in the image, the three-dimensional contour of the laser scan, and the spectral characteristics of the acoustic signal.

[0068] Optionally, in order to eliminate the differences in dimensions and numerical ranges between the initial eigenvectors of different modalities, these initial eigenvectors need to be preprocessed. Preprocessing includes two parts: normalization and standardization.

[0069] Optionally, normalization can be used to scale the values ​​in each feature vector to a uniform range, such as the [0,1] interval.

[0070] For example, the Min-Max normalization method can be used to normalize the initial feature vector (X) of the two-dimensional feature information. image ), the initial feature vector (X) of three-dimensional feature information lidar ), initial feature vector (X) of sound feature information audio Normalization is performed separately, specifically using the following formula:

[0071]

[0072] Among them, X norm X can be used to represent the eigenvector obtained by normalizing the initial eigenvector; X can be used to represent the initial eigenvector; X max It can be used to represent the maximum value of the initial eigenvector; X min It can be used to represent the minimum value of the initial eigenvector.

[0073] Optionally, after normalizing the initial feature vectors, standardization can be performed. The purpose of standardization is to give the features zero mean and unit variance, thereby improving the convergence and prediction accuracy of the model.

[0074] For example, standardization can be achieved using a Z-score, which can be performed using the following formula:

[0075]

[0076] Among them, X zscore It can be used to represent Xnorm The eigenvectors obtained by standardization; μ can be used to represent the mean of the eigenvectors; σ can be used to represent the standard deviation of the eigenvectors.

[0077] Optionally, the preprocessed feature vectors, i.e., the target feature vectors, can be mapped to a unified target feature space. The purpose of the above steps is to represent features from different modalities within the same framework, facilitating subsequent fusion calculations. Mapping to the target feature space can be achieved through feature embedding layers, for example, using multilayer perceptrons (MLPs) or convolutional neural networks (CNNs) to generate new embedding vectors.

[0078] For example, encoding standardized feature vectors through an MLP or convolutional network can generate an embedding vector E in a unified feature space, as shown below. image E lidar E audio (That is, the target feature vector to be fused):

[0079] E modality =MLP(X modality )

[0080] Among them, E image It can be used to represent the target feature vector corresponding to two-dimensional feature information; E lidar It can be used to represent the target feature vector corresponding to three-dimensional feature information; E audio It can be used to represent the target feature vector corresponding to sound feature information; X modality It can be used to represent the pre-processed (normalized and standardized) feature vectors corresponding to specific modalities (such as images, laser data, and sound).

[0081] Optionally, the mapped target feature vectors are fused in the target feature space to generate a comprehensive feature vector. The fusion method can be a weighted summation based on an attention mechanism, or it can use a multimodal fusion network, such as a multimodal autoencoder, a multimodal variational autoencoder, or a modal interaction network. The self-attention mechanism allows the model to dynamically allocate weights based on the importance of features, thus focusing more on certain key features during the fusion process.

[0082] As an optional embodiment, in the target feature space, the target feature vectors corresponding to two-dimensional feature information, three-dimensional feature information, and sound feature information are fused to obtain a comprehensive feature vector corresponding to the comprehensive feature information. This includes: in the target feature space, using a self-attention mechanism, determining the correlation between the two-dimensional feature information, three-dimensional feature information, and sound feature information based on the target feature vector; determining the weights corresponding to the two-dimensional feature information, three-dimensional feature information, and sound feature information respectively based on the correlation; and fusing the target feature vectors based on the weights to obtain a comprehensive feature vector.

[0083] In this embodiment, during the fusion of target feature vectors corresponding to two-dimensional feature information, three-dimensional feature information, and sound feature information in the target feature space, a self-attention mechanism can be used to determine the correlation between the three feature vectors based on the target feature vectors. Weights are then determined for each feature vector based on the correlation, allowing the three target feature vectors to be fused to obtain a fused comprehensive feature vector.

[0084] Optionally, a self-attention mechanism is used in the target feature space to fuse the target feature vectors corresponding to two-dimensional feature information, three-dimensional feature information, and sound feature information to obtain a comprehensive feature vector. The above method aims to enhance the interaction between features and dynamically adjust the weights according to the importance of each modality feature, thereby improving the representativeness and predictive ability of the comprehensive feature vector.

[0085] Optionally, in the target feature space, the self-attention mechanism can be used to calculate the correlation between two-dimensional feature vectors, three-dimensional feature vectors, and sound feature vectors. The self-attention mechanism can capture the dependencies within and between feature vectors, thereby determining which features are more critical to the final load assessment.

[0086] Optionally, during the computation of the self-attention mechanism, each feature vector can be decomposed into a query vector (Q), a key vector (K), and a value vector (V). For feature fusion, these three can be different representations of the same feature vector. The dot product between the query vector and the key vector is calculated to obtain an attention weight matrix. The dot product reflects the similarity between feature vectors. A softmax operation is performed on the attention weight matrix to ensure that the sum of the attention weights is 1, while amplifying the weights of highly relevant features and suppressing low-relevance features. The softmaxed weight matrix is ​​then multiplied by the value vector to obtain the weighted feature representation.

[0087] For example, self-attention can be calculated using the following formula:

[0088]

[0089] Where Q, K, and V can be used to represent the query, key, and value matrices of three modal features: image, laser, and sound; T can be used to represent transpose; d k It can be used to represent the dimension of a key vector.

[0090] Optionally, based on the relevance calculated by the self-attention mechanism, a weight can be assigned to each of the two-dimensional feature vector, three-dimensional feature vector, and sound feature vector. The weights reflect the importance of each modal feature in the overall feature set. The self-attention mechanism automatically learns these weights by calculating the similarity between features across modalities. The weight determination process can be considered part of the model training process, where the weights are optimized to achieve appropriate fusion results and prediction accuracy.

[0091] For example, the weights can be calculated using the following formula:

[0092] α image ,α lidar ,α audio =softmax([E image E lidar E audio ])

[0093] Where, α image It can be used to represent the weights corresponding to two-dimensional feature information; α lidar It can be used to represent the weights corresponding to three-dimensional feature information; α audio It can be used to represent the weights corresponding to sound feature information.

[0094] Optionally, based on determined weights, the target feature vectors are fused to generate a comprehensive feature vector. This process typically uses a weighted summation method.

[0095] For example, the target feature vector can be fused based on weights using the following formula:

[0096] F fused =α image ·E image +α lidar ·E lidar +α audio ·E audio

[0097] Among them, F fused It can be used to represent a comprehensive feature vector.

[0098] Optionally, weights learned through a self-attention mechanism can ensure that key features receive more attention during the fusion process, thereby improving the predictive power of the comprehensive feature vector.

[0099] In this embodiment of the invention, the self-attention mechanism dynamically adjusts weights based on the correlation between feature vectors, thereby focusing more on features that are more important for load prediction during the fusion process and improving prediction accuracy. The self-attention mechanism reveals the interaction relationships between different feature vectors, enabling the model to learn more complex and nuanced feature representations, enhancing its ability to identify vehicle overload conditions. Through weight allocation, even if the data quality of one modality is poor, the model can still rely on high-quality data from other modalities, thereby reducing misjudgments and improving the system's robustness. The self-attention mechanism has high computational efficiency, enabling rapid generation of comprehensive feature vectors when processing real-time data, making it suitable for dynamic vehicle overload detection systems.

[0100] In summary, utilizing a self-attention mechanism to fuse multimodal feature vectors in the target feature space can effectively determine the correlation and weights between features, generating a comprehensive feature vector with strong representativeness and high prediction accuracy. This method not only improves the intelligence level of feature fusion but also enhances the real-time processing capability and robustness of the vehicle overload detection system, representing a key step in realizing non-contact vehicle overload detection technology based on multimodal information.

[0101] As an optional embodiment, the two-dimensional feature information includes the density information of the target object, the model information of the target vehicle, and the tire deformation information of the target vehicle. Step S202, obtaining the density information, includes: identifying the stacking height and coverage area information of the target object from the vehicle image of the target vehicle, and determining the density information based on the stacking height and coverage area information. The image content of the vehicle image includes at least the target vehicle and the target object, and the coverage area information is used to represent the proportion of the target object to the area of ​​the target vehicle. Obtaining the model information includes: identifying the appearance features of the target vehicle from the vehicle image, and determining the model information based on the appearance features. Obtaining the tire deformation information includes: identifying the upper radius and lower radius of the tire of the target vehicle from the vehicle image, and determining the tire deformation information based on the difference between the upper radius and the lower radius. The upper radius is the radius of the tire above the axle, and the lower radius is the radius of the tire below the axle. The tire deformation information is used to represent the degree of deformation caused by the target object to the tire. The method further includes: vectorizing the density information, model information, and tire deformation information to obtain an initial feature vector corresponding to the two-dimensional feature information.

[0102] In this embodiment, the two-dimensional feature information includes the density information of the target object, the vehicle model information of the target vehicle, and the tire deformation information of the target vehicle. Specifically, in acquiring the density information, the stacking height and coverage area information of the target object can be identified from the vehicle image. The density information can be determined based on the stacking height and coverage area information. In acquiring the vehicle model information, the appearance features of the target vehicle can be identified from the vehicle image, and the vehicle model information can be determined based on these appearance features. In acquiring the tire deformation information, the upper and lower radii of the tires of the target vehicle can be identified from the vehicle image. The tire deformation information can be determined based on the difference between the upper and lower radii. The image content of the vehicle image includes at least the target vehicle and the target object. The coverage area information (coverage area) can be used to represent the proportion of the target object to the carrying area of ​​the target vehicle. The carrying area can be the cargo area of ​​a truck or the passenger area of ​​a bus. The density information can be used to represent the density of the stacked goods or the number of people carried; for example, it can be a cargo density vector. Vehicle type information can be used to indicate the type of the target vehicle, such as a small truck, large bus, medium-sized truck, large truck, container truck, etc. This is just an example and not a specific limitation. The upper radius of the tire can be the radius of the tire above the axle. The lower radius can be the radius of the tire below the axle. Tire deformation information can be used to indicate the degree of deformation caused by the target object to the tire, also known as the degree of deformation (compression).

[0103] Optionally, the density information, vehicle model information, and tire deformation information mentioned above can be vectorized to obtain the initial feature vector X corresponding to the two-dimensional feature information. image .

[0104] Optionally, two-dimensional feature information can be obtained through image analysis, including the density information of the target object, the model information of the target vehicle, and the tire deformation information of the target vehicle. The extraction and vectorization of the above features are the fundamental steps for realizing image-based load analysis and overload detection.

[0105] Optionally, image segmentation and object detection techniques are used to identify the outline of the target object (e.g., loaded cargo) from the image of the target vehicle. Based on the cargo outline, the stacking height and coverage area of ​​the cargo are calculated. The stacking height can be determined by measuring the distance from the top outline of the cargo to the vehicle baseline, while the coverage area is the projected area of ​​the cargo in the vehicle's loading area, used to assess the proportion of space occupied by the cargo.

[0106] Optionally, density information can be calculated based on the stacking height and coverage area of ​​the goods. Density information reflects the stacking height of goods per unit area, which helps to assess the load distribution of the goods.

[0107] For example, the cargo density vector can be determined using the following formula:

[0108] Cargo density vector = [stacking height, coverage area]

[0109] Optionally, during the identification of appearance features, deep learning models can be used. For example, these models can identify the appearance features of the target vehicle in the vehicle image (e.g., using the YOLO algorithm), including the vehicle's shape, size, and type. These features are crucial for determining the vehicle's type. Based on the identified appearance features, a pre-trained classification model is used to classify the target vehicle into specific models, such as small trucks, large buses, medium-duty trucks, large trucks, and container trucks. The classification result can be encoded into a vector, and one-hot encoding can be used to convert the classification information into a numerical representation for easier subsequent processing.

[0110] It should be noted that the methods and processes described above for identifying exterior features from vehicle images are merely illustrative examples and are not intended to impose specific limitations. Any method capable of identifying the exterior features of a target vehicle from a vehicle image to determine vehicle model information falls within the protection scope of this invention.

[0111] Optionally, during the analysis of tire deformation information, the tire profile can be analyzed from the vehicle image, and the radii of the tire above and below the axle can be measured separately. The difference between the upper radius (R1) and the lower radius (R2) reflects the degree of tire deformation under load. Based on the difference between the upper and lower radii, the degree of tire deformation can be calculated.

[0112] For example, tire deformation information can be determined using the following formula:

[0113] D = R2 / R1

[0114] Here, D can be used to represent tire deformation information (degree of deformation / compression). The value of this tire deformation information is negatively correlated with the degree of tire deformation. That is, the smaller the ratio, the greater the degree of tire deformation, and the vehicle may be overloaded.

[0115] Optionally, the extracted density information, vehicle type information, and tire deformation information are converted into numerical vector representations, i.e., an initial feature vector is generated. Density information and tire deformation information can be directly encoded into numerical vectors, while vehicle type information is converted into a vector through one-hot encoding. For example, if the target vehicle type is a medium-sized truck, its one-hot encoded vector can be [0,1,0,0,0,0].

[0116] Optionally, these three vectors can be fused into a single comprehensive two-dimensional feature vector (initial feature vector). This vector contains the necessary two-dimensional feature information and can be used as input for subsequent multimodal data fusion and load prediction.

[0117] In this embodiment of the invention, the above method can extract two-dimensional feature information about cargo density, vehicle type, and tire deformation from vehicle images and convert it into a numerical vector representation, providing crucial feature input for subsequent comprehensive analysis. The extraction and vectorization of this feature information is an indispensable component of image-based load analysis and overload detection systems, helping to more accurately assess the vehicle's load status.

[0118] As an optional embodiment, the three-dimensional feature information includes the outline data of the target vehicle and the speed information of the target vehicle. Step S202, acquiring the outline data, includes: scanning the target vehicle using a laser scanning device to acquire point cloud data of the target vehicle; determining the outline data based on the point cloud data; acquiring the speed information, includes: determining the speed information based on the point cloud data; the method further includes: vectorizing the outline data and the speed information to obtain the initial feature vector corresponding to the three-dimensional feature information.

[0119] In this embodiment, during the acquisition of outline data, the target vehicle can be scanned using laser scanning equipment deployed around it to obtain point cloud data. The outline data can then be determined based on the point cloud data. Similarly, during the acquisition of speed information, the speed information can be determined based on the point cloud data.

[0120] Optionally, after acquiring the outline data and velocity information, the two data can be vectorized to obtain the initial feature vector X corresponding to the three-dimensional feature information. lidar .

[0121] Optionally, the three-dimensional feature information of the target vehicle, including the vehicle's outline data and speed information, can be obtained through lidar point cloud data, and this information can be converted into an initial feature vector for subsequent multimodal data fusion and load analysis.

[0122] As an optional embodiment, acquiring the outline data includes: a determination step, which involves determining a point cloud pair from any two frames of point cloud data during the target vehicle's movement, minimizing the distance between the two point clouds in the point cloud pair, and determining a translation transformation matrix and a rotation transformation matrix; an alignment step, which involves aligning the two point clouds in the point cloud pair according to the translation transformation matrix and the rotation transformation matrix, and acquiring the alignment error; in response to the alignment error being greater than an alignment error threshold, returning to the determination step and the alignment step until the alignment error is less than or equal to the alignment error threshold; in response to the alignment error being less than or equal to the alignment error threshold, acquiring the outline data from the aligned point cloud; and acquiring speed information, which includes acquiring a first time point when the target vehicle enters the target area and a second time point when the target vehicle leaves the target area, and determining speed information based on the length information of the target area, the first time point, and the second time point, wherein the target area is the area where the point cloud data is acquired.

[0123] In this embodiment, the three-dimensional feature information may include the outline data and speed information of the target vehicle. During the acquisition of the outline data, a determination step can be performed first. In this determination step, a point cloud pair consisting of two point clouds can be determined from any two frames of point cloud data during the target vehicle's movement. The distance between the two point clouds is minimized, and the corresponding translation and rotation transformation matrices are determined. Then, an alignment step can be performed. In the alignment step, the two point clouds in the point cloud pair are aligned according to the translation and rotation transformation matrices, and the corresponding alignment error is obtained. The relationship between the alignment error and an alignment error threshold can be further determined. If the alignment error is greater than the alignment error threshold, the determination and alignment steps can be repeated until the alignment error is less than or equal to the alignment error threshold. If the alignment error is less than or equal to the alignment error threshold, the outline data can be obtained from the aligned point cloud. During the acquisition of speed information, the first time point when the target vehicle enters the target area and the second time point when the target vehicle leaves the target area can be obtained. Velocity information can be determined based on the length of the target region, the first time point, and the second time point. The target region can be the area from which point cloud data is acquired, for example, the radar scanning range.

[0124] Optionally, during the acquisition of the outline data, the Iterative ClosestPoint (ICP) algorithm can be used to register multiple frames of point cloud data, thereby generating an accurate 3D profile of the target vehicle. Any two frames are selected from the continuous point cloud data during the target vehicle's movement as a point cloud pair. These two frames contain the vehicle's position and shape information at different time points. To map these two point clouds to the same coordinate system, the distance error between the two point clouds needs to be calculated, and this error is minimized by adjusting the translation and rotation parameters. Through iterative optimization, the translation and rotation transformation matrices are finally determined. These two matrices describe the relative movement and rotation between the two point clouds in space. The translation and rotation transformation matrices are applied to align the two point clouds in the point cloud pair to reduce the alignment error. The alignment error is checked to see if it is less than a preset alignment error threshold. If the alignment error is greater than the threshold, the point cloud pair determination and alignment steps are repeated until the error meets the requirements. This ensures the accuracy and reliability of the registration. Once the point cloud data registration is complete, i.e. the alignment error is less than or equal to the threshold, the outline data of the target vehicle, including the vehicle's length, width and height, can be extracted from the aligned point cloud.

[0125] Optionally, in the process of acquiring speed information, in order to determine the speed of the target vehicle within a specific area, it is necessary to acquire the time points when it enters and exits the area, and combine this with the known length information of the area. Record the first time point (t1) when the target vehicle enters the target area (radar scanning range) and the second time point (t2) when it exits the area.

[0126] For example, speed information (vehicle speed) can be determined using the following formula:

[0127]

[0128] Here, v can be used to represent velocity information; d can be used to represent length information, that is, the distance value of the area set within the radar scanning range.

[0129] Optionally, after acquiring the outline data and velocity information, these two types of three-dimensional feature information are vectorized to facilitate subsequent model fusion and processing. For example, the vehicle's length, width, and height dimensions can be encoded into a numerical vector, such as [Length, Width, Height]. The calculated velocity information is converted into an independent numerical vector, such as [Speed]. These vectors are then combined into an initial feature vector of three-dimensional feature information, which can be in the form of X. lidar = [Length, Width, Height, Speed]. The above vector contains the three-dimensional contour dimensions and speed information of the target vehicle, and serves as the input for subsequent fusion analysis and load prediction.

[0130] In this embodiment of the invention, the above method enables efficient and accurate acquisition of the target vehicle's outline data and speed information from the point cloud data of the lidar, and transforms them into an initial feature vector in vector form. This three-dimensional feature information plays a crucial role in the vehicle overload detection system, forming the basis for multimodal data fusion together with two-dimensional image features and sound features, thus facilitating a more comprehensive assessment of the vehicle's load status.

[0131] As an optional embodiment, step S202, acquiring the sound feature information of the target vehicle carrying the target object, includes: acquiring sound signals using a sound sensor during the driving of the target vehicle; performing time-frequency transformation processing on the sound signals to obtain an audio frequency domain signal; determining the peak frequency and peak amplitude based on the audio frequency domain signal, and processing the audio frequency domain signal using a Mel frequency cepstral filter to obtain the corresponding Mel frequency cepstral coefficients; the method further includes: vectorizing the peak frequency, peak amplitude, and Mel frequency cepstral coefficients to obtain an initial feature vector corresponding to the sound feature information.

[0132] In this embodiment, during the acquisition of the acoustic feature information of the target vehicle carrying the target object, an acoustic sensor can be used to collect the acoustic signals generated during the vehicle's movement. Time-frequency transformation processing can be performed on the acoustic signals to obtain the corresponding audio-frequency domain signals. The peak frequency and peak amplitude can be determined based on the audio-frequency domain signals. A Mel-frequency cepstral filter is used to process the audio-frequency domain signals to obtain Mel-frequency cepstral coefficients. The peak frequency, peak amplitude, and Mel-frequency cepstral coefficients can be vectorized to obtain the initial feature vector corresponding to the acoustic feature information.

[0133] Alternatively, the sound signal can be transformed from the time domain to the frequency domain using the following formula:

[0134]

[0135] Where ω can be used to represent frequency; F(ω) can be used to represent the frequency domain signal after transformation from the time domain to the frequency domain; N can be used to represent the number of sampling points; i can be used to represent an imaginary number; and f(t) can be used to represent the sound signal in the time domain.

[0136] Optionally, the main spectral features, primarily including the peak frequency (peak point frequency) f, are extracted from the transformed frequency domain signal. peak Peak amplitude (peak value) A peak .

[0137] As an optional embodiment, acquiring the sound feature information of a target vehicle carrying a target object includes: dividing a sound signal of a first time length acquired during the vehicle's operation into multiple sub-sound signals of a second time length, wherein the second time length is shorter than the first time length; performing windowing processing on each of the multiple sub-sound signals of the second time length to obtain windowed sub-sound signals; converting the windowed sub-sound signals into sub-frequency domain signals using Fourier transform; converting the sub-frequency domain signals into target sub-frequency domain signals and enhancing the target sub-frequency domain signals to obtain enhanced target sub-frequency domain signals, wherein the frequencies of the sub-frequency domain signals are linear frequencies, the frequencies of the target sub-frequency domain signals are nonlinear frequencies, and the degree of difference in nonlinear frequencies in the enhanced target sub-frequency domain signals is greater than the degree of difference in nonlinear frequencies in the unenhanced target sub-frequency domain signals; determining frequency cepstral coefficients based on the target sub-frequency domain signals, and determining sound feature information based on the frequency cepstral coefficients, wherein the sound feature information is used to represent the energy distribution at different nonlinear frequencies; the method further includes: vectorizing at least the frequency cepstral coefficients corresponding to the multiple second time lengths to obtain initial feature vectors corresponding to the sound feature information.

[0138] In this embodiment, during the acquisition of sound feature information, the sound signal of a first time length acquired during the target vehicle's movement can be divided into multiple sub-sound signals of a second time length. Windowing processing can be applied to these multiple sub-sound signals of the second time length to obtain windowed sub-sound signals. Using Fourier transform, the windowed sub-sound signals can be converted into sub-frequency domain signals. These sub-frequency domain signals can be converted into target sub-frequency domain signals, and then enhanced to obtain enhanced target sub-frequency domain signals. Based on the target sub-frequency domain signals, frequency cepstral coefficients can be determined, and sound feature information can be determined based on these frequency cepstral coefficients. The second time length is shorter than the first time length. The frequency of the sub-frequency domain signals is a linear frequency. The frequency of the target sub-frequency domain signals is a non-linear frequency, such as a Mel frequency. The degree of difference in non-linear frequencies in the enhanced target sub-frequency domain signals is greater than the degree of difference in non-linear frequencies in the unenhanced target sub-frequency domain signals. The target sub-frequency domain signal can be the filtered result obtained by filtering through a Mel Frequency Cepstral Coefficients (Mel) filter bank. The frequency cepstral coefficients can be Mel Frequency Cepstral Coefficients (MFCC) coefficients. Windowing can be applied to each frame of the sub-audio signal of the second time length using a Hamming window, or other types of window functions; no specific restrictions are placed here.

[0139] Optionally, at least the frequency cepstral coefficients corresponding to multiple second time lengths can be vectorized to obtain the initial feature vector X corresponding to the sound feature information. audio .

[0140] Optionally, acquiring sound feature information involves converting the sound signal of the target vehicle during its operation into frequency cepstral coefficients (MFCCs) and vectorizing them for subsequent multimodal data fusion and load analysis.

[0141] Optionally, the long-duration sound signal (first time length) acquired during the target vehicle's movement can be divided into multiple shorter sub-sound signals (second time length) to facilitate the capture of the instantaneous features of the sound signal. The selection of the segment length (second time length) needs to balance the stability of the features and the sensitivity to changes, and is usually chosen to be from tens of milliseconds to hundreds of milliseconds.

[0142] Optionally, each segmented sub-audio signal can be windowed, typically using HammingWindow or other types of window functions. The purpose of windowing is to reduce spectral leakage caused by abrupt start and end of the signal, thereby improving the accuracy of spectral analysis.

[0143] For example, the window function takes the form of:

[0144]

[0145] Where ω(n) can be used to represent the value of the window function; n can be used to represent the index of the window function, that is, the sampling point position of the signal within the window; N can be used to represent the length of the window function, that is, the length of the windowed signal.

[0146] Optionally, a Fast Fourier Transform (FFT) can be performed on the windowed sub-sound signal to convert the time-domain signal into a frequency-domain signal, in order to analyze the spectral characteristics of the sound signal. The FFT can reveal the frequency components of the signal, which is crucial for extracting sound features.

[0147] Optionally, the frequency domain signal is transformed from a linear frequency scale to a Mel frequency scale. The Mel frequency scale is closer to the human ear's perception of different frequencies. The spectrum is then filtered using a Mel filter bank to extract the energy within the Mel frequency bands. The Mel filter bank divides the spectrum into multiple nonlinear frequency intervals, each corresponding to a Mel filter. The filtering results reflect the energy distribution of the signal within different Mel frequency bands.

[0148] For example, a linear frequency can be converted into a nonlinear frequency (such as the MEL frequency) using the following formula:

[0149]

[0150] Among them, f mel It can be used to represent the Mel frequency of the target sub-frequency domain signal after frequency conversion; f can represent the linear frequency domain of the sub-frequency domain signal.

[0151] Optionally, enhancement processing refers to performing logarithmic transformation and difference calculations on the extracted Mel spectrum to enhance the contrast between different Mel frequencies. For example, logarithmic transformation can compress high-energy regions, making the energy distribution more uniform. Difference calculations can capture the changes in the spectrum over time, improving the dynamic representation capability of features. It should be noted that the above method of enhancing differences through logarithmic operations is only an example and is not a specific limitation.

[0152] Optionally, a Discrete Cosine Transform (DCT) is performed on the enhanced Mel spectrum to generate a set of frequency cepstral coefficients (MFCCs). DCT is a spectral analysis tool that redistributes the energy of a signal onto a set of orthogonal cepstral coefficients, where the lower-order coefficients typically carry most of the signal's energy and characteristic information.

[0153] Optionally, the MFCC coefficients corresponding to each sub-sound signal are transformed into a vector, represented as C = [c1, c2, ..., c k ], where c i This represents the i-th MFCC coefficient. The MFCC vectors corresponding to all sub-sound signals are combined or summarized to generate the initial feature vector of the sound feature information. It should be noted that if the sound feature information includes features other than frequency, these can also be vectorized and used to form the initial feature vector corresponding to the sound feature information.

[0154] In this embodiment of the invention, the above method can efficiently extract key frequency cepstral coefficient (MFCC) features from the driving sound of a target vehicle, vectorize them, and provide sound modality feature input for subsequent multimodal data fusion. MFCC features have proven to be very effective in speech recognition and audio analysis, capable of capturing frequency information in sound signals that is sensitive to human hearing.

[0155] In this embodiment of the invention, a method is provided to determine whether a target vehicle is overloaded by comparing the predicted load of the target vehicle with the target load threshold corresponding to the target vehicle. The specific analysis is as follows:

[0156] As an optional embodiment, step S208, determining the load status of the target vehicle based on the load prediction result, includes: determining the load status of the target vehicle based on the load prediction result, including: determining the target load threshold corresponding to the target vehicle based on the vehicle model information; determining the load status as an overload state in response to the overload prediction result being greater than the target load threshold; and determining the load status as a non-overload state in response to the load prediction result being less than or equal to the target load threshold.

[0157] In this embodiment, during the process of determining the load status of the target vehicle based on the load prediction result, the target load threshold for the corresponding vehicle model is determined according to the vehicle model information identified from the vehicle image of the target vehicle. The relationship between the overload prediction result and the target load threshold can be used to determine whether the target vehicle is overloaded or not. The target load threshold, determined using the vehicle model information, can represent the maximum load-bearing capacity of the target vehicle in a non-overloaded state; it can also be referred to as the vehicle's maximum load capacity or standard load.

[0158] Optionally, if the overload prediction result is greater than the target load threshold, it indicates that the target vehicle is overloaded. If the load prediction result is less than or equal to the target load threshold, it indicates that the target vehicle is not overloaded.

[0159] In this embodiment of the invention, a method for determining whether a target vehicle is in an overloaded state based on load reset confidence is also proposed, and the specific analysis is as follows:

[0160] As an optional embodiment, step S208, based on the load prediction result, determines the load status of the target vehicle, including: based on the load prediction result and the target load threshold corresponding to the target vehicle, determining the overload confidence of the target vehicle, wherein the target load threshold is used to represent the maximum load-bearing capacity of the target vehicle in a non-overloaded state; and determining the load status based on the overload confidence.

[0161] In this embodiment, during the process of determining the load status of the target vehicle based on the load prediction results, the overload confidence level of the target vehicle can be determined based on the load prediction results and the target load threshold corresponding to the target vehicle. The load status can be determined based on the overload confidence level.

[0162] Optionally, the load prediction result is calculated using the multimodal data fusion and deep learning model from previous steps. It reflects the vehicle load estimate obtained after comprehensive calculation based on image, laser, and sound information. The target load threshold refers to the maximum permissible load capacity of the target vehicle under normal, non-overloaded conditions, which is typically based on the vehicle type, design, and regulations. For example, the standard load capacity of a medium-duty truck might be 10 tons, while the standard load capacity of a large truck might reach 20 tons or more. This threshold is determined by the vehicle type information and provides a clear reference point for judging whether the load exceeds the prescribed limits. The overload confidence score is calculated by comparing the load prediction result with the target load threshold. It represents the degree of deviation of the load prediction result from the standard load and the likelihood that this deviation is caused by overloading.

[0163] For example, the overload confidence level can be determined using the following formula:

[0164]

[0165] Where C overload W can be used to represent the overload confidence level; W can be used to represent the load prediction result, that is, the vehicle weight output by the multilayer perceptron of the attention mechanism; W max It can be used to represent the target load threshold, and can be obtained from the standard library by information such as vehicle model and outline data.

[0166] Optionally, the load status of the target vehicle can be determined based on the calculated overload confidence level. If the overload confidence level exceeds a preset threshold, the system will determine that the vehicle is overloaded; otherwise, if the overload confidence level is lower than or equal to this threshold, the system will determine that the vehicle is under normal load.

[0167] In this embodiment of the invention, the overload confidence level is calculated by comparing the predicted load with the standard load of the target vehicle, thereby determining whether the vehicle is overloaded. This method provides a quantitative indicator to assess overload conditions, which is particularly important for non-contact vehicle overload detection systems because it is based on the comprehensive analysis of multimodal information, improving the accuracy and reliability of the judgment.

[0168] As an optional embodiment, determining the load state based on the overload confidence level includes: determining the load state as an overload state in response to the overload confidence level being greater than the overload confidence level threshold; and determining the load state as a non-overload state in response to the overload confidence level being less than or equal to the overload confidence level threshold.

[0169] In this embodiment, during the process of determining the load status based on the overload confidence level, the relationship between the overload confidence level and the overload confidence threshold can be determined. If the overload confidence level is greater than the overload confidence threshold, the load status can be determined to be an overloaded state. If the overload confidence level is less than or equal to the overload confidence threshold, the load status can be determined to be a non-overloaded state.

[0170] Optionally, in this alternative embodiment, the process of determining the vehicle's load status based on the overload confidence level involves a key judgment criterion—the overload confidence threshold. By comparing the overload confidence level with this threshold, an accurate judgment can be made as to whether the vehicle is overloaded.

[0171] Optionally, an overload confidence score for the target vehicle is calculated. This score represents the relative deviation between the predicted load and the vehicle's specified load threshold, and the likelihood that this deviation is interpreted as overloading. The overload confidence score can be compared to a preset overload confidence threshold. This threshold, determined in advance during the design phase, represents an acceptable range of deviation; exceeding this range indicates a risk of overloading.

[0172] Optionally, if the overload confidence level is greater than the overload confidence threshold, the target vehicle's load status can be determined to be overloaded. This means that the deviation between the predicted load and the standard load exceeds the system's expected safety range, and the vehicle may be carrying goods exceeding its design load, thus posing a potential threat to vehicle safety, road safety, and other traffic participants. Conversely, if the overload confidence level is less than or equal to the overload confidence threshold, the target vehicle's load status can be determined to be not overloaded. This indicates that the deviation between the vehicle's predicted load and the standard load is within an acceptable range, and the vehicle's load is at a safe and legal level.

[0173] Optionally, setting the overload confidence threshold is a crucial part of the system design. This threshold is typically based on multiple factors, including vehicle type, load regulations, road conditions, safety standards, and system accuracy. For example, in applications with high load control requirements, the threshold may be set lower to ensure timely detection even for minor overloads; while in scenarios with less stringent accuracy requirements, the threshold can be set relatively higher to reduce false alarm rates. The decision-making process of comparing the overload confidence level with the threshold ensures the reliability of the load status judgment. By combining the load prediction results obtained through multimodal data fusion with a rigorously set overload confidence threshold, misjudgments caused by fluctuations or errors in a single data source can be effectively avoided, improving the overall system stability and accuracy.

[0174] Optionally, once the system determines the vehicle's load status, it will take appropriate measures. If the vehicle is overloaded, the system may trigger an alarm, record violation information, or send a notification to relevant departments; if the vehicle is not overloaded, the system will allow it to continue driving normally.

[0175] In this embodiment of the invention, the above method, based on the relationship between overload confidence level and overload confidence threshold, enables timely and accurate judgment of load status, providing strong support for road safety and traffic management. This embodiment demonstrates the practicality of the non-contact vehicle overload detection system in complex traffic environments, and its importance in reducing road accidents and improving traffic efficiency.

[0176] It should be noted that the above-described process and method for determining whether a target vehicle is overloaded are merely illustrative examples and are not intended to impose specific limitations. Any process or method that can comprehensively measure whether a target vehicle is overloaded from multiple modal perspectives—two-dimensional space, three-dimensional space, and acoustic space—is within the scope of protection of this invention.

[0177] In this embodiment of the invention, if it is necessary to determine the load status of a target vehicle carrying a target object, two-dimensional feature information, three-dimensional feature information, and sound feature information of the target vehicle in the corresponding dimensions can be obtained. The two-dimensional feature information, three-dimensional feature information, and sound feature information can be fused to obtain comprehensive feature information. This comprehensive feature information can be input into a pre-trained prediction model. Using the comprehensive feature information, the prediction model can predict the load of the target vehicle, obtaining a load prediction result. That is, the prediction model can predict the weight of the current target object and the target vehicle. Based on the load prediction result, it can be determined whether the current target vehicle is overloaded or not. In this embodiment, by combining multimodal data fusion and a deep learning prediction model, flexible and accurate judgment of the vehicle's load status is achieved in various complex scenarios. The above method can effectively overcome the limitations of related contact detection methods, thereby achieving the technical effect of improving the flexibility of determining the vehicle's load status and solving the technical problem of low flexibility in determining the vehicle's load status.

[0178] The following describes in detail another optional implementation method.

[0179] Currently, with the development of the transportation industry, safety issues caused by vehicle overloading have gradually become a focus of attention for countries around the world. Most existing methods for detecting vehicle overloading rely on contact-based sensing devices, typically including ground-mounted weighing equipment (such as weighing platforms and sensors). While these methods can accurately measure vehicle weight to a certain extent, they have several drawbacks in practical applications, including: high deployment costs (installation of contact-based equipment usually requires significant engineering investment and has high equipment costs); high maintenance and operating costs (weighing equipment is easily damaged and requires regular maintenance, increasing operating costs); and limited installation locations (such equipment must be installed in specific locations, making it difficult to meet the needs of mobile or remote detection).

[0180] In summary, contact weighing platforms and similar equipment have certain limitations in terms of cost, maintenance, applicability, and efficiency. Therefore, the technical problem of low flexibility in determining the load status of vehicles remains.

[0181] This invention proposes a non-contact vehicle overload detection method based on the comprehensive calculation of image, laser, and sound information. This method overcomes the problems of high cost, complex maintenance, and limited detection location associated with traditional detection methods by combining multimodal information such as images, lasers, and sound, achieving a highly efficient, flexible, and low-cost overload detection system. In this embodiment, the fusion of multimodal data (i.e., two-dimensional feature information, three-dimensional feature information, and sound feature information) can complementaryly capture the features reflected in the load state from multiple dimensions, improving the accuracy and comprehensiveness of the judgment. Furthermore, since the prediction model is trained on a large number of samples, it can learn the complex intrinsic relationships between various features, accurately reflecting the vehicle's load state even with minor feature changes, thus improving prediction accuracy. In summary, the combination of multimodal data fusion and a deep learning prediction model enables flexible and accurate judgment of vehicle load state in various complex scenarios. This method effectively overcomes the limitations of related contact detection methods, thereby improving the technical effect of increasing the flexibility of determining the vehicle's load state and solving the technical problem of low flexibility in determining the vehicle's load state.

[0182] The method will be further described below.

[0183] In this embodiment, during the acquisition and feature extraction of multimodal data, image data acquisition and feature extraction, laser data acquisition and feature extraction, and sound data acquisition and feature extraction can be performed.

[0184] Optionally, during the image data acquisition and feature extraction process, a camera device is used to acquire vehicle images, and a neural network (e.g., YOLO) is used to extract the main image features of the vehicle, such as cargo density estimation, vehicle type analysis, and tire deformation analysis.

[0185] Optionally, the stacking and distribution of cargo on the truck roof can be identified using an image segmentation algorithm to assess cargo density. After detecting the cargo using a YOLO network, its stacking height and coverage area are calculated. The cargo height is the ratio of the cargo's vertical outline to the baseline in the image, and the coverage area is the proportion of the cargo to the truck's cargo-carrying area. Finally, the cargo density index is encoded as a vector, with the formula: Cargo Density Vector = [Stacking Height, Coverage Area].

[0186] Optionally, Table 1 is a coding vector table corresponding to vehicle type classification in an embodiment of the present invention. As shown in Table 1, YOLO image recognition can classify the vehicle type while detecting the target vehicle. Vehicles can be roughly divided into several categories, such as passenger and freight vehicles. Freight vehicles can be further classified into box trucks, etc. Since vehicle type classification is discrete data, one-hot encoding is used to vectorize it into a numerical vector. For example, the coding vector for a medium-sized freight vehicle is [0,1,0,0,0,0].

[0187] Table 1. Encoding vector table for a vehicle type classification

[0188] small truck 1 0 0 0 0 0 large passenger bus 0 1 0 0 0 0 medium-sized trucks 0 0 1 0 0 0 large trucks 0 0 0 1 0 0 extra-large trucks 0 0 0 0 1 0 Container truck 0 0 0 0 0 1

[0189] Optionally, Figure 3 This is a schematic diagram of the upper and lower radii of a tire according to an embodiment of the present invention, as shown below. Figure 3 As shown, this method identifies tire deformation and infers the impact of load on the vehicle structure. In tire deformation detection, the symmetry between the top and bottom of the axle is used to assess tire deformation under load. We can determine whether the tire has undergone significant compressive deformation under load by analyzing the difference between the upper and lower radii of the tire. R1 is the tire radius above the axle, which can be measured by the distance between the top contour perpendicular to the axle and the axle center in the image. R2 is the tire radius below the axle, measured similarly by the distance from the lower edge of the tire to the axle center in the image.

[0190] For example, tire deformation information can be determined using the following formula:

[0191] D = R2 / R1

[0192] Here, D can be used to represent tire deformation information (degree of deformation / compression). The value of this tire deformation information is negatively correlated with the degree of tire deformation. That is, the smaller the ratio, the greater the degree of tire deformation, and the vehicle may be overloaded.

[0193] Optionally, the features obtained from the image can be grouped into a vector, called X. imgae .

[0194] Optionally, during the acquisition and feature extraction of laser data, a laser scanning device (such as a lidar) is used to obtain the three-dimensional contour information of the vehicle, and the following features are extracted: three-dimensional contour features and velocity features.

[0195] Optionally, point cloud data of the vehicle and cargo are generated, their volume is analyzed, and the vehicle's outline data is obtained. In the outline calculation, LiDAR is used to acquire point cloud data of the vehicle and its cargo. Multiple frames of point clouds are registered to reconstruct the vehicle outline. In this invention, the ICP algorithm is used for point cloud registration to align multiple frames of data. The steps of the ICP algorithm are: 1) Select corresponding point pairs: In two frames of point clouds, find the closest point pairs; 2) Calculate the transformation matrix: By minimizing the distance between corresponding point pairs, determine the translation and rotation transformation matrices; 3) Iterative optimization: Repeat the calculation of the transformation matrix to gradually align the multiple frames of point clouds until the error converges.

[0196] Optionally, the registered point clouds are combined to form a complete vehicle outline, thereby obtaining the accurate outline dimensions of the vehicle. This includes the vehicle's length, width, and height information.

[0197] Optionally, vehicle speed information can be accurately obtained by tracking the vehicle's travel path. Obtaining the speed first requires setting up an area within the radar scanning range, with a known distance value d. During the laser tracking of the vehicle, the time it enters the area is recorded as t1, and the time it leaves is recorded as t2, thus calculating the vehicle speed v.

[0198] For example, speed information (vehicle speed) can be determined using the following formula:

[0199]

[0200] Optionally, the features obtained by the laser can be grouped into a vector, called X. lidar .

[0201] Optionally, during the acquisition and feature extraction of sound data, the sound characteristics of the vehicle while it is in motion are recorded by a sound acquisition device installed near the vehicle, which mainly includes: Fast Fourier Transform (FFT) analysis and Mel frequency cepstral coefficient (MFCC) feature extraction.

[0202] Optionally, during the Fast Fourier Transform (FFT) analysis, the acquired time-domain signal is subjected to a Fast Fourier Transform to convert it into a frequency-domain signal in order to extract spectral information.

[0203] Alternatively, the sound signal can be transformed from the time domain to the frequency domain using the following formula:

[0204]

[0205] Where ω can be used to represent frequency; F(ω) can be used to represent the frequency domain signal after transformation from the time domain to the frequency domain; N can be used to represent the number of sampling points; i can be used to represent an imaginary number; and f(t) can be used to represent the sound signal in the time domain.

[0206] Optionally, the main spectral features, primarily including the peak frequency (peak point frequency) f, are extracted from the transformed frequency domain signal. peak Peak amplitude (peak value) A peak .

[0207] Optionally, in the Mel frequency cepstral coefficient (MFCC) feature extraction process, MFCC obtains frequency characteristics that are closer to human hearing by decomposing the sound signal into different frequency components, which is suitable for detecting subtle sound changes.

[0208] Optionally, the audio signal can be divided into multiple short frames to capture changes in a shorter time.

[0209] Optionally, a Hamming window is applied to each frame to reduce edge artifacts, with the window function being:

[0210]

[0211] Where ω(n) can be used to represent the value of the window function; n can be used to represent the index of the window function, that is, the sampling point position of the signal within the window; N can be used to represent the length of the window function, that is, the length of the windowed signal.

[0212] Optionally, each frame can be subjected to a Fast Fourier Transform to convert it to the frequency domain.

[0213] Optionally, the frequency domain signal is passed through a Mel filter bank to obtain a frequency band distribution that conforms to human auditory perception. Mel frequency f mel The conversion formula with linear frequency f is:

[0214]

[0215] Among them, f mel It can be used to represent the Mel frequency of the target sub-frequency domain signal after frequency conversion; f can represent the linear frequency domain of the sub-frequency domain signal.

[0216] Optionally, the logarithm of the filtering result can be taken to enhance the difference.

[0217] Optionally, the results can be subjected to DCT transformation to generate MFCC coefficients. These coefficients form an eigenvector, representing the energy distribution of sound at different frequencies.

[0218] Optionally, the MFCC coefficients corresponding to each sub-sound signal are transformed into a vector, represented as C = [c1, c2, ..., c k ], where c i This represents the i-th MFCC coefficient.

[0219] Optionally, the features obtained from the audio information can be grouped into a vector, called X. audio .

[0220] In this embodiment, during the fusion and load analysis of multimodal features, the feature vector X of each mode is... image ,X lidar ,X audio Perform Min-Max normalization on each, mapping them to the interval [0,1]:

[0221]

[0222] Among them, X normX can be used to represent the eigenvector obtained by normalizing the initial eigenvector; X can be used to represent the initial eigenvector; X max It can be used to represent the maximum value of the initial eigenvector; X min It can be used to represent the minimum value of the initial eigenvector.

[0223] Optionally, after normalizing the initial feature vectors, standardization can be performed. Standardization can be achieved using the Z-score, specifically using the following formula:

[0224]

[0225] Among them, X zscore It can be used to represent X norm The eigenvectors obtained by standardization; μ can be used to represent the mean of the eigenvectors; σ can be used to represent the standard deviation of the eigenvectors.

[0226] Alternatively, encoding the standardized feature vectors through an MLP or convolutional network can generate an embedding vector E in a unified feature space, as shown in the following form. image E lidar E audio (That is, the target feature vector to be fused):

[0227] E modality =MLP(X modality )

[0228] Among them, E image It can be used to represent the target feature vector corresponding to two-dimensional feature information; E lidar It can be used to represent the target feature vector corresponding to three-dimensional feature information; E audio It can be used to represent the target feature vector corresponding to sound feature information; X modality It can be used to represent the pre-processed (normalized and standardized) feature vectors corresponding to specific modalities (such as images, laser data, and sound).

[0229] In this embodiment, during the multimodal feature fusion calculation process, a complementary fusion method is used in a unified feature space to perform multimodal data fusion calculation and generate a comprehensive feature vector.

[0230] Optionally, a self-attention mechanism can be used to achieve complementary fusion of information between modalities, enabling features from different modalities to complement each other. The self-attention calculation is as follows:

[0231]

[0232] Where Q, K, and V can be used to represent the query, key, and value matrices of three modal features: image, laser, and sound; T can be used to represent transpose; d k It can be used to represent the dimension of a key vector.

[0233] Optionally, weights are calculated for each modal feature, prioritizing the extraction of important information and reducing redundancy. For example, vehicle outline information (laser data) may have a higher weight in load estimation, while sound data may have a lower weight in some cases. The weight allocation is illustrated below:

[0234] α image ,α lidar ,α audio =softmax([E image E lidar E audio ])

[0235] Where, α image It can be used to represent the weights corresponding to two-dimensional feature information; α lidar It can be used to represent the weights corresponding to three-dimensional feature information; α audio It can be used to represent the weights corresponding to sound feature information.

[0236] Optionally, the final generated comprehensive feature vector F fused Represented as:

[0237] F fused =α image ·E image +α lidar ·E lidar +α audio ·E audio

[0238] Optionally, a deep learning model can be used for load prediction during the load prediction and overload judgment process. Specifically, a multilayer perceptron based on an attention mechanism can be used to predict the load based on the final fused features, where the fused features F... fused The vehicle weight W is taken as input and output.

[0239] Optionally, the overload confidence level C is calculated by comparing the predicted load with the vehicle's standard load. overload And set a threshold to determine if it is overloaded:

[0240]

[0241] Where C overload W can be used to represent the overload confidence level; W can be used to represent the load prediction result, that is, the vehicle weight output by the multilayer perceptron of the attention mechanism; W max It can be used to represent the target load threshold.

[0242] Optionally, when the overload confidence level is greater than a threshold, it can be determined that the vehicle is at risk of being overloaded.

[0243] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0244] This embodiment also provides a device for determining the load status of a vehicle. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0245] Figure 4 This is a structural block diagram of a vehicle load state determination device according to an embodiment of the present invention, as shown below. Figure 2 As shown, the device includes: an acquisition unit 402, a fusion unit 404, a prediction unit 406, and a determination unit 408.

[0246] The acquisition unit 402 is used to acquire two-dimensional feature information, three-dimensional feature information, and sound feature information of the target vehicle carrying the target object.

[0247] The fusion unit 404 is used to fuse two-dimensional feature information, three-dimensional feature information and sound feature information to obtain comprehensive feature information.

[0248] The prediction unit 406 is used to predict the load of the target vehicle based on comprehensive feature information and using a prediction model to obtain the load prediction result.

[0249] The determination unit 408 is used to determine the load status of the target vehicle based on the load prediction results.

[0250] The vehicle load status determination device provided in this embodiment of the invention acquires two-dimensional feature information, three-dimensional feature information, and sound feature information of a target vehicle carrying a target object through an acquisition unit 402; performs feature fusion on the two-dimensional feature information, three-dimensional feature information, and sound feature information through a fusion unit 404 to obtain comprehensive feature information; predicts the load of the target vehicle based on the comprehensive feature information using a prediction model through a prediction unit 406 to obtain a load prediction result; and determines the load status of the target vehicle based on the load prediction result through a determination unit 408. This solves the technical problem of low flexibility in determining the load status of a vehicle and achieves the technical effect of improving the flexibility in determining the load status of a vehicle.

[0251] The following is a detailed description of the vehicle load determination device described in the embodiments of the present invention.

[0252] As an optional embodiment, the device further includes: an image acquisition device for acquiring a vehicle image of the target vehicle while the target vehicle is in motion; a laser scanning device for acquiring point cloud data of the target vehicle while the target vehicle is in motion; a sound acquisition device for acquiring sound signals of the target vehicle while the target vehicle is in motion; and a data processing unit for determining two-dimensional feature information based on the vehicle image, determining three-dimensional feature information based on the point cloud data, and determining sound feature information based on the sound signal in response to receiving the vehicle image, point cloud data, and sound signal.

[0253] It should be noted that the above-mentioned units can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above-mentioned units are located in the same processor; or, the above-mentioned units are located in different processors in any combination.

[0254] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0255] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0256] Figure 5 This is a schematic diagram of the structure of an electronic device for determining the load state of a vehicle according to an embodiment of the present invention, as shown below. Figure 5As shown, embodiments of the present invention also provide an electronic device 500, including a processor 501 and a memory 502, wherein the memory 502 stores a computer program, and the processor 501 is configured to run the computer program to perform the steps in any of the above method embodiments.

[0257] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0258] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0259] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0260] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for determining the load condition of a vehicle, characterized in that, include: The method acquires two-dimensional feature information, three-dimensional feature information, and sound feature information of a target vehicle carrying a target object. The two-dimensional feature information is used to represent at least the features of the target vehicle and the target object in two-dimensional space, the three-dimensional feature information is used to represent at least the features of the target vehicle and the target object in three-dimensional space, and the sound feature information is used to represent the features of the sound generated by the target vehicle during its operation. The two-dimensional feature information, the three-dimensional feature information, and the sound feature information are fused to obtain comprehensive feature information; Based on the comprehensive feature information, the load of the target vehicle is predicted using a prediction model to obtain the load prediction result. Based on the load prediction results, the load status of the target vehicle is determined, wherein the load status indicates whether the target vehicle is in a non-overloaded or overloaded state.

2. The method for determining the load status of a vehicle according to claim 1, characterized in that, The two-dimensional feature information, the three-dimensional feature information, and the sound feature information are fused to obtain comprehensive feature information, including: Obtain the target feature vectors corresponding to the two-dimensional feature information, the three-dimensional feature information, and the sound feature information respectively, and map the target feature vectors into the target feature space. The target feature vectors are obtained by normalizing and standardizing the initial feature vectors corresponding to the two-dimensional feature information, the three-dimensional feature information, and the sound feature information respectively. In the target feature space, the target feature vectors corresponding to the two-dimensional feature information, the three-dimensional feature information, and the sound feature information are fused to obtain the comprehensive feature vector corresponding to the comprehensive feature information.

3. The method for determining the load state of a vehicle according to claim 2, characterized in that, In the target feature space, the target feature vectors corresponding to the two-dimensional feature information, the three-dimensional feature information, and the sound feature information are fused to obtain the comprehensive feature vector corresponding to the comprehensive feature information, including: In the target feature space, using a self-attention mechanism, the correlation between the two-dimensional feature information, the three-dimensional feature information, and the sound feature information is determined based on the target feature vector; Based on the relevance, the weights corresponding to the two-dimensional feature information, the three-dimensional feature information, and the sound feature information are determined respectively. Based on the weights, the target feature vectors are fused to obtain the comprehensive feature vector.

4. The method for determining the load status of a vehicle according to claim 1, characterized in that, The two-dimensional feature information includes the density information of the target object, the vehicle model information of the target vehicle, and the tire deformation information of the target vehicle. Obtaining the density information includes: From the vehicle image of the target vehicle, the stacking height information and coverage area information of the target object are identified, and the density information is determined based on the stacking height information and the coverage area information. The image content of the vehicle image includes at least the target vehicle and the target object, and the coverage area information is used to represent the proportion of the target object in the loading area of ​​the target vehicle. Obtaining the vehicle model information includes: identifying the appearance features of the target vehicle from the vehicle image, and determining the vehicle model information based on the appearance features; Obtaining the tire deformation information includes: identifying the upper radius and lower radius of the tire of the target vehicle from the vehicle image, and determining the tire deformation information based on the difference between the upper radius and the lower radius, wherein the upper radius is the radius of the tire above the axle, the lower radius is the radius of the tire below the axle, and the tire deformation information is used to represent the degree of deformation of the tire; The method further includes: vectorizing the density information, the vehicle model information, and the tire deformation information to obtain the initial feature vector corresponding to the two-dimensional feature information.

5. The method for determining the load status of a vehicle according to claim 1, characterized in that, The three-dimensional feature information includes the outline data of the target vehicle and the speed information of the target vehicle, wherein obtaining the outline data includes: The target vehicle is scanned using a laser scanning device to obtain point cloud data of the target vehicle; Based on the point cloud data, the outline data is determined; Obtaining the speed information includes: determining the speed information based on the point cloud data; The method further includes: vectorizing the outline data and the velocity information to obtain the initial feature vector corresponding to the three-dimensional feature information.

6. The method for determining the load state of a vehicle according to claim 1, characterized in that, Obtain the acoustic characteristic information of the target vehicle carrying the target object, including: During the movement of the target vehicle, sound signals are acquired using a sound sensor; The sound signal is subjected to time-frequency transformation processing to obtain an audio frequency domain signal; Based on the acoustic frequency domain signal, the peak frequency and peak amplitude are determined, and the acoustic frequency domain signal is processed using a Mel frequency cepstral filter to obtain the corresponding Mel frequency cepstral coefficients. The method further includes: vectorizing the peak frequency, the peak amplitude, and the Mel frequency cepstral coefficients to obtain an initial feature vector corresponding to the sound feature information.

7. The method for determining the load status of a vehicle according to claim 1, wherein determining the load status of the target vehicle based on the load prediction result includes: Based on the vehicle model information of the target vehicle, determine the target load threshold corresponding to the target vehicle; The load prediction result is compared with the target load threshold; In response to the load prediction result being greater than the target load threshold, the load state is determined to be the overload state; In response to the load prediction result being less than or equal to the target load threshold, the load state is determined to be the non-overloaded state.

8. A device for determining the load status of a vehicle, characterized in that, include: The acquisition unit is used to acquire two-dimensional feature information, three-dimensional feature information, and sound feature information of a target vehicle carrying a target object. The two-dimensional feature information is used to represent at least the features of the target vehicle and the target object in two-dimensional space, the three-dimensional feature information is used to represent at least the features of the target vehicle and the target object in three-dimensional space, and the sound feature information is used to represent the features of the sound generated by the target vehicle during driving. The fusion unit is used to perform feature fusion on the two-dimensional feature information, the three-dimensional feature information, and the sound feature information to obtain comprehensive feature information; The prediction unit is used to predict the load of the target vehicle based on the comprehensive feature information and using a prediction model to obtain the load prediction result. The determining unit is used to determine the load status of the target vehicle based on the load prediction result, wherein the load status is used to indicate whether the target vehicle is in a non-overloaded state or an overloaded state.

9. The device for determining the load status of a vehicle according to claim 8, characterized in that, The device further includes: An image acquisition device is used to acquire vehicle images of the target vehicle during its movement. A laser scanning device is used to acquire point cloud data of the target vehicle during its movement. A sound acquisition device is used to acquire the sound signal of the target vehicle during its operation. A data processing unit is configured to, in response to receiving the vehicle image, the point cloud data, and the sound signal, determine the two-dimensional feature information based on the vehicle image, determine the three-dimensional feature information based on the point cloud data, and determine the sound feature information based on the sound signal.

10. A processor, characterized in that, The processor is used to run a program, wherein the program is executed by the processor to perform the method according to any one of claims 1 to 7.

Images