Multimodal vehicle weighing method and device based on vehicle-road cloud

Abstract

The present application provides a multimodal vehicle weighing method and device based on vehicle-road cloud, which relates to the field of vehicle weighing. The method includes: in response to the dynamic weighing operation for the vehicle to be tested, the first vehicle information, the second vehicle information and the third vehicle information of the vehicle to be tested are respectively obtained through vehicle identification technology, radar speed measurement technology and optical fiber sensing technology; the vehicle characteristic map corresponding to the vehicle to be tested is obtained through vehicle identification technology; a vehicle map neural network model is constructed according to the first vehicle information and the vehicle characteristic map, a speed map neural network model is constructed according to the first vehicle information, the second vehicle information and the vehicle characteristic map, and a quality map neural network model is constructed according to the first vehicle information, the third vehicle information and the vehicle characteristic map; a multimodal fusion network model is constructed through the vehicle map neural network model, the speed map neural network model and the quality map neural network model, and the actual weight of the vehicle to be tested is calculated according to the multimodal fusion network model. The present application solves the problem that the vehicle weighing accuracy is still low in the weighing system when the vehicle is weighed only by identifying the deformation characteristics of the vehicle tires.

Description

Technical Field

[0001] The present application relates to the field of vehicle weighing, and in particular to a multimodal vehicle weighing method and device based on vehicle-road cloud. Background Art

[0002] With the rapid development of intelligent transportation systems, weighing systems combined with vehicle-road-cloud technology have gradually become a key tool for achieving efficient traffic management. However, the existing weighing systems in toll stations usually rely on a single type of sensor to detect vehicle weight. Since factors such as the number of axles, running speed, and acceleration of the vehicle will affect the measurement accuracy of the sensor, this method has certain limitations in weighing accuracy. Therefore, how to improve the measurement accuracy of the weighing system in toll stations has become an urgent problem to be solved.

[0003] Currently, many deep learning-based methods are used to improve the accuracy and reliability of weighing systems. These methods mainly estimate weight by identifying the deformation characteristics of vehicle tires. Although this method is effective to a certain extent, it still faces some challenges and limitations. For example, the identification of tire deformation may be affected by many factors, such as tire wear, tire pressure changes, and differences between different models, all of which may affect the final measurement results. In addition, when a vehicle passes at high speed, the characteristics of tire deformation may become blurred. Therefore, if the vehicle is weighed only by identifying the deformation characteristics of the vehicle tire, there is still a problem of low vehicle weighing accuracy in the weighing system.

[0004] Therefore, there is an urgent need for a multimodal vehicle weighing method and device based on vehicle-road-cloud. Summary of the invention

[0005] The present application provides a multimodal vehicle weighing method and device based on vehicle-road cloud, which solves the problem of low vehicle weighing accuracy in the weighing system when vehicle weighing is performed only by identifying the deformation characteristics of vehicle tires.

[0006] In a first aspect of the present application, a multimodal vehicle weighing method based on vehicle-road cloud is provided, the method comprising: in response to a dynamic weighing operation on a vehicle to be tested, obtaining first vehicle information, second vehicle information and third vehicle information of the vehicle to be tested respectively through vehicle identification technology, radar speed measurement technology and optical fiber sensing technology; obtaining a vehicle feature map corresponding to the vehicle to be tested through vehicle identification technology, the vehicle feature map including multiple features of the vehicle to be tested, the multiple features including appearance features, structural features and performance features; constructing a vehicle map neural network model according to the first vehicle information and the vehicle feature map, constructing a speed map neural network model according to the first vehicle information, the second vehicle information and the vehicle feature map, and constructing a quality map neural network model according to the first vehicle information, the third vehicle information and the vehicle feature map; constructing a multimodal fusion network model through the vehicle map neural network model, the speed map neural network model and the quality map neural network model, and calculating the actual weight of the vehicle to be tested according to the multimodal fusion network model.

[0007] Optionally, obtaining first vehicle information, second vehicle information and third vehicle information of the vehicle to be tested specifically includes: obtaining license plate information, vehicle model information, axle number information and first entry speed information corresponding to the vehicle to be tested as first vehicle information, the first entry speed information being obtained through vehicle identification technology; obtaining second entry speed information and acceleration information corresponding to the vehicle to be tested as second vehicle information, the second entry speed information being obtained through optical fiber sensing technology; obtaining mass distribution information and total mass information corresponding to the vehicle to be tested as third vehicle information.

[0008] Optionally, a vehicle feature map corresponding to the vehicle to be tested is obtained through vehicle recognition technology, specifically including: obtaining multiple vehicle images corresponding to the vehicle to be tested through vehicle recognition technology; obtaining the number of valid features in a target image, the target image being any image among the multiple vehicle images; determining whether the number of valid features is greater than or equal to a preset number of valid features; if the number of valid features is greater than or equal to the preset number of valid features, using the target image as the vehicle feature map corresponding to the vehicle to be tested.

[0009] Optionally, a multimodal fusion network model is constructed through a vehicle graph neural network model, a speed graph neural network model and a quality graph neural network model, and the actual weight of the vehicle to be tested is calculated according to the multimodal fusion network model, specifically including: obtaining a first vehicle graph output feature corresponding to the vehicle to be tested according to the vehicle graph neural network model, obtaining a first speed output feature corresponding to the vehicle to be tested according to the speed graph neural network model, and obtaining a first mass output feature corresponding to the vehicle to be tested according to the quality graph neural network model; performing corresponding preprocessing operations on the first vehicle graph output feature, the first speed output feature and the first mass output feature, respectively, to obtain a second vehicle graph output feature, a second speed output feature and a second mass output feature, the preprocessing operation including a maximum pooling operation, a fixed core multi-scale parsing operation and a multi-core parallel parsing operation; performing feature fusion on the second vehicle graph output feature, the second speed output feature and the second mass output feature through the multimodal fusion network model, and obtaining the comprehensive weighing feature corresponding to the vehicle to be tested according to the following formula:

[0010] ;

[0011] in, For comprehensive weighing characteristics, Output features for the second vehicle graph, is the speed output characteristic, is the quality output feature, Indicates the concatenation and fusion operation of two features. represents a convolution with a convolution kernel of 1, Represents a convolution with a convolution kernel of 3; based on the comprehensive weighing features, the actual weight of the vehicle to be tested is calculated.

[0012] Optionally, obtaining the second vehicle image output feature specifically includes: obtaining the second vehicle image output feature corresponding to the vehicle to be tested by using the following formula:

[0013] ;

[0014] in, Output features for the first vehicle graph, Output features for the second vehicle graph, is the feature partition matrix, is the convolution kernel Row, No. The weight of the column, is the size of the convolution kernel.

[0015] Optionally, obtaining the second speed output characteristic specifically includes: obtaining the second speed output characteristic by the following formula:

[0016] ;

[0017] in, is the first speed output feature, is the second speed output characteristic, , , , are all preset weight coefficients. Indicates that the maximum pooling operation is performed on the first speed output feature. is the size of the pooling kernel set, According to the pooling kernel The size of the first speed output feature is partitioned, is a protection factor used to avoid the denominator being zero.

[0018] Optionally, obtaining the second quality output feature specifically includes: obtaining the second quality output feature by using the following formula:

[0019] ;

[0020] in, is the first quality output feature, is the second quality output feature, , is the preset weight coefficient, is the size of the pooling kernel set, Indicates that the maximum pooling operation is performed on the first quality output feature. According to the pooling kernel The first quality output feature is partitioned by the size of is a protection factor used to prevent the denominator from being zero.

[0021] In a second aspect of the present application, a multimodal vehicle weighing device based on a vehicle-road cloud is provided, the device comprising a vehicle feature acquisition module and a multimodal fusion network model construction module, wherein:

[0022] The vehicle feature acquisition module is used to respond to the dynamic weighing operation on the vehicle to be tested, and respectively obtain the first vehicle information, the second vehicle information and the third vehicle information of the vehicle to be tested through vehicle identification technology, radar speed measurement technology and optical fiber sensing technology; obtain the vehicle feature map corresponding to the vehicle to be tested through vehicle identification technology, and the vehicle feature map includes multiple features of the vehicle to be tested, and the multiple features include appearance features, structural features and performance features.

[0023] A multimodal fusion network model construction module is used to construct a vehicle graph neural network model based on the first vehicle information and the vehicle characteristic map, to construct a speed graph neural network model based on the first vehicle information, the second vehicle information and the vehicle characteristic map, and to construct a quality graph neural network model based on the first vehicle information, the third vehicle information and the vehicle characteristic map; a multimodal fusion network model is constructed through the vehicle graph neural network model, the speed graph neural network model and the quality graph neural network model, and the actual weight of the vehicle to be tested is calculated based on the multimodal fusion network model.

[0024] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes any one of the methods described above.

[0025] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to perform any of the above methods.

[0026] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0027] 1. When the dynamic weighing operation is performed on the vehicle to be tested, the first vehicle information, the second vehicle information and the third vehicle information of the vehicle to be tested are respectively obtained through vehicle identification technology, radar speed measurement technology and optical fiber sensing technology, and the vehicle characteristic map corresponding to the vehicle to be tested is obtained through vehicle identification technology, so as to construct a vehicle map neural network model according to the first vehicle information and the vehicle characteristic map, construct a speed map neural network model according to the first vehicle information, the second vehicle information and the vehicle characteristic map, and construct a quality map neural network model according to the first vehicle information, the third vehicle information and the vehicle characteristic map, so as to construct a multimodal fusion network model through the vehicle map neural network model, the speed map neural network model and the quality map neural network model, and calculate the actual weight of the vehicle to be tested according to the multimodal fusion network model, so as to construct a multimodal fusion network by integrating vehicle identification, radar speed measurement and optical fiber sensing technology, so as to comprehensively obtain the characteristics, speed and quality information of the vehicle from multiple angles, improve the accuracy and stability of dynamic weighing, and ensure the accuracy and reliability of vehicle weight estimation.

[0028] 2. By using the target image whose number of effective features is greater than or equal to the preset number of effective features as the vehicle feature map corresponding to the vehicle to be tested, it can be ensured that the selected vehicle feature map has sufficient feature information to accurately reflect the appearance and structure of the vehicle, thereby improving the input data quality of the subsequent neural network model, and enhancing the training effects of the vehicle map neural network model, speed map neural network model and mass map neural network model, so that the multimodal fusion network model can more accurately capture the relevant characteristics of the vehicle in the dynamic weighing operation, thereby improving the accuracy and reliability of the actual weight estimation of the vehicle.

[0029] 3. A multimodal fusion network model is constructed through the vehicle graph neural network model, the speed graph neural network model and the mass graph neural network model, and the actual weight of the vehicle to be tested is calculated according to the multimodal fusion network model. Then, the comprehensive weighing features are calculated and converted according to the multimodal fusion network model to obtain the final vehicle body weight after removing the vehicle kinematic behavior factors, that is, the actual weight of the vehicle to be tested. This actual weight is fed back to the toll station checkpoint, which can realize accurate billing of passing vehicles and improve the fairness and accuracy of charging. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of a multi-modal vehicle weighing method based on vehicle-road-cloud provided in an embodiment of the present application;

[0031] Figure 2 is a high-level input vector acquisition flow chart provided in an embodiment of the present application;

[0032] Figure 3 It is a module schematic diagram of a multi-modal vehicle weighing device based on vehicle-road-cloud provided in an embodiment of the present application;

[0033] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0034] Explanation of the reference numerals: 31, vehicle feature acquisition module; 32, multimodal fusion network model construction module; 401, processor; 402, communication bus; 403, user interface; 404, network interface; 405, memory. DETAILED DESCRIPTION

[0035] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0036] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0037] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0038] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0039] With the rapid development of intelligent transportation systems, weighing systems combined with vehicle-road-cloud technology have gradually become a key tool for achieving efficient traffic management. However, the existing weighing systems in toll stations usually rely on a single type of sensor to detect vehicle weight. Since factors such as the number of axles, running speed, and acceleration of the vehicle will affect the measurement accuracy of the sensor, this method has certain limitations in weighing accuracy. Therefore, how to improve the measurement accuracy of the weighing system in toll stations has become an urgent problem to be solved.

[0040] Currently, many deep learning-based methods are used to improve the accuracy and reliability of weighing systems. These methods mainly estimate weight by identifying the deformation characteristics of vehicle tires. Although this method is effective to a certain extent, it still faces some challenges and limitations. For example, the identification of tire deformation may be affected by many factors, such as tire wear, tire pressure changes, and differences between different models, all of which may affect the final measurement results. In addition, when a vehicle passes at high speed, the characteristics of tire deformation may become blurred. Therefore, if the vehicle is weighed only by identifying the deformation characteristics of the vehicle tire, there is still a problem of low vehicle weighing accuracy in the weighing system.

[0041] Therefore, there is an urgent need for a multimodal vehicle weighing method and device based on vehicle-road-cloud.

[0042] Please refer to Figure 1, which shows a flow chart of a multimodal vehicle weighing method based on vehicle-road-cloud provided in an embodiment of the present application, and the flow chart mainly includes the following steps: S101 to S104.

[0043] Step S101, in response to a dynamic weighing operation on a vehicle to be tested, first vehicle information, second vehicle information and third vehicle information of the vehicle to be tested are respectively obtained through vehicle identification technology, radar speed measurement technology and optical fiber sensing technology.

[0044] Specifically, when weighing and charging vehicles at toll stations, the passing vehicles are monitored in real time, and the current position of the vehicle to be tested is confirmed through a high-precision positioning system, and it is determined whether the vehicle to be tested is close to the toll station checkpoint. The vehicle to be tested is any vehicle that is being weighed and charged. When the vehicle to be tested approaches the toll station checkpoint, in response to the dynamic weighing operation on the vehicle to be tested, the vehicle is guided through different detection areas in sequence. The detection areas include a first detection area, a second detection area, and a third detection area. The first detection area is equipped with a vehicle identification device unit. The vehicle identification device unit obtains the first entry information through the vehicle identification technology. The first entry speed information includes the license plate information, vehicle model information, axle number information and the first entry speed information corresponding to the vehicle to be tested. The vehicle identification technology is OCR (Optical Detection and Recognition). Character Recognition, optical character recognition) recognition technology; a radar speed measuring device unit is installed in the second detection area. The radar speed measuring device unit obtains the second vehicle information of the vehicle to be tested through the radar speed measuring technology. The second vehicle information includes the second entry speed information and acceleration information. The operation mode of the radar speed measuring device unit is as follows: a continuous microwave signal is sent to the vehicle to be tested through the high-precision microwave transmitter built into the radar speed measuring device unit. The microwave signal will undergo a Doppler frequency shift after encountering the moving vehicle to be tested, and then it will be captured by the high-sensitivity receiver on the radar speed measuring device unit. After that, the radar speed measuring device extracts the Doppler frequency shift signal through time-frequency transformation such as fast Fourier transform information, and calculates the second approaching speed information and acceleration information of the vehicle to be tested relative to the traveling direction of the radar speed measuring device unit; the optical fiber sensing device unit is installed in the third detection area, and the optical fiber sensing device unit obtains the third vehicle information of the vehicle to be tested through the optical fiber sensing technology. The third vehicle information includes the mass distribution information and the total mass information corresponding to the vehicle to be tested. The operation mode of the optical fiber sensing device unit is as follows: when the vehicle to be tested passes through the optical fiber sensing device unit, the optical fiber Bragg grating sensor in the optical fiber sensing device unit is used to capture the tiny optical fiber deformation caused by the gravity of the vehicle to be tested, and converts the optical fiber deformation passing through the optical signal into an electrical signal, thereby calculating the mass distribution information and the total mass information of the vehicle to be tested.

[0045] Step S102, obtaining a vehicle characteristic map corresponding to the vehicle to be tested by vehicle recognition technology, wherein the vehicle characteristic map includes multiple features of the vehicle to be tested, and the multiple features include appearance features, structural features, and performance features.

[0046] Specifically, through the vehicle recognition technology in step S101, multiple vehicle images corresponding to the vehicle to be tested are obtained; the number of valid features in the target image is obtained, and the target image is any image in the multiple vehicle images; it is determined whether the number of valid features is greater than or equal to the preset number of valid features; if the number of valid features is greater than or equal to the preset number of valid features, the target image is used as the vehicle feature map corresponding to the vehicle to be tested; if the number of valid features is less than the preset number of valid features, the target image is reselected, and the number of valid features in the reselected target image is obtained and the above judgment process is repeated until all vehicle images are traversed. Through the above steps, at least one vehicle feature map corresponding to the vehicle to be tested will be obtained. If there is no target image that meets the conditions among the multiple vehicle images corresponding to the vehicle to be tested, the multiple vehicle images corresponding to the vehicle to be tested are obtained again through the vehicle recognition technology and the above traversal process is repeated. Among them, the valid features in the vehicle feature map, that is, the multiple features corresponding to the vehicle to be tested, include but are not limited to: appearance features, structural features, performance features, etc., wherein the appearance features include but are not limited to: front shape features, vehicle logo features, annual inspection mark features, etc., and the structural features and performance features include but are not limited to: vehicle length features, vehicle width features, vehicle height features, wheelbase features, track features, etc.

[0047] Step S103, constructing a vehicle graph neural network model based on the first vehicle information and the vehicle characteristic map, constructing a speed graph neural network model based on the first vehicle information, the second vehicle information and the vehicle characteristic map, and constructing a quality graph neural network model based on the first vehicle information, the third vehicle information and the vehicle characteristic map.

[0048] Specifically, the first vehicle information, the second vehicle information, the third vehicle information and the multiple features in the vehicle feature map are vectorized to obtain multiple feature vectors, and a vehicle map neural network model, a speed map neural network model and a quality map neural network model are constructed according to the multiple feature vectors. The specific steps of vectorization are implemented by the following formula:

[0049] ;

[0050] in, is the feature vector obtained by vectorizing the first vehicle information and the vehicle feature map, is the axle number information, is a feature vector obtained by vectorizing the first vehicle information, the second vehicle information and the vehicle feature map, is the first entry speed information, is the second entry speed information, is the acceleration information, is a feature vector obtained by vectorizing the first vehicle information, the third vehicle information and the vehicle feature map, is the quantity distribution information, Total quality information.

[0051] Step S104, constructing a multimodal fusion network model through the vehicle graph neural network model, the speed graph neural network model and the mass graph neural network model, and calculating the actual weight of the vehicle to be tested according to the multimodal fusion network model.

[0052] Specifically, , , As the input vector, it is processed by the linear transformation network, nonlinear activation layer, graph attention network layer and fully connected layer respectively to obtain , , The corresponding higher-level input vectors , , , , , Compared to , , It has richer information expression capabilities and is more suitable for feature representation of specific tasks (such as speed prediction, mass distribution analysis, etc.). Please refer to Figure 2 , which shows a high-level input vector acquisition flow chart provided in the embodiment of this application. Since the actual weight calculated in this application needs to meet the adaptability of multi-source data and the real-time transmission of system data, it is also necessary to separately , , Perform multi-dimensional information analysis to improve the computing speed while meeting the adaptability of multi-source data. The specific steps are as follows:

[0053] S1. For For example, the first vehicle graph output feature corresponding to the vehicle to be tested is obtained according to the vehicle graph neural network model, that is, As the input feature, it goes through three layers of convolution in sequence, and obtains the output feature of the first vehicle map to better adapt to subsequent processing tasks. The specific operation rules are as follows:

[0054] ;

[0055] in, Output features of the first vehicle image. Then, the corresponding preprocessing operation is performed on the output features of the first vehicle image to obtain the output features of the second vehicle image. The specific operation rules are as follows:

[0056] ;

[0057] in, Output features for the first vehicle graph, Output features for the second vehicle graph, is the feature partition matrix, is the convolution kernel Row, No. The weight of the column, is the size of the convolution kernel.

[0058] S2. For For example, the first vehicle graph output feature corresponding to the vehicle to be tested is obtained according to the vehicle graph neural network model, that is, As the input feature, it goes through three layers of convolution in sequence, and obtains the output feature of the first vehicle map to better adapt to subsequent processing tasks. The specific operation rules are as follows:

[0059] ;

[0060] in, is the first speed output feature. Then, the first speed output feature is subjected to corresponding preprocessing operations to obtain the second speed output feature. The specific operation rules are as follows:

[0061] ;

[0062] in, is the second speed output characteristic, is the second speed output characteristic, , , , are all preset weight coefficients. Indicates that the maximum pooling operation is performed on the first speed output feature. is the size of the pooling kernel set, According to the pooling kernel The size of the first speed output feature is partitioned, is a protection factor used to avoid the denominator being zero.

[0063] S3. For For example, the first vehicle graph output feature corresponding to the vehicle to be tested is obtained according to the vehicle graph neural network model, that is, As the input feature, it goes through three layers of convolution in sequence, and obtains the output feature of the first vehicle map to better adapt to subsequent processing tasks. The specific operation rules are as follows:

[0064] ;

[0065] in, is the first quality output feature. Then, the first quality output feature is subjected to corresponding preprocessing operations to obtain the second quality output feature. The specific operation rules are as follows:

[0066] ;

[0067] in, is the first quality output feature, is the second quality output feature, , is the preset weight coefficient, is the size of the pooling kernel set, Indicates that the maximum pooling operation is performed on the first quality output feature. According to the pooling kernel The first quality output feature is partitioned by the size of is a protection factor used to prevent the denominator from being zero.

[0068] Through the above preprocessing operations, the preprocessing operations include maximum pooling operations, fixed-core multi-scale parsing operations, and multi-core parallel parsing operations, and the second vehicle image output features, the second speed output features, and the second mass output features are fused through the multimodal fusion network model, and the comprehensive weighing features corresponding to the vehicle to be tested are obtained according to the following formula:

[0069] ;

[0070] in, For comprehensive weighing characteristics, Output features for the second vehicle graph, is the speed output characteristic, is the quality output feature, Indicates the concatenation and fusion operation of two features. represents a convolution with a convolution kernel of 1, It represents a convolution with a convolution kernel of 3. After that, the comprehensive weighing features are input into the multimodal fusion network model. Through a series of convolutional layers, including but not limited to nonlinear layers, multi-layer perceptron MLP layers, normalization layers and other components, the comprehensive weighing features are calculated and transformed to obtain the final vehicle body weight after removing the vehicle kinematic behavior factors, that is, the actual weight of the vehicle to be tested. This actual weight is fed back to the toll station checkpoint, which can realize accurate billing of passing vehicles and improve the fairness and accuracy of charging.

[0071] The present application adopts the above method, and when the dynamic weighing operation is performed on the vehicle to be tested, the first vehicle information, the second vehicle information and the third vehicle information of the vehicle to be tested are respectively obtained through vehicle identification technology, radar speed measurement technology and optical fiber sensing technology, and the vehicle characteristic map corresponding to the vehicle to be tested is obtained through vehicle identification technology, so as to construct a vehicle map neural network model according to the first vehicle information and the vehicle characteristic map, construct a speed map neural network model according to the first vehicle information, the second vehicle information and the vehicle characteristic map, and construct a quality map neural network model according to the first vehicle information, the third vehicle information and the vehicle characteristic map, so as to construct a multimodal fusion network model through the vehicle map neural network model, the speed map neural network model and the quality map neural network model, and calculate the actual weight of the vehicle to be tested according to the multimodal fusion network model, so as to construct a multimodal fusion network by integrating vehicle identification, radar speed measurement and optical fiber sensing technology, so as to comprehensively obtain the characteristics, speed and quality information of the vehicle from multiple angles, improve the accuracy and stability of dynamic weighing, and ensure the accuracy and reliability of vehicle weight estimation.

[0072] Please refer to Figure 3 , which shows a module schematic diagram of a multimodal vehicle weighing device based on vehicle-road cloud provided in an embodiment of the present application, the device includes a vehicle feature acquisition module 31 and a multimodal fusion network model construction module 31, wherein:

[0073] The vehicle feature acquisition module 31 is used to respond to the dynamic weighing operation on the vehicle to be tested, and respectively obtain the first vehicle information, the second vehicle information and the third vehicle information of the vehicle to be tested through vehicle identification technology, radar speed measurement technology and optical fiber sensing technology; obtain the vehicle feature map corresponding to the vehicle to be tested through vehicle identification technology, and the vehicle feature map includes multiple features of the vehicle to be tested, and the multiple features include appearance features, structural features and performance features.

[0074] The multimodal fusion network model construction module 31 is used to construct a vehicle graph neural network model based on the first vehicle information and the vehicle characteristic map, to construct a speed graph neural network model based on the first vehicle information, the second vehicle information and the vehicle characteristic map, and to construct a quality graph neural network model based on the first vehicle information, the third vehicle information and the vehicle characteristic map; to construct a multimodal fusion network model through the vehicle graph neural network model, the speed graph neural network model and the quality graph neural network model, and to calculate the actual weight of the vehicle to be tested based on the multimodal fusion network model.

[0075] In a possible implementation, the vehicle feature acquisition module 31 is used to obtain the first vehicle information, the second vehicle information and the third vehicle information of the vehicle to be tested, specifically including: obtaining the license plate information, vehicle model information, axle number information and the first entry speed information corresponding to the vehicle to be tested as the first vehicle information, and the first entry speed information is obtained through vehicle identification technology; obtaining the second entry speed information and acceleration information corresponding to the vehicle to be tested as the second vehicle information, and the second entry speed information is obtained through optical fiber sensing technology; obtaining the mass distribution information and total mass information corresponding to the vehicle to be tested as the third vehicle information.

[0076] In one possible implementation, the vehicle feature acquisition module 31 is used to obtain a vehicle feature map corresponding to the vehicle to be tested through vehicle recognition technology, specifically including: obtaining multiple vehicle images corresponding to the vehicle to be tested through vehicle recognition technology; obtaining the number of valid features in a target image, where the target image is any image among the multiple vehicle images; determining whether the number of valid features is greater than or equal to a preset number of valid features; if the number of valid features is greater than or equal to the preset number of valid features, using the target image as the vehicle feature map corresponding to the vehicle to be tested.

[0077] In a possible implementation, the multimodal fusion network model construction module 31 is used to construct a multimodal fusion network model through a vehicle graph neural network model, a speed graph neural network model and a quality graph neural network model, and calculate the actual weight of the vehicle to be tested according to the multimodal fusion network model, specifically including: obtaining a first vehicle graph output feature corresponding to the vehicle to be tested according to the vehicle graph neural network model, obtaining a first speed output feature corresponding to the vehicle to be tested according to the speed graph neural network model, and obtaining a first mass output feature corresponding to the vehicle to be tested according to the quality graph neural network model; performing corresponding preprocessing operations on the first vehicle graph output feature, the first speed output feature and the first mass output feature, respectively, to obtain a second vehicle graph output feature, a second speed output feature and a second mass output feature, the preprocessing operation including a maximum pooling operation, a fixed core multi-scale parsing operation and a multi-core parallel parsing operation; performing feature fusion on the second vehicle graph output feature, the second speed output feature and the second mass output feature through the multimodal fusion network model, and obtaining the comprehensive weighing feature corresponding to the vehicle to be tested according to the following formula:

[0078] ;

[0079] in, For comprehensive weighing characteristics, Output features for the second vehicle graph, is the speed output characteristic, is the quality output feature, Indicates the concatenation and fusion operation of two features. represents a convolution with a convolution kernel of 1, Represents a convolution with a convolution kernel of 3; based on the comprehensive weighing features, the actual weight of the vehicle to be tested is calculated.

[0080] In a possible implementation, the multimodal fusion network model building module 31 is used to obtain the second vehicle graph output feature, specifically including: obtaining the second vehicle graph output feature corresponding to the vehicle to be tested by the following formula:

[0081] ;

[0082] in, Output features for the first vehicle graph, Output features for the second vehicle graph, is the feature partition matrix, is the convolution kernel Row, No. The weight of the column, is the size of the convolution kernel.

[0083] In a possible implementation manner, the multimodal fusion network model building module 31 is used to obtain the second speed output feature, specifically including: obtaining the second speed output feature through the following formula:

[0084] ;

[0085] in, is the first speed output feature, is the second speed output characteristic, , , , are all preset weight coefficients. Indicates that the maximum pooling operation is performed on the first speed output feature. is the size of the pooling kernel set, According to the pooling kernel The size of the first speed output feature is partitioned, is a protection factor used to avoid the denominator being zero.

[0086] In a possible implementation manner, the multimodal fusion network model construction module 31 is used to obtain the second quality output feature, specifically including: obtaining the second quality output feature through the following formula:

[0087] ;

[0088] in, is the first quality output feature, is the second quality output feature, , is the preset weight coefficient, is the size of the pooling kernel set, Indicates that the maximum pooling operation is performed on the first quality output feature. According to the pooling kernel The first quality output feature is partitioned by the size of is a protection factor used to prevent the denominator from being zero.

[0089] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0090] The present application also provides an electronic device. Figure 4 , Figure 4 4 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include: at least one processor 401, at least one communication bus 402, a user interface 403, at least one network interface 404, and a memory 405.

[0091] The communication bus 402 is used to realize the connection and communication between these components.

[0092] The user interface 403 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 403 may also include a standard wired interface and a wireless interface.

[0093] The network interface 404 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0094] Among them, the processor 401 may include one or more processing cores. The processor 401 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 405, and calling data stored in the memory 405. Optionally, the processor 401 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 401 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 401, and it can be implemented separately through a chip.

[0095] Among them, the memory 405 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 405 includes a non-transitory computer-readable storage medium. The memory 405 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 405 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 405 may optionally also be at least one storage device located away from the aforementioned processor 401. Refer to Figure 4 , the memory 405 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a vehicle weighing application based on a multimodal fusion network.

[0096] exist Figure 4In the electronic device shown, the user interface 403 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 401 can be used to call the vehicle weighing application based on the multimodal fusion network stored in the memory 405. When executed by one or more processors 401, the electronic device executes one or more of the methods described in the above embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0097] The present application also provides a computer-readable storage medium, which stores instructions. When executed by one or more processors, the electronic device executes one or more of the methods described in the above embodiments.

[0098] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0100] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0101] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0102] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0103] The above are only exemplary embodiments disclosed in this application, and the scope of the disclosure of this application cannot be limited by this. That is, any equivalent changes and modifications made according to the teachings disclosed in this application are still within the scope of the disclosure of this application. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other implementation schemes disclosed in this application.

[0104] This application is intended to cover any modifications, uses or adaptive changes disclosed in this application, which follow the general principles disclosed in this application and include common knowledge or customary technical means in the technical field not recorded in this application.

Claims

1. A multi-modal vehicle weighing method based on vehicle-road-cloud, characterized in that: The method comprises: In response to a dynamic weighing operation on a vehicle to be tested, first vehicle information, second vehicle information and third vehicle information of the vehicle to be tested are respectively obtained through vehicle identification technology, radar speed measurement technology and optical fiber sensing technology, specifically including: obtaining license plate information, vehicle model information, axle number information and first entry speed information corresponding to the vehicle to be tested as the first vehicle information, the first entry speed information being obtained through the vehicle identification technology; obtaining second entry speed information and acceleration information corresponding to the vehicle to be tested as the second vehicle information, the second entry speed information being obtained through the optical fiber sensing technology; obtaining mass distribution information and total mass information corresponding to the vehicle to be tested as the third vehicle information; Acquire a vehicle characteristic map corresponding to the vehicle to be tested by the vehicle identification technology, wherein the vehicle characteristic map includes multiple features of the vehicle to be tested, and the multiple features include appearance features, structural features, and performance features; A vehicle graph neural network model is constructed according to the first vehicle information and the vehicle feature map, a speed graph neural network model is constructed according to the first vehicle information, the second vehicle information and the vehicle feature map, and a quality graph neural network model is constructed according to the first vehicle information, the third vehicle information and the vehicle feature map; a plurality of features in the first vehicle information, the second vehicle information, the third vehicle information and the vehicle feature map are respectively vectorized to obtain a plurality of feature vectors, and the vehicle graph neural network model, the speed graph neural network model and the quality graph neural network model are constructed according to the plurality of feature vectors; the specific steps of vectorization are implemented by the following formula: ； in, is a feature vector obtained by vectorizing the first vehicle information and the vehicle feature map, is the axle number information, is a feature vector obtained by vectorizing the first vehicle information, the second vehicle information and the vehicle feature map, is the first entry speed information, is the second entry speed information, is the acceleration information, is a feature vector obtained by vectorizing the first vehicle information, the third vehicle information and the vehicle feature map, is the mass distribution information, is the total mass information; A multimodal fusion network model is constructed by using the vehicle graph neural network model, the speed graph neural network model and the quality graph neural network model, and the actual weight of the vehicle to be tested is calculated according to the multimodal fusion network model, specifically including: obtaining a first vehicle graph output feature corresponding to the vehicle to be tested according to the vehicle graph neural network model, obtaining a first speed output feature corresponding to the vehicle to be tested according to the speed graph neural network model, and obtaining a first quality output feature corresponding to the vehicle to be tested according to the quality graph neural network model; performing corresponding preprocessing operations on the first vehicle graph output feature, the first speed output feature and the first quality output feature, respectively, to obtain a second vehicle graph output feature, a second speed output feature and a second quality output feature, the preprocessing operation including a maximum pooling operation, a fixed core multi-scale parsing operation and a multi-core parallel parsing operation; wherein, the first speed output feature is subjected to the corresponding multi-core parallel parsing operation to obtain the second speed output feature; the first quality output feature is subjected to the corresponding multi-core parallel parsing operation to obtain the second quality output feature; the specific operation rules are as follows: ； in, is the first speed output feature, is the second speed output characteristic, , , , are all preset weight coefficients. Indicates that the maximum pooling operation is performed on the first speed output feature. is the size of the pooling kernel set, According to the pooling kernel The size of the first speed output feature is partitioned, is the first quality output feature, Output feature for the second quality, , is the preset weight coefficient, is the size of the pooling kernel set, represents performing the maximum pooling operation on the first quality output feature, According to the pooling kernel The first quality output feature is partitioned by the size of is a protection factor used to avoid the denominator being 0; The second vehicle image output feature, the second speed output feature, and the second mass output feature are fused by the multimodal fusion network model, and the comprehensive weighing feature corresponding to the vehicle to be tested is obtained according to the following formula: ； in, is the comprehensive weighing feature, Output features for the second vehicle graph, is the speed output characteristic, Output characteristics for said quality, Indicates the concatenation and fusion operation of two features. represents a convolution with a convolution kernel of 1, represents a convolution with a convolution kernel of 3; and calculates the actual weight of the vehicle to be tested according to the comprehensive weighing feature.

2. The method according to claim 1, characterized in that The step of obtaining the vehicle characteristic map corresponding to the vehicle to be tested by using the vehicle recognition technology specifically includes: Acquire multiple vehicle images corresponding to the vehicle to be tested by using the vehicle recognition technology; Acquire the number of valid features in a target image, wherein the target image is any image among the plurality of vehicle images; Determining whether the number of valid features is greater than or equal to a preset number of valid features; If the number of valid features is greater than or equal to the preset number of valid features, the target image is used as the vehicle feature map corresponding to the vehicle to be tested.

3. The method according to claim 1, characterized in that Obtaining the output feature of the second vehicle image specifically includes: The second vehicle image output feature corresponding to the vehicle to be tested is obtained by the following formula: ； in, Output features for the first vehicle graph, Output features for the second vehicle graph, is the feature partition matrix, is the convolution kernel Row, No. The weight of the column, is the size of the convolution kernel.

4. A multi-modal vehicle weighing device based on vehicle-road cloud, characterized in that: The device includes a vehicle feature acquisition module and a multimodal fusion network model construction module, wherein: The vehicle feature acquisition module is used to respond to the dynamic weighing operation on the vehicle to be tested, and respectively acquire the first vehicle information, the second vehicle information and the third vehicle information of the vehicle to be tested through vehicle identification technology, radar speed measurement technology and optical fiber sensing technology, specifically including: acquiring the license plate information, vehicle model information, axle number information and first entry speed information corresponding to the vehicle to be tested as the first vehicle information, and the first entry speed information is acquired through the vehicle identification technology; acquiring the second entry speed information and acceleration information corresponding to the vehicle to be tested as the second vehicle information, and the second entry speed information is acquired through the optical fiber sensing technology; acquiring the mass distribution information and total mass information corresponding to the vehicle to be tested as the third vehicle information; acquiring the vehicle feature map corresponding to the vehicle to be tested through the vehicle identification technology, and the vehicle feature map includes multiple features of the vehicle to be tested, and the multiple features include appearance features, structural features and performance features; The multimodal fusion network model construction module is used to construct a vehicle graph neural network model according to the first vehicle information and the vehicle feature map, to construct a speed graph neural network model according to the first vehicle information, the second vehicle information and the vehicle feature map, and to construct a quality graph neural network model according to the first vehicle information, the third vehicle information and the vehicle feature map; vectorize the first vehicle information, the second vehicle information, the third vehicle information and the vehicle feature map respectively to obtain multiple feature vectors, and construct the vehicle graph neural network model, the speed graph neural network model and the quality graph neural network model according to the multiple feature vectors; the specific steps of vectorization are implemented by the following formula: ； in, is a feature vector obtained by vectorizing the first vehicle information and the vehicle feature map, is the axle number information, is a feature vector obtained by vectorizing the first vehicle information, the second vehicle information and the vehicle feature map, is the first entry speed information, is the second entry speed information, is the acceleration information, is a feature vector obtained by vectorizing the first vehicle information, the third vehicle information and the vehicle feature map, is the mass distribution information, The method comprises the following steps: the first vehicle graph output feature corresponding to the vehicle to be tested is obtained according to the vehicle graph neural network model, the first speed output feature corresponding to the vehicle to be tested is obtained according to the speed graph neural network model, and the first quality output feature corresponding to the vehicle to be tested is obtained according to the quality graph neural network model; the first vehicle graph output feature, the first speed output feature and the first quality output feature are respectively preprocessed to obtain the second vehicle graph output feature, the second speed output feature and the second quality output feature, wherein the preprocessing operation includes a maximum pooling operation, a fixed core multi-scale parsing operation and a multi-core parallel parsing operation; wherein the first speed output feature is subjected to the corresponding multi-core parallel parsing operation to obtain the second speed output feature; the first quality output feature is subjected to the corresponding multi-core parallel parsing operation to obtain the second quality output feature; the specific operation rules are as follows: ； in, is the first speed output feature, is the second speed output characteristic, , , , are all preset weight coefficients. Indicates that the maximum pooling operation is performed on the first speed output feature. is the size of the pooling kernel set, According to the pooling kernel The size of the first speed output feature is partitioned, is the first quality output feature, Output feature for the second quality, , is the preset weight coefficient, is the size of the pooling kernel set, represents performing the maximum pooling operation on the first quality output feature, According to the pooling kernel The first quality output feature is partitioned by the size of is a protection factor used to avoid the denominator being 0; the second vehicle image output feature, the second speed output feature, and the second mass output feature are feature-fused through the multimodal fusion network model, and the comprehensive weighing feature corresponding to the vehicle to be tested is obtained according to the following formula: ； in, is the comprehensive weighing feature, Output features for the second vehicle graph, is the speed output characteristic, Output characteristics for the quality, Indicates the concatenation and fusion operation of two features. represents a convolution with a convolution kernel of 1, represents a convolution with a convolution kernel of 3; and calculates the actual weight of the vehicle to be tested according to the comprehensive weighing feature.

5. An electronic device, characterized in that: It includes a processor, a communication bus, a user interface, a network interface and a memory, wherein the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 3 is performed.

Images