A multimodal non-contact edge computing method and system for truck load sensing

Abstract

This invention provides a multimodal, non-contact edge computing method and system for truck load sensing, comprising: when a truck enters a detection area, vibration and acoustic sensors respectively collect vibration and acoustic signals; a time synchronization and triggering module provides a unified time reference and sends a hardware trigger signal to a dual-light sensor module; the dual-light sensor module accordingly uses an industrial visible light camera and an infrared thermal imager to simultaneously collect visible light and infrared thermal images of the truck. All collected multimodal data are sent to an edge computing unit, which performs time alignment and preprocessing based on a unified timestamp to construct multimodal samples; subsequently, the edge computing unit inputs the samples into a lightweight load classification inversion model for feature extraction and fusion inference, outputting the truck load level and identification confidence score; finally, the load level, confidence score, and vehicle passage information are uploaded to a backend platform via a communication module.

Description

Technical Field

[0001] This invention relates to the field of vehicle inspection technology, and in particular to a multimodal non-contact edge computing method and system for truck load sensing. Background Technology

[0002] Overloading of trucks on highways is a major cause of damage to road structure and serious traffic accidents. Traditional highway overloading control mainly relies on contact dynamic weighing equipment such as weighbridges and piezoelectric quartz sensors. Such equipment has high civil construction costs, is easy to damage the road surface, and its service life is significantly shortened due to the long-term rolling of heavy vehicles. In recent years, non-contact overloading control technology has gradually emerged. However, the existing technology still has the following defects: (1) The existing non-contact detection scheme mainly relies on single visible light visual perception, which has poor robustness in complex field environments such as night, rain, snow, strong light, and obstruction; (2) Although some systems have tried to introduce sound or vibration sensors, they lack a time synchronization mechanism at the underlying hardware level. They usually use software timestamps, which leads to errors of more than milliseconds between acoustic, vibration, and visual data, affecting the accuracy of subsequent multimodal fusion inversion models; (3) Existing terminals are mostly used as simple data acquisition nodes, which are highly dependent on cloud computing power, have high requirements for communication bandwidth and insufficient real-time performance, making it difficult to meet the real-time alarm requirements of highway overloading control. Summary of the Invention

[0003] The purpose of this invention is to provide a non-contact edge computing device for truck load sensing that integrates acoustic, vibration, optical, and thermal multimodal technologies, in order to solve the problems of existing truck load detection relying on contact weighing equipment, high construction and maintenance costs, insufficient stability of single-modal recognition, difficulty in synchronous fusion of multi-source data, and limited on-site real-time processing capabilities.

[0004] To address the above problems, this invention provides a multimodal, non-contact truck load sensing edge calculation method, comprising: Step S1: In response to the truck entering the detection area, vibration signals are collected by vibration sensors and acoustic signals are collected by acoustic sensors. Step S2: A unified time reference is provided by the time synchronization and triggering module, and a hardware trigger signal is sent to the dual-light sensor module. After receiving the hardware trigger signal, the dual-light sensor module acquires a visible light image of the truck through an industrial visible light camera and an infrared thermal image of the truck through an infrared thermal imager. Step S3: The collected vibration signal, acoustic signal, visible light image and infrared thermal image are sent to the edge computing unit; The edge computing unit performs time alignment and preprocessing on the received vibration signal, acoustic signal, visible light image and infrared thermal image according to the unified timestamp corresponding to the unified time base, and constructs multimodal samples; In step S4, the edge computing unit inputs multimodal samples into the lightweight load grading inversion model deployed on it for feature extraction and multimodal fusion inference, and outputs the load level and recognition confidence of the truck. Step S5: Upload the output load level, identification confidence level and vehicle passage information to the backend platform through the communication module.

[0005] Furthermore, the specific process of constructing multimodal samples in step S3 includes: Hardware trigger time corresponding to hardware trigger signal Based on the sampling frequency of the acoustic vibration synchronous acquisition module... Calculate the time position corresponding to each sampling point or each segment of sampling data in acoustic and vibration signals; Based on the exposure time or frame acquisition time recorded by the industrial visible light camera and infrared thermal imager, determine the acoustic signal segment and vibration signal segment corresponding to each frame of visible light image and infrared thermal image. Within the preset time window Δ t Internally, extract the acoustic signal sequence. and vibration signal sequence And select the timestamp that falls into the time window Δ t Visible light image frames within and infrared thermal image frames The combination constitutes a multimodal sample X. .

[0006] Furthermore, the specific process of feature extraction and multimodal fusion inference in step S4 includes: Step S601, for the acoustic signal sequence and vibration signal sequence Perform short-time Fourier transforms and then perform modulo-squared and logarithmic scaling to obtain the acoustic time-frequency energy spectrum. F a Vibration time-frequency energy spectrum F v ;Will F a and F v Input the acoustic and vibration feature extraction branches of the lightweight load grading inversion model respectively, and output the acoustic feature vectors. V a and vibration eigenvectors V v ; Step S602, frame the visible light image. and infrared thermal image frames The visual thermal imaging feature extraction branch of the input lightweight load grading inversion model extracts the vehicle's visual structural features and thermal distribution features through a parameter-discretion-free lightweight convolutional neural network, and outputs a visual feature vector. V vis and infrared thermal feature vector V ir ; Step S603, convert the acoustic feature vector V a Vibration characteristic vector V v Visual feature vectors V vis and infrared thermal feature vector V ir By concatenating the features, a joint multimodal feature vector is obtained. V concat ;Will V concat The input adaptive channel attention fusion module generates channel weight vectors. oh and utilize oh right V concat Element-wise weighting is performed to obtain the multimodal fusion feature vector. V fuse ; Step S604: Fuse the multimodal feature vectors V fuse The load level classification module is input, and the predicted probability corresponding to each preset load level is calculated and output through a fully connected layer and a Softmax function. The category with the highest predicted probability is selected as the load level output for trucks, and the maximum predicted probability is used as the recognition confidence level.

[0007] Furthermore, the preset load levels include five levels: no load, light load, medium load, heavy load, and suspected overload.

[0008] Furthermore, the lightweight load grading inversion model introduces a focus loss function for network weight optimization during the training phase, and the total loss function... L total Represented as , in, C num This represents the total number of preset load level categories; y c This represents the one-hot encoding of the true label, when the true class of the sample is the first... c Class Time y c =1, otherwise y c =0; αc Indicates the first c Class balance parameter, P c For the first c Class prediction probability, c This represents the modulation coefficients for easy and difficult samples.

[0009] The present invention also provides a multimodal non-contact truck load sensing edge computing system, comprising: The sensing layer includes a vibration sensor, an acoustic sensor, and a dual-light sensor module. The vibration sensor is used to collect vibration signals when the truck passes through the detection area. The acoustic sensor is used to collect acoustic signals when the truck passes through the detection area. The dual-light sensor module includes an industrial visible light camera and an infrared thermal imager, which are used to collect visible light images and infrared thermal images when the truck passes through the detection area, respectively. Data Acquisition and Synchronization Layer: Includes an acoustic and vibration synchronization acquisition module and a time synchronization and triggering module; the acoustic and vibration synchronization acquisition module is connected to the vibration sensor and acoustic sensor, and is used to condition, convert analog to digital and sample the received analog signals synchronously, and output synchronous acoustic and vibration data; the time synchronization and triggering module is used to provide a unified time reference and send hardware trigger signals to the dual-optical sensor module; Edge computing layer: includes edge computing units, which are connected to the acoustic vibration synchronous acquisition module and the dual-light sensor module respectively. They are used to receive synchronous acoustic vibration data, visible light images and infrared thermal images, and perform time alignment, preprocessing, feature extraction, multimodal fusion and load level inference based on a unified timestamp, and output the load level and recognition confidence of the truck. Communication module: Connects to the edge computing unit to upload the load level, identification confidence level, and vehicle traffic information output by the edge computing unit; and backend platform: Connects to the communication module to receive, store, display, retrieve, issue warnings, and manually verify the uploaded data.

[0010] Furthermore, the acoustic and vibration synchronous acquisition module includes an FPGA control unit, a multi-channel analog-to-digital converter, an ICP microphone interface, an ICP accelerometer interface, a signal conditioning circuit, and a local buffer unit; the FPGA control unit is used to generate a unified sampling clock and control the synchronous sampling of multiple acoustic channels and multiple vibration channels.

[0011] Furthermore, the time synchronization and triggering module includes a precision time synchronization unit, a hardware triggering circuit, and a timestamp recording unit; the precision time synchronization unit adopts any one of the following to achieve microsecond-level clock synchronization: IEEE 1588 precision time protocol, GPS time synchronization, Beidou time synchronization, network time synchronization, or local high-precision clock source; the hardware triggering circuit is used to send trigger pulses to industrial visible light cameras and infrared thermal imagers to achieve synchronous exposure or frame acquisition.

[0012] Furthermore, a lightweight load hierarchical inversion model is deployed in the edge computing unit. The model includes an acoustic vibration feature extraction branch, a visual thermal imaging feature extraction branch, an adaptive channel attention fusion module, and a load level classification module. The visual thermal imaging feature extraction branch includes a visible light image branch and an infrared thermal image branch, both of which adopt a lightweight convolutional neural network structure with non-shared parameters.

[0013] Furthermore, it also includes a protective enclosure, which is used to house the acoustic and vibration synchronization acquisition module, the time synchronization and triggering module, the edge computing unit, the communication module, as well as power supply, heat dissipation, lightning protection and surge protection components; the protective enclosure has an IP or higher protection rating and is fixedly installed on the roadside poles, gantry, brackets or bridge components in the road detection area.

[0014] Compared with the prior art, the present invention has the following advantages: (1) The present invention adopts a non-contact detection method, which does not require the installation of sensors on the vehicle being tested, nor does it require the destruction of the road surface to bury the weighing sensors, thus reducing the construction difficulty and subsequent maintenance costs; (2) The present invention simultaneously collects acoustic signals, vibration signals, visible light images and infrared thermal images, which can reflect the load status of trucks from multiple dimensions such as vehicle dynamic response, vehicle structural characteristics and vehicle thermal distribution characteristics, reducing the recognition instability caused by environmental interference of a single mode; (3) The present invention, through synchronous acquisition of sound and vibration, unified time reference and hardware triggering mechanism, enables sound (3) The signal, vibration signal, visible light image and infrared thermal image can correspond to the same truck passage process, which is convenient for multimodal data fusion; (4) The present invention deploys the load classification inversion model on the edge computing unit, which can complete multimodal data processing and load level real-time reasoning on the road site, reducing the dependence on cloud computing resources and communication bandwidth; (5) The present invention uploads the load classification results and vehicle passage information to the background platform through the communication module, which is convenient for highway overload control, vehicle screening, early warning management and manual review; (6) The present invention is encapsulated in a protective chassis, which is suitable for long-term continuous operation in the road field environment. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the system of the present invention.

[0016] Figure 2 This is a flowchart of the system of the present invention.

[0017] Figure 3 This is a schematic diagram of multimodal data time alignment in this invention.

[0018] The reference numerals in the attached figures are explained as follows: 10. Sensing layer; 11. Vibration sensor; 12. Acoustic sensor; 13. Dual-light sensor module; 20. Data acquisition and synchronization layer; 21. Acoustic and vibration synchronization acquisition module; 22. Time synchronization and triggering module; 30. Edge computing layer; 31. Edge computing unit; 41. Communication module; 50. Backend platform; 60. Protective enclosure. Detailed Implementation

[0019] like Figure 1 As shown, this embodiment provides a non-contact edge computing device for sensing truck loads using integrated acoustic, vibration, optical, and thermal multimodal sensors. The device includes a sensing layer 10, a data acquisition and synchronization layer 20, an edge computing layer 30, a communication module 41, a backend platform 50, and a protective enclosure 60.

[0020] The sensing layer 10 includes a vibration sensor 11, an acoustic sensor 12, and a dual-light sensor module 13. The vibration sensor 11 collects vibration signals from the road surface, roadside infrastructure, bridge components, or mounting bases caused by the truck passing through the detection area. The acoustic sensor 12 collects engine noise, tire-road contact noise, vehicle structure radiated sound, and other acoustic signals related to the vehicle's operating status generated when the truck passes through the detection area. The dual-light sensor module 13 collects visible light and infrared thermal images of the truck passing through the detection area.

[0021] The data acquisition and synchronization layer 20 includes an acoustic and vibration synchronization acquisition module 21 and a time synchronization and triggering module 22. The acoustic and vibration synchronization acquisition module 21 is used to perform signal conditioning, analog-to-digital conversion, and synchronous sampling on the analog signals output by the vibration sensor 11 and the acoustic sensor 12. The time synchronization and triggering module 22 is used to provide a unified time reference for the acoustic and vibration synchronization acquisition module 21, the dual-optical sensor module 13, and the edge computing unit 31, and to send a hardware trigger signal to the dual-optical sensor module 13.

[0022] The edge computing layer 30 includes an edge computing unit 31. The edge computing unit 31 receives synchronous acoustic and vibration data from the acoustic and vibration synchronous acquisition module 21, as well as visible light images and infrared thermal images from the dual-light sensor module 13. The edge computing unit 31 performs time alignment, preprocessing, feature extraction, multimodal fusion, and load level inference on the received multimodal data, and outputs the truck load level.

[0023] The communication module 41 is connected to the edge computing unit 31 and is used to upload the truck load level, vehicle image, infrared thermal image, license plate information, passage time, lane number, device number, and recognition confidence level output by the edge computing unit 31 to the backend platform 50. The backend platform 50 is used to store, display, retrieve, issue warnings for, and manually verify the uploaded data.

[0024] The protective enclosure 60 houses the acoustic and vibration synchronization acquisition module 21, the time synchronization and triggering module 22, the edge computing unit 31, the communication module 41, and power supply, heat dissipation, lightning protection, and surge protection components. The protective enclosure 60 can be installed on the roadside, gantry, pole, or bracket in the road inspection area.

[0025] Vibration sensor 11 sends vibration signals to acoustic vibration synchronous acquisition module 21, and acoustic sensor 12 sends acoustic signals to acoustic vibration synchronous acquisition module 21; time synchronization and triggering module 22 provides a unified time reference to acoustic vibration synchronous acquisition module 21 and sends hardware trigger pulses to dual-optical sensor module 13; dual-optical sensor module 13 sends photothermal image data to edge computing unit 31; acoustic vibration synchronous acquisition module 21 sends synchronized acoustic vibration data to edge computing unit 31; edge computing unit 31 outputs load classification results to communication module 41, and communication module 41 uploads them to background platform 50.

[0026] The acoustic and vibration synchronous acquisition module 21 includes an FPGA control unit, a multi-channel analog-to-digital converter, an acoustic sensor interface, a vibration sensor interface, a signal conditioning circuit, and a local buffer unit. The FPGA control unit generates a unified sampling clock and controls the synchronous sampling of multiple acoustic and vibration channels. The signal conditioning circuit filters, amplifies, impedance-matches, or isolates the signals output by the acoustic sensor 12 and vibration sensor 11. The multi-channel analog-to-digital converter converts the signal-conditioned analog signal into a digital signal. The local buffer unit temporarily stores the sampled data in case of data transmission delays or communication anomalies.

[0027] The acoustic sensor interface is an ICP microphone interface, and the vibration sensor interface is an ICP accelerometer interface. The acoustic-vibration synchronous acquisition module 21 includes four ICP accelerometer interfaces and four ICP microphone interfaces, used to achieve synchronous acquisition of four vibration signals and four acoustic signals. The number of acoustic and vibration channels can be adjusted according to the number of lanes, the detection area, the sensor deployment method, and the required detection accuracy. For example, the acoustic-vibration synchronous acquisition module 21 can be configured with one, two, eight, or more acoustic and vibration sensor interfaces.

[0028] The dual-light sensor module 13 includes an industrial visible light camera and an infrared thermal imager. The industrial visible light camera is used to acquire visible light images of the truck as it passes through the detection area. The edge computing unit 31 can identify or extract information such as truck model, body outline, number of axles, axle group type, wheel position, license plate area, and vehicle traffic status based on the visible light images. The infrared thermal imager is used to acquire infrared thermal images of the truck as it passes through the detection area. The edge computing unit 31 can extract information such as tire temperature, wheel hub temperature, braking area temperature, heat distribution under the vehicle, and heat distribution in the cargo area based on the infrared thermal images.

[0029] The dual-light sensor module 13 can connect to the edge computing unit 31 via Ethernet, USB, MIPI, GMSL, Camera Link, GigE Vision, or other high-speed data interfaces, and send the acquired visible light images and infrared thermal images to the edge computing unit 31 for processing. The industrial visible light camera and infrared thermal imager receive the same hardware trigger signal and then synchronously expose or capture frames, thus making the visible light images and infrared thermal images correspond to the acoustic and vibration data in time.

[0030] The time synchronization and triggering module 22 includes a precision time synchronization unit, a hardware triggering circuit, and a timestamp recording unit. The precision time synchronization unit provides a unified time reference for the acoustic-vibration synchronization acquisition module 21, the dual-light sensor module 13, and the edge computing unit 31. Preferably, the precision time synchronization unit uses the IEEE 1588 precision time protocol to achieve microsecond-level clock synchronization within the device or between multiple devices. The precision time synchronization unit can also use GPS time synchronization, BeiDou time synchronization, network time synchronization, or a local high-precision clock source. The hardware triggering circuit sends trigger pulses to the industrial visible light camera and infrared thermal imager. The trigger pulses can be sent through a general-purpose input / output interface, an optocoupler isolation interface, a TTL interface, an RS485 interface, or other trigger interfaces. After receiving the trigger pulses, the industrial visible light camera and infrared thermal imager complete image acquisition, ensuring that the acquisition time of the visible light image and infrared thermal image corresponds to the acquisition time of the acoustic signal and vibration signal. The timestamp recording unit adds a unified timestamp to the acoustic signal, vibration signal, visible light image, and infrared thermal image. For continuously acquired acoustic and vibration data, the timestamp recording unit records the sampling start time, sampling frequency, and sampling sequence number; for image data, the timestamp recording unit records the exposure time, frame acquisition time, or image arrival time.

[0031] In this way, acoustic signals, vibration signals, visible light images, and infrared thermal images can be aligned under a unified time reference, avoiding multimodal data mismatch caused by acquisition delays or clock drift from different sensors.

[0032] like Figure 3As shown, the edge computing unit 31 performs time alignment on the acoustic signal, vibration signal, visible light image, and infrared thermal image based on a unified timestamp. Let the trigger time for the vehicle entering the detection area be... The sampling frequency of the acoustic vibration synchronous acquisition module 21 is The sampling sequence number of the acoustic vibration data is n Then the first n The time corresponding to each sampling point satisfy The acoustic and vibration synchronous acquisition module 21 records the sampling start time, sampling frequency, and sampling sequence number for continuous sampling data when acquiring acoustic and vibration signals. The edge computing unit 31 calculates the time position corresponding to each sampling point or each segment of sampling data based on the sampling start time and sampling frequency.

[0033] After receiving the hardware trigger signal, the dual-light sensor module 13 records the exposure time or frame acquisition time for the visible light image frame and the infrared thermal image frame. The edge computing unit 31 determines the acoustic signal segment and vibration signal segment corresponding to each frame of the visible light image and the infrared thermal image based on the image frame timestamp.

[0034] Let Δt be the time window corresponding to the passage of the same truck, and let the starting point of the time window Δt be... The destination is ,in, Indicates the length of time that is retained before the triggering time. Indicates the length of time that is retained after the triggering time.

[0035] Edge computing unit 31 in the time window Δ t Internal extraction of acoustic signal sequence and vibration signal sequence And select timestamps that fall within the time window Δ t Visible light image frames within and infrared thermal image frames Combine them into a set of multimodal samples X , 。 The time window Δ t Adjustments can be made based on vehicle speed, lane width, sensor installation location, vehicle target detection results, or vehicle tracking results.

[0036] Through the above method, the present invention can perform time matching of multi-source heterogeneous data belonging to the same truck passage process, avoid the incorrect fusion of data from different vehicles or different times, thereby improving the reliability of load level inference.

[0037] After receiving the multimodal data sent by the acoustic and vibration synchronous acquisition module 21 and the dual-light sensor module 13, the edge computing unit 31 aligns and slices the data according to a unified timestamp, combines the acoustic signal segment, vibration signal segment, visible light image frame and infrared thermal image frame corresponding to the same truck passing through the detection area into a set of multimodal samples, and inputs the multimodal samples into the lightweight load grading inversion model.

[0038] The lightweight load classification inversion model includes an acoustic and vibration feature extraction branch, a visual thermal imaging feature extraction branch, an adaptive channel attention fusion module, and a load level classification module. This model extracts load-related features from acoustic signals, vibration signals, visible light images, and infrared thermal images, respectively, and performs fusion inference on different modal features to output the truck load level and recognition confidence.

[0039] Specifically, the multimodal feature fusion and load inversion process of edge computing unit 31 includes the following sub-steps: Step S601: Construction of multimodal data slices and time-aligned samples. This is based on the hardware trigger time generated by the time synchronization and triggering module 22. Based on this, edge computing unit 31 extracts a time window Δ t Multimodal data within. Let the sampling frequency of the acoustic vibration synchronous acquisition module 21 be... Then in the time window Δ t The time series of acoustic signals acquired internally is defined as: ; The vibration signal time series is defined as: : in, Indicates the first n One acoustic sampling point, Indicates the first n One vibration sampling point, n ∈[1, N ], N Indicates the time window Δ t The total number of sampling points within, and satisfying .

[0040] Edge computing unit 31 acquires the time window Δ t Corresponding visible light image frame and infrared thermal image frames Visible light image matrix The size is Infrared thermal image matrix The size is ,in, H Indicates the image height.W This indicates the image width; 3 represents the RGB three channels of the visible light image; and 1 represents a single channel of temperature or grayscale information in the infrared thermal image. The number of channels for both the visible light and infrared thermal images can be adjusted according to the data format of the sensor used.

[0041] Edge computing unit 31 will , , and Combined into a set of multimodal samples X: , Where X represents a set of multimodal samples input to the lightweight load grading inversion model.

[0042] Step S602: Time-frequency domain feature mapping of acoustic and vibration signals. Edge computing unit 31 performs time-series mapping on the acoustic signal. and vibration signal time series Time-frequency domain transformations were performed to obtain the acoustic and vibration dynamic response characteristics of the vehicle as it passed through the detection area.

[0043] acoustic signals For example, performing a short-time Fourier transform on it yields the acoustic time spectrum. The calculation formula is as follows: , in, n The value range is from 0 to L-1. Indicates the acoustic signal at the th m The first time frame, the first k Complex spectral values ​​at each frequency point L Indicates the length of the sliding window. D Indicates the window sliding step size. w ( n ) represents the window function. j Represents the imaginary unit. m Indicates the time frame number. k Indicates the frequency point number.

[0044] The total number of frames M satisfies: , Where floor(·) represents the floor function, and K represents the number of frequency points.

[0045] 31 pairs of edge computing units The acoustic time-frequency energy spectrum is obtained by squaring the modulus and performing logarithmic scaling. : , in, Indicates the firstm The first time frame, the first k Acoustic time-frequency energy values ​​at each frequency point e A constant to prevent zero values ​​from occurring in logarithmic operations.

[0046] Edge computing unit 31 pairs vibration signal time series Performing the same time-frequency domain transformation yields the vibration time-frequency energy spectrum. .in, and All dimensions are M × K Edge computing unit 31 will and Input the acoustic vibration feature extraction branch separately to obtain the acoustic feature vector. and vibration eigenvectors : , in, d 1 represents the dimension of the acoustic eigenvector and the vibration eigenvector.

[0047] Step S603: Visual thermal imaging feature extraction from visible light image and infrared thermal image. Edge computing unit 31 processes visible light image frames... and infrared thermal image frames Input visual thermal imaging feature extraction branch.

[0048] The visual thermal imaging feature extraction branch includes a visible light image branch and an infrared thermal image branch. The visible light image branch is used to extract at least one visual structural feature from vehicle outline, vehicle type, number of axle groups, wheel position, body size, and vehicle posture; the infrared thermal image branch is used to extract at least one thermal distribution feature from tire temperature, wheel hub temperature, braking area temperature, heat distribution in the undercarriage area, and heat distribution in the cargo area.

[0049] The visible light image branch and the infrared thermal image branch employ lightweight convolutional neural networks with non-shared parameters. These lightweight convolutional neural networks can be MobileNetV3, ShuffleNet, lightweight ResNet, or other lightweight network structures suitable for edge computing platforms. The visible light image branch outputs a visual feature vector. V vis The infrared thermal image branch outputs infrared thermal feature vectors. V ir : , in, d 2 represents the dimension of the visual feature vector and the infrared thermal feature vector. V vis and Vir The dimensions can be the same, or they can be set to different dimensions depending on the network structure.

[0050] Step S604: Multimodal feature fusion based on adaptive channel attention. In order to reduce the impact of reliability fluctuations of a single mode on the load level inference results in complex road environments, the edge computing unit 31 performs weighted fusion of acoustic features, vibration features, visual features and infrared thermal features through the adaptive channel attention fusion module.

[0051] First, the acoustic feature vector V a Vibration characteristic vector V v Visual feature vectors V vis and infrared thermal feature vector V ir By concatenating the features, a joint multimodal feature vector is obtained. V concat : , Where ⊕ represents the vector concatenation operation, d total = 2 d 1+2 d 2.

[0052] Then, V concat Input the adaptive channel attention module to generate channel weight vectors. oh : , in, oh Represents the channel weight vector. W 1 represents the weight matrix of the first fully connected layer. W 2 represents the weight matrix of the second fully connected layer. b 1 represents the bias vector of the first fully connected layer. b 2 represents the bias vector of the second fully connected layer. d (·) represents a non-linear activation function. s (·) represents the Sigmoid activation function.

[0053] W 1 and W The dimensions of 2 can be represented as follows: , in, r This represents the dimensionality reduction scaling factor, used to reduce the number of parameters in the attention module. b The dimension of 1 is dtotal / r , b The dimension of 2 is d total The sigmoid activation function is used to map channel weights to a range of 0 to 1.

[0054] Finally, using the channel weight vector oh For joint multimodal feature vectors V concat Element-wise weighting is performed to obtain the multimodal fusion feature vector. V fuse : , in, This indicates element-wise multiplication.

[0055] Step S605: Load level classification and confidence output. Edge computing unit 31 will fuse the multimodal feature vectors. V fuse Input load level classification module. The load level classification module includes one or more fully connected layers and outputs the predicted probability corresponding to each load level through the Softmax function.

[0056] Let the total number of preset load level categories be . C num The classification module outputs the first... c The original logical value corresponding to each category is z c Then the first c Predicted probabilities for each category P c satisfy: , in, z i The output of the classification module represents the first... i The original logical values ​​of each category; P c satisfy: , In this embodiment C num The value is 5, corresponding to five load levels: unloaded, lightly loaded, medium-loaded, heavily loaded, and suspected overloaded. Alternatively, the load levels can be divided into normal, warning, and suspected overload, or into other numbers of levels, depending on the actual needs of overload control management.

[0057] Edge computing unit 31 selects the category with the highest predicted probability as the truck load level output, and uses the maximum predicted probability as the recognition confidence level for this detection.

[0058] Step S606, Training method of the load grading inversion model. The load grading inversion model is obtained through training with multimodal samples labeled with load levels. The multimodal samples include acoustic signal time series. X a Vibration signal time series X v Visible light image frames I vis Infrared thermal image frames I ir And the corresponding truck load rating label.

[0059] To address the class imbalance issue arising from a large number of normally loaded vehicles and a small number of suspected or severely overloaded vehicles on the road, a focus loss function can be used to optimize the network weights during the model training phase. Total loss function. L total It can be represented as: , in, y c This represents the one-hot encoding of the true label, when the true class of the sample is the first... c Class Time y c =1, otherwise y c =0; α c Indicates the first c The class balancing parameter is used to increase the weight of minority class samples in the loss function; P c The first one obtained in step S605 c Class prediction probability; c This represents the modulation coefficient for easy and difficult samples, used to reduce the loss contribution of easy-to-classify samples during training, making the model pay more attention to difficult-to-classify samples.

[0060] The lightweight load hierarchical inversion model can also be trained using cross-entropy loss function, weighted cross-entropy loss function, label smoothing loss function, or other classification loss function.

[0061] In actual deployment, the lightweight load classification inversion model can be trained offline on the server or training platform, and the trained model parameters can be deployed to the edge computing unit 31. The edge computing unit 31 performs online inference on the road site to achieve real-time output of truck load levels.

[0062] The model structure, feature extraction method, fusion method, and loss function described above are only one preferred implementation. In other embodiments, other model structures capable of multimodal feature extraction, fusion, and load level classification may also be used.

[0063] like Figure 2 As shown, the workflow of the system in this embodiment includes the following steps: Step S1: The truck enters the inspection area. In step S2, the vibration sensor 11 and the acoustic sensor 12 respectively collect vibration signals and acoustic signals, which are then synchronously collected by the acoustic-vibration synchronous acquisition module 21. Step S3: The time synchronization and triggering module 22 provides a unified time reference and sends a hardware trigger signal to the dual-light sensor module 13; Step S4: After receiving the hardware trigger signal, the dual-light sensor module 13 acquires a visible light image of the vehicle through an industrial visible light camera and an infrared thermal image of the vehicle through an infrared thermal imager. In step S5, the edge computing unit 31 receives acoustic signals, vibration signals, visible light images and infrared thermal images, and performs time alignment and preprocessing on the multimodal data according to a unified timestamp. In step S6, the edge computing unit 31 extracts the acoustic and vibration dynamic response features, vehicle visual structural features, and thermal distribution features, and performs fusion reasoning on the above multimodal features to output the truck load level and recognition confidence. In step S7, the communication module 41 uploads the load classification results, identification confidence level and vehicle passage information to the backend platform 50.

[0064] After completing the load level inference, the edge computing unit 31 packages the load classification results, vehicle images, infrared thermal images, license plate information, passage time, lane number, device number, and recognition confidence, and uploads them to the backend platform 50 through the communication module 41.

[0065] The communication module 41 may include a 4G communication unit, a 5G communication unit, an Ethernet communication unit, an optical fiber communication unit, or a combination thereof. The backend platform 50 can store, display, retrieve, alarm, statistically analyze, and manually verify the received data.

[0066] When a vehicle is identified as potentially overloaded or exceeding weight limits, the backend platform 50 can generate an early warning message and push the message to traffic management personnel, the overload control management system, or other external management platforms.

[0067] In the event of a communication link failure, the edge computing unit 31 can temporarily store the detection results and associated images in the local storage unit. Once communication is restored, the device automatically re-transmits the temporarily stored data to the backend platform 50 to ensure the integrity of the detection records.

[0068] Example 9: Protective Structure and On-site Installation Method The device of this invention can be installed on roadside poles, gantry frames, brackets, bridge components, or other fixed installation structures in the road detection area. The dual-light sensor module 13 is positioned facing the vehicle's travel direction or the side of the lane to acquire vehicle appearance images and infrared thermal images. The acoustic sensor 12 can be located outside the protective housing 60 or extend from the housing through the protective structure to acquire acoustic signals when a vehicle passes. The vibration sensor 11 can be installed on the roadside foundation structure, bridge components, guardrail foundation, or road edge structure to acquire vibration signals caused by vehicle passage.

[0069] The protective enclosure 60 may be equipped with a waterproof structure, a dustproof structure, a heat dissipation structure, a lightning surge protection structure, a temperature and humidity monitoring structure, and an anti-condensation structure. In a preferred embodiment, the protective enclosure 60 meets the IP65 protection rating. In other embodiments, the protective enclosure 60 may also adopt IP66, IP67, or other higher protection ratings to meet the long-term operation requirements under different road environments.

[0070] The protective enclosure 60 can house a power module, a heat dissipation duct, a heat conduction structure, a surge protection module, a grounding structure, and an equipment status monitoring module. The power module can be connected to mains power, a solar power system, an energy storage battery system, or a combination thereof, to power the acoustic vibration synchronization acquisition module 21, the time synchronization and triggering module 22, the dual-light sensor module 13, the edge computing unit 31, and the communication module 41.

Claims

1. A multimodal, non-contact edge computing method for truck load sensing, characterized in that, include: Step S1: In response to the truck entering the detection area, vibration signals are collected by vibration sensors and acoustic signals are collected by acoustic sensors. Step S2: A unified time reference is provided by the time synchronization and triggering module, and a hardware trigger signal is sent to the dual-light sensor module. After receiving the hardware trigger signal, the dual-light sensor module acquires a visible light image of the truck through an industrial visible light camera and an infrared thermal image of the truck through an infrared thermal imager. Step S3: The collected vibration signal, acoustic signal, visible light image and infrared thermal image are sent to the edge computing unit; The edge computing unit performs time alignment and preprocessing on the received vibration signal, acoustic signal, visible light image and infrared thermal image according to the unified timestamp corresponding to the unified time base, and constructs multimodal samples; In step S4, the edge computing unit inputs multimodal samples into the lightweight load grading inversion model deployed on it for feature extraction and multimodal fusion inference, and outputs the load level and recognition confidence of the truck. Step S5: Upload the output load level, identification confidence level and vehicle passage information to the backend platform through the communication module.

2. The method according to claim 1, characterized in that, The specific process of constructing multimodal samples in step S3 includes: Hardware trigger time corresponding to hardware trigger signal Based on the sampling frequency of the acoustic vibration synchronous acquisition module... Calculate the time position corresponding to each sampling point or each segment of sampling data in acoustic and vibration signals; Based on the exposure time or frame acquisition time recorded by the industrial visible light camera and infrared thermal imager, determine the acoustic signal segment and vibration signal segment corresponding to each frame of visible light image and infrared thermal image. Within the preset time window Δ t Internally, extract the acoustic signal sequence. and vibration signal sequence And select the timestamp that falls into the time window Δ t Visible light image frames within and infrared thermal image frames The combination constitutes a multimodal sample X. .

3. The method according to claim 2, characterized in that, The specific process of feature extraction and multimodal fusion inference in step S4 includes: Step S601, for the acoustic signal sequence and vibration signal sequence Perform short-time Fourier transforms and then perform modulo-squared and logarithmic scaling to obtain the acoustic time-frequency energy spectrum. F a Vibration time-frequency energy spectrum F v ;Will F a and F v Input the acoustic and vibration feature extraction branches of the lightweight load grading inversion model respectively, and output the acoustic feature vectors. V a and vibration eigenvectors V v ; Step S602, frame the visible light image. and infrared thermal image frames The visual thermal imaging feature extraction branch of the input lightweight load grading inversion model extracts the vehicle's visual structural features and thermal distribution features through a parameter-discretion-free lightweight convolutional neural network, and outputs a visual feature vector. V vis and infrared thermal feature vector V ir ; Step S603, convert the acoustic feature vector V a Vibration characteristic vector V v Visual feature vectors V vis and infrared thermal feature vector V ir By concatenating the features, a joint multimodal feature vector is obtained. V concat ;Will V concat The input adaptive channel attention fusion module generates channel weight vectors. ω and utilize ω right V concat Element-wise weighting is performed to obtain the multimodal fusion feature vector. V fuse ; Step S604: Fuse the multimodal feature vectors V fuse The load level classification module is input, and the predicted probability corresponding to each preset load level is calculated and output through a fully connected layer and a Softmax function. The category with the highest predicted probability is selected as the load level output for trucks, and the maximum predicted probability is used as the recognition confidence level.

4. The method according to claim 3, characterized in that, The preset load levels include five levels: no load, light load, medium load, heavy load, and suspected overload.

5. The method according to claim 4, characterized in that, The lightweight load grading inversion model introduces a focus loss function for network weight optimization during the training phase, and the total loss function... L total Represented as ， in, C num This represents the total number of preset load level categories; y c This represents the one-hot encoding of the true label, when the true class of the sample is the first... c Class Time y c =1, otherwise y c =0; α c Indicates the first c Class balance parameter, P c For the first c Class prediction probability, γ This represents the modulation coefficients for easy and difficult samples.

6. A multimodal, non-contact truck load sensing edge computing system, characterized in that, include: The sensing layer (10) includes a vibration sensor (11), an acoustic sensor (12), and a dual-light sensor module (13). The vibration sensor (11) is used to collect vibration signals when the truck passes through the detection area. The acoustic sensor (12) is used to collect acoustic signals when the truck passes through the detection area. The dual-light sensor module (13) includes an industrial visible light camera and an infrared thermal imager, which are used to collect visible light images and infrared thermal images when the truck passes through the detection area, respectively. Data acquisition and synchronization layer (20): includes an acoustic and vibration synchronization acquisition module (21) and a time synchronization and triggering module (22); the acoustic and vibration synchronization acquisition module (21) is connected to the vibration sensor (11) and the acoustic sensor (12) and is used to condition, convert analog to digital and sample synchronously the received analog signals and output synchronous acoustic and vibration data; the time synchronization and triggering module (22) is used to provide a unified time reference and send hardware trigger signals to the dual-light sensor module (13); Edge computing layer (30): includes edge computing unit (31), which is connected to acoustic vibration synchronous acquisition module (21) and dual light sensor module (13) respectively. It is used to receive synchronous acoustic vibration data, visible light image and infrared thermal image, and perform time alignment, preprocessing, feature extraction, multimodal fusion and load level inference according to unified timestamp, and output the load level and recognition confidence of truck. Communication module (41): It is connected to the edge computing unit (31) for uploading the load level, identification confidence and vehicle passage information output by the edge computing unit (31); And the back-end platform (50): communicates with the communication module (41) to receive and store, display, retrieve, warn and manually review the uploaded data.

7. The system according to claim 6, characterized in that, The acoustic and vibration synchronous acquisition module (21) includes an FPGA control unit, a multi-channel analog-to-digital converter, an ICP microphone interface, an ICP accelerometer interface, a signal conditioning circuit, and a local buffer unit; the FPGA control unit is used to generate a unified sampling clock and control the synchronous sampling of multiple acoustic channels and multiple vibration channels.

8. The system according to claim 6, characterized in that: The time synchronization and triggering module (22) includes a precision time synchronization unit, a hardware triggering circuit and a timestamp recording unit; the precision time synchronization unit adopts any one of the following to achieve microsecond-level clock synchronization: IEEE 1588 precision time protocol, GPS time synchronization, Beidou time synchronization, network time synchronization or local high-precision clock source; the hardware triggering circuit is used to send trigger pulses to industrial visible light cameras and infrared thermal imagers to achieve synchronous exposure or frame acquisition.

9. The system according to claim 6, characterized in that: The edge computing unit (31) is equipped with a lightweight load hierarchical inversion model, which includes an acoustic vibration feature extraction branch, a visual thermal imaging feature extraction branch, an adaptive channel attention fusion module, and a load level classification module. The visual thermal imaging feature extraction branch includes a visible light image branch and an infrared thermal image branch, both of which adopt a lightweight convolutional neural network structure with non-shared parameters.

10. The system according to claim 6, characterized in that, It also includes a protective enclosure (60), which is used to house the acoustic and vibration synchronization acquisition module (21), the time synchronization and triggering module (22), the edge computing unit (31), the communication module (41), and power supply, heat dissipation, lightning protection and surge protection components; the protective enclosure (60) has a protection level of IP65 or above and is fixedly installed on the roadside poles, gantry, brackets or bridge components in the road detection area.

Images