Machine learning assisted road multi-mode terminal combined inspection intelligent network

Abstract

The invention discloses a machine learning assisted road multi-mode terminal combined inspection intelligent network. In a road scene, multi-mode communication and sensing terminals such as an unmanned aerial vehicle group, a millimeter wave radar, a camera and a sensor are combined to perform combined inspection on a road. Routing inspection information is shared and received in a centralized manner through the routing inspection intelligent network, and is transmitted to the routing inspection intelligent cloud. And the inspection smart cloud performs multi-mode identification and correction on the data acquired by the multi-mode terminal based on a machine learning algorithm, acquires information such as vehicle types, vehicle density and vehicle speeds on the road, and displays the information on the road visualization terminal. Meanwhile, the inspection smart cloud sends regulation and control information to the multi-mode terminal based on the road information, and controls the position, orientation and working mode of each terminal so as to adapt to the inspection requirement of the current road. Finally, the invention realizes an intelligent network with machine learning assistance, multi-mode terminal joint inspection and adaptive real-time regulation and control.

Description

Technical Field

[0001] The present invention belongs to the fields of wireless communication and intelligent transportation, and particularly relates to a machine learning-assisted intelligent network for joint inspection of multi-mode terminals on highways. Background Art

[0002] With the rapid development of global transportation infrastructure, transportation networks such as highways and urban expressways are becoming increasingly complex. Highway inspection is crucial for ensuring road safety and improving traffic management efficiency. Traditional highway inspection methods mainly rely on manual inspections or fixed monitoring devices such as cameras, radars, and ground sensors. These methods have many limitations. Manual inspections usually require inspectors to drive vehicles for inspections, with limited inspection ranges, poor real-time performance, and great difficulties in inspections under adverse weather or complex traffic conditions, posing certain safety hazards. Devices such as fixed cameras and millimeter-wave radars can only monitor roads within a fixed range, are greatly restricted by environmental conditions, cannot flexibly adjust the monitoring angle or position, and are difficult to meet the inspection requirements in large-scale and complex environments. At the same time, different inspection terminals usually operate independently, with different data formats and communication protocols, making it difficult to achieve data sharing and fusion processing, resulting in low utilization of inspection information and affecting the overall monitoring effect. In addition, existing highway inspection systems mainly rely on preset rules for data analysis, are difficult to adapt to complex and changeable traffic environments, lack intelligent and adaptive control mechanisms, and cannot dynamically optimize the allocation and scheduling of inspection resources.

[0003] In recent years, with the rapid development of artificial intelligence, unmanned aerial vehicles, the Internet of Things, and 5G communication technologies, new technical supports have been provided for intelligent highway inspection. A multi-mode inspection network based on machine learning can integrate multiple sensor terminals such as unmanned aerial vehicle swarms, millimeter-wave radars, cameras, and ground sensors, and through intelligent algorithms, achieve the fusion processing, dynamic adjustment, and optimal decision-making of inspection information. Therefore, there is an urgent need for a multi-mode intelligent network for joint inspection of terminals that can improve data processing capabilities while ensuring inspection accuracy, realize the adaptive control of inspection terminals, and thus improve the intelligent level and efficiency of highway inspection. Summary of the Invention

[0004] The object of the present invention is to provide a machine learning-assisted intelligent network for joint inspection of multi-mode terminals on highways. This network combines multi-mode communication and sensing terminals such as unmanned aerial vehicle swarms, millimeter-wave radars, cameras, and sensors to conduct intelligent joint inspections of highways. At the same time, the inspection intelligent cloud performs adaptive feedback regulation on the communication and sensing terminals on the highway according to the inspection information, so as to adapt to the complex and changeable inspection requirements of highways.

[0005] To achieve the above object, the technical solution adopted by the present invention is: a machine learning-assisted intelligent network for joint inspection of multi-mode terminals on highways, including a plurality of intelligent inspection terminals, an inspection intelligent cloud, and an inspection communication network; The inspection intelligent cloud includes a feature data extraction module, a machine learning network, a feedback information processing and generation module, an inspection control system, and a visualization terminal, which is responsible for the intelligent analysis and decision-making of inspection data; The inspection communication network includes an inspection signal receiving end, a feedback signal transmitting end, and a signal processing module, which is responsible for establishing a stable data communication link between the intelligent inspection terminal and the inspection intelligent cloud, and supporting data transmission, sharing, and regulation; The feature data extraction module is connected to the machine learning network and the signal processing module, and is responsible for receiving the inspection information processed by the signal processing module and extracting effective feature data and transmitting it to the machine learning network; the machine learning network is connected to the feature data extraction module, the feedback information processing and generation module, and the inspection control system, and is responsible for training the machine learning network based on the feature data; the feedback information processing and generation module is connected to the machine learning network and the inspection control system, and is responsible for generating adaptive feedback information based on the currently identified highway conditions to adjust the state of the intelligent inspection terminal; the inspection control system is connected to the machine learning network, the feedback information processing and generation module, and the visualization terminal, and is responsible for overall controlling the work of the feature data extraction module, the machine learning network, the feedback information processing and generation module, the inspection control system, and the visualization terminal in the inspection intelligent cloud, and at the same time organizing the output information and displaying it on the visualization terminal; the visualization terminal is connected to the inspection control system; The inspection signal receiving end is connected to the signal processing module and receives signals from each intelligent inspection terminal; the feedback signal transmitting end is connected to the signal processing module and the feedback information processing and generation module of the inspection intelligent cloud, and is responsible for sending feedback signals to each intelligent inspection terminal; the signal processing module is connected to the inspection signal receiving end and the feedback signal transmitting end, and is responsible for demodulating and decoding the inspection signal and modulating and encoding the feedback signal.

[0006] Furthermore, the intelligent inspection terminal includes a drone swarm, a millimeter-wave radar, a camera, and a ground sensor; the drone swarm, the millimeter-wave radar, the camera, and the ground sensor transmit inspection signals to the inspection signal receiving end in the inspection communication network, and at the same time receive feedback signals from the feedback signal transmitting end for adaptive adjustment.

[0007] Furthermore, the visualization terminal is equipped with an operation panel.

[0008] The present invention also discloses a method for implementing a machine learning-assisted intelligent network for joint inspection of multi-mode terminals on highways, including the following steps: Step 1: Multiple intelligent inspection terminals respectively obtain road environment information; Step 2: The intelligent inspection terminals send the obtained environment information to the inspection signal receiving end in the inspection communication network. After being processed by the signal processing module, the information is uploaded to the inspection intelligent cloud. The inspection intelligent cloud extracts feature data from the multi-modal highway information obtained from the intelligent inspection terminals through the feature data extraction module, and sends the feature data into the machine learning network for training. At the same time, based on the machine learning algorithm, vehicle features, traffic flow changes, and traffic anomalies are identified, and error data is corrected to improve the inspection accuracy; Step 3: The inspection intelligent cloud generates real-time control instructions according to the recognition results, and sends the instructions to each intelligent inspection terminal through the feedback signal transmitting end in the inspection communication network. After receiving the instructions, each intelligent inspection terminal adaptively adjusts the inspection strategy to optimize the inspection coverage and accuracy; Step 4: After receiving the control instructions, all intelligent inspection terminals adjust their own parameters in real time, continue to collect data and send the new data back to the inspection intelligent cloud, forming a closed-loop optimized inspection mechanism to realize the intelligent, adaptive and efficient operation of highway inspection; Step 5: The highway monitoring center displays the inspection results based on the visualization terminal in the inspection intelligent cloud. Provide real-time road conditions, traffic flow analysis and abnormal event warnings to assist traffic management personnel in making decisions.

[0009] Further, the road environment information obtained in Step 1 includes vehicle type, vehicle density, vehicle speed, and road condition information; the intelligent inspection terminals include a drone swarm, millimeter-wave radar, cameras, and ground sensors. The drone swarm and cameras capture highway image information from multiple angles, the millimeter-wave radar estimates the vehicle position information in a large range, and the ground sensors detect the vehicle passing situation and road surface conditions.

[0010] Further, in Step 1, the received signal obtained by the millimeter-wave radar cannot be directly used for the machine learning network. Therefore, it is necessary to obtain information such as the vehicle target position and speed based on the received signal, which specifically includes the following steps: The millimeter-wave radar uses frequency-modulated continuous wave FMCW, and its transmitted signal is expressed as: ; Among them, A is the signal amplitude, t is the time variable, f c is the carrier frequency, is the frequency modulation slope, B is the bandwidth, T is the modulation period; Step 1.1: Distance estimation The target reflected echo signal is delayed by τReturn: ; wherein, A r is the amplitude of the reflected echo signal, is the round-trip propagation time, R is the target distance, c is the speed of light; Mix the transmitted signal and the echo signal, and the mixed signal is: ; After low-pass filtering, an intermediate-frequency signal is obtained, and its frequency is : ; Thus, the target distance is obtained: ; Step 1.2, Velocity Estimation When the target has a velocity v , a Doppler frequency shift f D will be generated: ; Use pulse Doppler processing FFT to analyze the multi-linear frequency modulation signal to obtain the velocity v : ; Step 1.3, Angle Estimation The millimeter-wave radar uses multiple receiving antennas to form an array, and there is a phase difference in the signals at each receiving antenna m : ; wherein, d is the array antenna spacing, is the wavelength, is the direction of arrival DOA of the target; Use the MUSIC algorithm for angle estimation, construct the covariance matrix R, and perform eigenvalue decomposition: ; wherein, is the noise subspace, is the array manifold vector, is the direction of arrival spectrum function; find through peak search to obtain the target angle information.

[0011] Furthermore, in step 2, based on the machine learning algorithm, identifying vehicle features, traffic flow changes, traffic anomalies, and correcting error data specifically includes the following steps: Step 2.1, Input Data Preprocessing Given the input data , where N is the number of samples, d and d is the feature dimension of each sample; First, perform normalization. The normalized input data is : ; where is the mean, is the i th input data, is the standard deviation; Step 2.2, Neural Network Feature Extraction Set to use a multi-layer perceptron MLP or a convolutional neural network CNN to extract features. The forward propagation of the neural network is expressed as: Step 2.2.1, Feature Extraction of the Fully Connected Layer The calculation of each layer is as follows: ; where is the feature of the l th layer, is the feature of the l th layer, is the weight matrix, d l is the matrix dimension, is the bias vector, is the non-linear activation function, ; Finally, in the feature extraction layer Bottleneck Layer of the Lth layer, the extracted high-dimensional feature vector is expressed as: ; Step 2.2.2, Feature Extraction of the Convolutional Neural Network CNN For image or time series data, use CNN for feature extraction. The calculation formula of the convolutional layer is: ; where represents the convolution operation; After the feature passes through pooling, the information with the highest local weight is extracted, and finally it is converted into a feature vector in the fully connected layer; Step 2.3, Feature Dimensionality Reduction If the feature dimension is too high, use principal component analysis PCA or autoencoder Autoencoder for dimensionality reduction; Step 2.3.1, PCA Dimensionality Reduction Construct the covariance matrix: ; Solve the eigenvalue decomposition: ; where are the eigenvalues, are the eigenvectors. Select the eigenvectors corresponding to the top k largest eigenvalues . The final feature Z is: ; Step 2.3.2, Autoencoder dimensionality reduction The Autoencoder uses a neural network to encode and decode the input. The final feature Z is expressed as: ; where is the encoder weight matrix, is the encoder bias; Step 2.4, Feature fusion and classification The extracted features Z are used for classification, clustering, or detection tasks. If a neural network classifier is used, the calculation result of the final output layer is as follows: ; and are the classifier parameters. The Softmax calculates the class probabilities: ; where is the true class label; Finally, use the cross-entropy loss for optimization: ; where is the predicted class probability; Send the extracted feature data into a machine learning network for training. The trained machine learning network is used for multi-modal recognition.

[0012] Furthermore, in step 3, adaptively adjust the inspection strategy, including adjusting the UAV flight trajectory, camera orientation, and radar scanning angle; the recognition results on which the control command generation is based include vehicle position, vehicle density, vehicle size and weight, and the recognition of traffic jams and vehicle collisions.

[0013] A machine learning-assisted intelligent network for joint inspection of multi-mode terminals on highways of the present invention has the following advantages: Compared with traditional highway inspection methods, the present invention provides an intelligent and adaptive inspection network, greatly improving the inspection efficiency, data fusion ability, and intelligent control ability of inspection terminals. The specific manifestations are as follows: 1. The system of the present invention improves the collection accuracy and coverage of highway environmental information through joint inspection of multi-mode terminals, overcomes the limitations of single inspection means, and realizes more comprehensive road monitoring.

[0014] 2. Different from the traditional inspection method that relies on manual labor and fixed monitoring equipment, the present invention performs fusion analysis on multi-mode data based on machine learning algorithms, can accurately identify traffic flow, vehicle types, and abnormal events, and improves the reliability and practicality of data.

[0015] 3. The inspection intelligent network of the present invention can realize real-time transmission and sharing of data, enabling each inspection terminal and base station to work together, and adaptively adjusting the inspection strategy according to the road conditions, improving the flexibility of the inspection system.

[0016] 4. The base station can dynamically optimize the working mode of the inspection terminal through control instructions, adjust the flight path of the drone, the orientation of the camera, and the radar scanning range, reduce the waste of inspection resources, and improve the overall inspection efficiency.

[0017] 5. The present invention adopts an intelligent visual monitoring system to display the highway inspection results in real time, provides an efficient and intuitive decision-making basis for traffic management departments, and helps to improve the highway management level.

[0018] 6. The entire system is based on existing machine learning, wireless communication, drone, and intelligent sensing technologies, is easy to implement and promote, and has high practical value and application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a block diagram of the intelligent network for joint inspection of multi-mode terminals on highways assisted by machine learning; Figure 2 is a block diagram of the intelligent cloud for inspection; Figure 3 is a block diagram of the inspection communication networking; Figure 4 is a schematic diagram of the machine learning network structure; Figure 5 is a flowchart of the working process of the closed-loop optimized inspection mechanism in the inspection intelligent network. DETAILED DESCRIPTION OF THE INVENTION

[0020] The present invention will be further described in detail below with reference to the accompanying drawings.

[0021] The intelligent network for joint inspection of highway multi-mode terminals proposed by the present invention consists of multiple intelligent inspection terminals, an inspection intelligent cloud, and an inspection communication network. The overall block diagram of the system of the present invention is as shown in Figure 1 shown. Among them, the block diagram of the structure of the inspection intelligent cloud is as shown in Figure 2 shown, and the block diagram of the structure of the inspection communication network is as shown in Figure 3 shown.

[0022] The intelligent inspection terminal consists of a drone swarm, a millimeter-wave radar, a camera, and ground sensors, which are various modes of communication and sensing devices, and are used to conduct all-round inspections of the highway environment and obtain multi-modal information. Among them, the drone swarm, millimeter-wave radar, camera, and ground sensors transmit inspection signals to the inspection signal receiving end in the inspection communication network, and at the same time receive feedback signals from the feedback signal transmitting end for adaptive adjustment.

[0023] The inspection intelligent cloud consists of a feature data extraction module, a machine learning network, a feedback information processing and generation module, an inspection control system, and a visualization terminal, and is responsible for the intelligent analysis and decision-making of inspection data. Among them, the feature data extraction module is connected to the machine learning network and the signal processing module in the inspection communication network, and is responsible for receiving the inspection information after signal processing and extracting effective feature data and transmitting it to the machine learning network. The machine learning network is connected to the feature data extraction module, the feedback information processing and generation module, and the inspection control system, and is responsible for training the machine learning network based on the feature data, so as to identify traffic flow, vehicle types, and abnormal events in actual applications. The feedback information processing and generation module is connected to the machine learning network and the inspection control system, and is responsible for generating adaptive feedback information based on the currently identified highway conditions to adjust the state of the intelligent inspection terminal. The inspection control system is connected to the machine learning network, the feedback information processing and generation module, and the visualization terminal, and is responsible for overall controlling the work of each module of the inspection intelligent cloud, and at the same time sorting out the information output by each module and displaying it on the visualization terminal. The visualization terminal is connected to the inspection control system, and is responsible for displaying the highway inspection results to the user. At the same time, it can be equipped with an operation panel for the user to directly control the inspection intelligent cloud.

[0024] The inspection communication network consists of an inspection signal receiving end, a feedback signal transmitting end, and a signal processing module, and is responsible for establishing a stable data communication link between the intelligent inspection terminal and the inspection intelligent cloud, and supporting data transmission, sharing, and regulation. Among them, the inspection signal receiving end is connected to the signal processing module and receives signals from each intelligent inspection terminal. The feedback signal transmitting end is connected to the signal processing module and the feedback information processing and generation module of the inspection intelligent cloud, and is responsible for sending feedback signals to each intelligent inspection terminal. The signal processing module is connected to the inspection signal receiving end and the feedback signal transmitting end, and is responsible for demodulating and decoding the inspection signals and modulating and encoding the feedback signals.

[0025] The following is a more detailed description of each step in conjunction with the accompanying drawings.

[0026] In Step 1, multiple intelligent inspection terminals respectively obtain road environment information, including vehicle type, vehicle density, vehicle speed, road conditions and other information. The drone swarm and cameras can capture road image information from multiple angles, the millimeter-wave radar estimates the vehicle position information in a large range, and the ground sensors detect the passing situation of vehicles and the road surface state. Here, the information obtained by the drone swarm and cameras is usually in the form of video, and the captured video information needs to be converted into multiple pictures and transmitted to the inspection intelligent cloud. The information obtained by the ground sensors is an electrical signal. For example, when a vehicle passes, it is "1", and when no vehicle passes, it is "0". After sampling at a certain time slot, it can be sent to the inspection intelligent cloud for the machine learning network.

[0027] In Step 1, the received signal obtained by the millimeter-wave radar cannot be directly used for the machine learning network. Therefore, it is necessary to obtain information such as the target position and speed based on the received signal. The following is an example estimation method: The millimeter-wave radar usually uses frequency-modulated continuous wave (FMCW), and its transmitted signal can be expressed as: ; where A is the signal amplitude, t is the time variable, f c is the carrier frequency, is the frequency modulation slope, B is the bandwidth, T is the modulation period; 1. Distance estimation The target reflected echo signal returns after a delay τ : ; where A r is the reflected echo signal amplitude, is the round-trip propagation time, R is the target distance, c is the speed of light; The transmitted signal and the echo signal are mixed (difference frequency) to obtain the mixed signal as: ; After low-pass filtering, the intermediate frequency signal is obtained, and its frequency is : ; From this, the target distance is obtained: ; 2. Speed Estimation When the target has a speed v Doppler frequency shift will occur f D : ; Pulse Doppler processing (FFT) is used to analyze the multi-linear frequency modulation signal to obtain the speed v : ; 3. Angle Estimation The millimeter-wave radar uses multiple receiving antennas to form an array, and there is a phase difference in the signals at each receiving antenna m : ; Among them, d is the array antenna spacing, is the wavelength, is the direction of arrival (DOA) of the target; The MUSIC algorithm is used for angle estimation, the covariance matrix R is constructed, and eigenvalue decomposition is performed: ; Among them, is the noise subspace, is the array manifold vector, is the direction of arrival spectrum function; the peak search is used to find to obtain the target angle information.

[0028] In Step 2, the intelligent inspection terminal sends the acquired environmental information to the inspection signal receiving end in the inspection communication network. After being processed by the signal processing module, the information is uploaded to the inspection intelligent cloud. The inspection intelligent cloud extracts feature data from the multi-modal highway information obtained from the intelligent inspection terminal through the feature data extraction module. At the same time, based on the machine learning algorithm, vehicle features, traffic flow changes, traffic anomalies, etc. are identified, and incorrect data is corrected to improve the inspection accuracy.

[0029] The following is a method for extracting machine learning feature data in Step 2: 1. Input Data Preprocessing Given the input data , where N is the number of samples, d is the feature dimension of each sample; To improve the training effect, standardization processing is first performed. The input data after standardization processing (zero mean, unit variance) is : ; Among them, is the mean, is the i th input data, is the standard deviation; 2. Neural Network Feature Extraction It is set to extract features using a multi - layer perceptron MLP or a convolutional neural network CNN. The forward propagation of the neural network is expressed as: (1) Feature Extraction of the Fully - Connected Layer The calculation of each layer is as follows: ; Among them, is the feature of the l th layer, is the feature of the l th layer, is the weight matrix, d l is the matrix dimension, is the bias vector, is the non - linear activation function, ; Finally, in the Bottleneck Layer of the feature extraction layer at the Lth layer, the extracted high - dimensional feature vector is expressed as: ; (2) Feature Extraction of the Convolutional Neural Network CNN For image or time - series data, use CNN for feature extraction. The calculation formula of the convolutional layer is: ; Among them, represents the convolution operation; After the feature passes through pooling, the information with the highest local weight is extracted, and finally it is converted into a feature vector in the fully - connected layer; (3) Feature Dimensionality Reduction (PCA or Autoencoder) If the feature dimension is too high, use principal component analysis PCA or autoencoder Autoencoder for dimensionality reduction; (1) PCA Dimensionality Reduction Construct the covariance matrix: ; Solve the eigenvalue decomposition: ; Among them are the eigenvalues, are the eigenvectors. Select the eigenvectors corresponding to the first k largest eigenvalues , and the final feature Z is: ; (2)Autoencoder dimensionality reduction Autoencoder encodes and decodes the input using a neural network, and the final feature Z is expressed as: ; where, is the encoder weight matrix, is the encoder bias; 4. Feature fusion and classification The extracted feature Z is used for classification, clustering or detection tasks; if a neural network classifier is used, the calculation result of the final output layer is as follows: ; and are the classifier parameters, and Softmax calculates the class probabilities: ; where, is the true class label; finally, cross-entropy loss is used for optimization: ; where, is the predicted class probability.

[0030] In step two, the extracted feature data can be fed into a machine learning network for training, and the trained machine learning network can be used for multi-modal recognition.

[0031] The machine learning network is currently very mature, and its structural schematic diagram is as Figure 4 shown, and its basic principle is as follows: The machine learning network is a data-driven intelligent computing model that can automatically learn features from training data and make predictions or decisions. It is usually composed of multiple layers of neurons, and each neuron simulates the neurons of the human brain and transmits and processes information through weight connections. The machine learning network generally consists of an input layer, a hidden layer and an output layer: Input layer: responsible for receiving raw data, such as images, texts, sensor data, etc.

[0032] Hidden layer: gradually extracts the deep features of the data through a series of non-linear transformations and calculations. The hidden layer can contain multiple neuron layers, and the output of each layer is used as the input of the next layer.

[0033] Output layer: generates the final prediction or classification result according to the calculation results of the previous layers, such as identifying object categories, analyzing traffic flow, detecting highway anomalies, etc.

[0034] The core workflow of a machine learning network can be summarized in the following steps: (1) Forward propagation During forward propagation, the input data passes through each layer of the network in sequence. Each layer performs a weighted calculation on the data and undergoes a non-linear transformation through an activation function. In this way, the network can gradually extract important features from the data and finally generate a prediction result at the output layer.

[0035] (2) Loss calculation To measure the error between the network's prediction result and the true result, the system calculates a loss value. The loss function is used to describe the performance of the model under the current parameter configuration. The smaller the loss value, the more accurate the network's prediction.

[0036] (3) Backward propagation The network calculates the impact of the error on the weights of each neuron through the backward propagation algorithm and adjusts the network parameters according to this impact to continuously optimize the model and gradually improve the prediction accuracy.

[0037] (4) Parameter update To optimize the network performance, algorithms such as gradient descent are used to adjust the weights and biases to gradually reduce the loss value. By continuously repeating this process, the model will gradually converge and learn how to make more accurate predictions.

[0038] Depending on the application scenario and task, machine learning networks can adopt different structures: (1) Feedforward neural network (FNN): The most basic neural network, where data flows from the input layer to the output layer, suitable for regression and classification tasks.

[0039] (2) Convolutional neural network (CNN): Mainly used for image processing, capable of automatically learning image features such as edges, shapes, and textures.

[0040] (3) Recurrent neural network (RNN): Suitable for processing time series data, such as speech recognition and text generation.

[0041] (4) Transformer network: Based on the self-attention mechanism, widely used in natural language processing and computer vision tasks, such as GPT and BERT.

[0042] In step three, the intelligent cloud for inspection generates real-time control instructions based on the recognition results and sends the instructions to each intelligent inspection terminal through the feedback signal transmitter in the inspection communication network. After receiving the feedback signal, each intelligent inspection terminal adaptively adjusts the inspection strategy, including adjusting the flight trajectory of the drone, the orientation of the camera, the radar scanning angle, etc., to optimize the inspection coverage and accuracy.

[0043] The recognition results for generating control instructions in Step 3 include, but are not limited to, vehicle position, vehicle density, vehicle size and weight, and the recognition of special events such as traffic jams and vehicle collisions. Based on the status of the recognition results, the network sends feedback signals from the intelligent inspection terminal, such as allocating more drones to areas with high vehicle density, adjusting the orientation of millimeter-wave radars, and tracking large-size and heavy-weight vehicles directionally, etc.

[0044] In Step 4, after all intelligent inspection terminals adjust the inspection strategies, they continue to collect data and transmit the new data back to the inspection intelligent cloud, forming a closed-loop optimized inspection mechanism to achieve the intelligent, adaptive, and efficient operation of highway inspections.

[0045] Figure 5 The working process of the closed-loop optimized inspection mechanism in Step 4 is shown. If the new data transmitted back by the intelligent inspection terminal is not ideal after adjusting the inspection strategy in Step 4, the inspection intelligent cloud needs to increase the control amplitude and dispatch other intelligent inspection terminals to complete the target.

[0046] In Step 5, the highway monitoring center displays the inspection results based on the visualization terminal in the inspection intelligent cloud. It provides real-time road conditions, traffic flow analysis, and early warnings for abnormal events to assist traffic management personnel in making decisions.

[0047] The visualization terminal in Step 5 is connected to the inspection control system in the inspection intelligent cloud. It can be equipped with a control panel, allowing operators to directly control each module of the inspection intelligent cloud and manually dispatch the resources of the intelligent inspection terminal to meet specific inspection requirements.

[0048] The present invention provides an intelligent network for joint inspection of highway multi-mode terminals assisted by machine learning. In a highway scenario, it jointly inspects the highway by integrating multi-mode communication and sensing terminals such as drone swarms, millimeter-wave radars, cameras, and sensors. The inspection information is shared and centrally received through the inspection intelligent network and transmitted to the inspection intelligent cloud. The inspection intelligent cloud performs multi-modal recognition and correction on the data obtained by the multi-mode terminals based on machine learning algorithms, obtains information such as vehicle types, vehicle density, and vehicle speeds on the highway, and displays it on the highway visualization terminal. At the same time, based on the highway information, the inspection intelligent cloud sends control information to the multi-mode terminals to control the positions, orientations, and working modes of each terminal to meet the current highway inspection requirements.

[0049] It is understood that the present invention is described by way of some embodiments. Those skilled in the art will appreciate that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the present invention. Additionally, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application belong to the scope protected by the present invention.

Claims

1. A machine learning-assisted intelligent network for joint inspection of multi-mode terminals on highways, characterized in that, It includes multiple intelligent inspection terminals, an inspection intelligent cloud, and an inspection communication network; The inspection intelligent cloud includes a feature data extraction module, a machine learning network, a feedback information processing and generation module, an inspection control system, and a visualization terminal, which is responsible for the intelligent analysis and decision-making of inspection data; The inspection communication network includes an inspection signal receiving end, a feedback signal transmitting end, and a signal processing module, which is responsible for establishing a stable data communication link between the intelligent inspection terminal and the inspection intelligent cloud; The feature data extraction module is connected to the machine learning network and the signal processing module, and is responsible for receiving the inspection information processed by the signal processing module and extracting the feature data and transmitting it to the machine learning network; The machine learning network is connected to the feature data extraction module, the feedback information processing and generation module, and the inspection control system, and is responsible for training the machine learning network based on the feature data; the feedback information processing and generation module is connected to the machine learning network and the inspection control system, and is responsible for generating adaptive feedback information based on the currently recognized road conditions; the inspection control system is connected to the machine learning network, the feedback information processing and generation module, and the visualization terminal, and is responsible for overall controlling the work of the feature data extraction module, the machine learning network, the feedback information processing and generation module, the inspection control system, and the visualization terminal of the inspection intelligent cloud, and at the same time organizing the output information and displaying it on the visualization terminal; the visualization terminal is connected to the inspection control system; The inspection signal receiving end is connected to the signal processing module and receives signals from each intelligent inspection terminal; the feedback signal transmitting end is connected to the signal processing module and the feedback information processing and generation module of the inspection intelligent cloud, and is responsible for sending feedback signals to each intelligent inspection terminal; The signal processing module is connected to the inspection signal receiving end and the feedback signal transmitting end, and is responsible for demodulating and decoding the inspection signal and modulating and encoding the feedback signal.

2. The intelligent network for joint inspection of highway multi-mode terminals assisted by machine learning according to claim 1, characterized in that, The intelligent inspection terminal includes a drone swarm, a millimeter-wave radar, a camera, and a ground sensor; the drone swarm, the millimeter-wave radar, the camera, and the ground sensor transmit inspection signals to the inspection signal receiving end in the inspection communication network, and at the same time receive feedback signals from the feedback signal transmitting end and make adaptive adjustments.

3. The intelligent network for joint inspection of highway multi-mode terminals assisted by machine learning according to claim 1, characterized in that, The visualization terminal is equipped with an operation panel.

4. A method for implementing a machine learning-assisted intelligent network for joint inspection of multi-mode terminals on highways, using a machine learning-assisted intelligent network for joint inspection of multi-mode terminals on highways according to any one of claims 1-3, characterized in that, It includes the following steps: Step 1: Multiple intelligent inspection terminals respectively obtain road environment information; Step 2: The intelligent inspection terminal sends the obtained environmental information to the inspection signal receiving end in the inspection communication network, and the information is uploaded to the inspection intelligent cloud after being processed by the signal processing module; the inspection intelligent cloud extracts feature data from the multi-modal road information obtained from the intelligent inspection terminal through the feature data extraction module, and sends the feature data into the machine learning network for training; at the same time, based on the machine learning algorithm, vehicle features, traffic flow changes, and traffic anomalies are identified, and incorrect data is corrected; Step 3: The inspection intelligent cloud generates real-time control instructions according to the recognition results, and sends the instructions to each intelligent inspection terminal through the feedback signal transmitting end in the inspection communication network; After each intelligent inspection terminal receives the instructions, it adaptively adjusts the inspection strategy to optimize the inspection coverage and accuracy; Step 4: After receiving the regulation instructions, all intelligent inspection terminals adjust their own parameters in real time, continue to collect data and transmit the new data back to the inspection intelligent cloud, forming a closed-loop optimized inspection mechanism; Step 5: Based on the visualization terminal in the inspection intelligent cloud, display the inspection results.

5. The implementation method of a machine learning-assisted intelligent network for joint inspection of multi-mode terminals on highways according to claim 4, characterized in that, The road environment information obtained in Step 1 includes vehicle type, vehicle density, vehicle speed, and road condition information; the intelligent inspection terminal includes a drone swarm, millimeter-wave radar, cameras, and ground sensors; the drone swarm and cameras capture road image information from multiple angles, the millimeter-wave radar estimates the vehicle position information in a large range, and the ground sensors detect the passing of vehicles and the road surface state.

6. The implementation method of a machine learning-assisted intelligent network for joint inspection of multi-mode terminals on highways according to claim 5, characterized in that, In Step 1, the millimeter-wave radar obtains vehicle position information, specifically including the following steps: The millimeter-wave radar uses Frequency Modulated Continuous Wave (FMCW), and its transmitted signal is expressed as: ； Among them, A is the signal amplitude, t is the time variable, f c is the carrier frequency, is the frequency modulation slope, B is the bandwidth, T is the modulation period; Step 1.1: Distance estimation The target echo signal returns after being delayed τ and returns as follows: ； Among them, A r is the amplitude of the reflected echo signal, is the round-trip propagation time, R is the target distance, c is the speed of light; Mix the transmitted signal and the echo signal, and the signal after mixing is as follows: ； The intermediate frequency signal is obtained through low-pass filtering, and its frequency is : ; Thus, the target distance is obtained: ； Step 1.2: Speed estimation When the target has a velocity v a Doppler frequency shift is generated f D : ； Analyze multi-linear frequency modulation signals using pulse Doppler processing FFT to obtain velocity v : ； Step 1.3: Angle estimation The millimeter-wave radar uses multiple receiving antennas to form an array, and there is a phase difference in the signals at each receiving antenna m : ； Among them, d is the array antenna spacing, is the wavelength, is the direction of arrival (DOA) of the target; The MUSIC algorithm is used for angle estimation, a covariance matrix R is constructed, and eigenvalue decomposition is performed: ； Among them, is the noise subspace, is the array manifold vector, is the direction-of-arrival spectrum function; by performing peak search to find , the target angle information is obtained.

7. The implementation method of a machine learning-assisted intelligent network for joint inspection of highway multi-mode terminals according to claim 4, characterized in that, In Step 2, based on the machine learning algorithm, vehicle features, traffic flow changes, and traffic anomalies are identified, and error data correction is specifically included in the following steps: Step 2.1: Input data preprocessing Given input data , where N is the number of samples, d is the feature dimension of each sample; First, perform normalization. The input data after normalization is : ； Among them, is the mean value, is the i th input data, is the standard deviation; Step 2.2: Neural network feature extraction It is set to use a Multi-Layer Perceptron (MLP) or Convolutional Neural Network (CNN) to extract features. The forward propagation of the neural network is expressed as: Step 2.2.1: Feature extraction of the fully connected layer The calculation of each layer is as follows: ； Among them, is the feature of the l th layer, is the feature of the l th layer, is the weight matrix, d l is the matrix dimension, is the bias vector, is the non-linear activation function, ; Finally, in the feature extraction layer Bottleneck Layer of the L-th layer, the extracted high-dimensional feature vector is expressed as: ； Step 2.2.2: Feature extraction of the Convolutional Neural Network (CNN) For image or time series data, CNN is used for feature extraction. The calculation formula of the convolutional layer is: ； Among them, represents a convolution operation; After the features pass through Pooling, the information with the highest local weight is extracted, and finally, it is converted into a feature vector in the fully connected layer; Step 2.3: Feature dimensionality reduction If the feature dimension is too high, Principal Component Analysis (PCA) or Autoencoder is used for dimensionality reduction; Step 2.3.1: PCA dimensionality reduction Construct a covariance matrix: ； Solve the eigenvalue decomposition: ； Among them is the eigenvalue, is the eigenvector. Select the eigenvectors corresponding to the first k largest eigenvalues . The final feature Z is: ； Step 2.3.2: Autoencoder dimensionality reduction Autoencoder uses a neural network to encode and decode the input, and the final feature Z is expressed as: ； Among them, is the encoder weight matrix, is the encoder bias; Step 2.4: Feature fusion and classification Extracted features Z For classification, clustering, or detection tasks; if a neural network classifier is used, the calculation result of the final output layer is as follows: ； and are classifier parameters, and Softmax calculates class probabilities: ； wherein, is the true class label; finally, optimize using cross-entropy loss: ； Among them, is the predicted class probability; The extracted feature data is sent to the machine learning network for training, and the trained machine learning network is used for multi-modal recognition.

8. The implementation method of a machine learning-assisted intelligent network for joint inspection of highway multi-mode terminals according to claim 4, characterized in that In Step 3, the inspection strategy is adaptively adjusted, including adjusting the flight trajectory of the drone, the orientation of the camera, and the radar scanning angle; the recognition results based on which the regulation instructions are generated include vehicle position, vehicle density, vehicle size and weight, and the recognition of traffic jams and vehicle collisions.

Images