Highway key information intelligent extraction system based on multi-modal deep learning

Abstract

The invention discloses a highway key information intelligent extraction system based on multi-modal deep learning, which relates to the technical field of highway monitoring and comprises a cloud computing terminal, a data acquisition module, an edge computing module, an information extraction module and an early warning module. Multi-modal data of a plurality of road sections are acquired through a data acquisition module, and task load calculation and data preprocessing are performed on the multi-modal data of each road section through an edge calculation module so as to decide to perform cloud processing or local processing on the multi-modal data. The information extraction module extracts information of the multi-modal data of each road section after data preprocessing to obtain road key information of each road section, the road key information is encrypted and then transmitted to the early warning module, and the early warning module analyzes the data according to the road key information of each road section and sends the analyzed data to the early warning module. And the abnormal road section is positioned.

Description

Technical Field

[0001] The present invention relates to the technical field of highway monitoring, and in particular to a highway key information intelligent extraction system based on multimodal deep learning. Background Art

[0002] At present, the extraction of key highway information in highway operation and maintenance management mainly relies on manual inspection and traditional monitoring equipment. These methods have the following limitations:

[0003] 1. Manual inspections are inefficient and limited by personnel experience and subjective judgment, which can easily lead to missed inspections and misjudgments.

[0004] 2. Traditional monitoring equipment can usually only provide video monitoring images, lacks the ability to deeply analyze and process data, and cannot automatically identify key information on the road, such as road damage, traffic congestion, accidents, etc.

[0005] 3. With the continuous expansion of the highway network, the amount of collected data has increased dramatically, and traditional data processing methods are difficult to meet the requirements of real-time and high efficiency.

[0006] 4. The existing monitoring system lacks effective data encryption and security protection, which easily leads to information leakage.

[0007] How to overcome the limitations of traditional highway operation and maintenance management methods in extracting key highway information is currently a top priority. Summary of the invention

[0008] In order to solve the above problems, the purpose of the present invention is to provide an intelligent highway key information extraction system based on multimodal deep learning.

[0009] The purpose of the present invention can be achieved by the following technical solutions: A highway key information intelligent extraction system based on multimodal deep learning, including a cloud computing terminal, the cloud computing terminal is communicatively connected with a data acquisition module, an edge computing module, an information extraction module and an early warning module;

[0010] The data acquisition module is used to arrange different types of data acquisition equipment in a plurality of highway sections, thereby acquiring multimodal data corresponding to each highway section;

[0011] The edge computing module is provided with a number of computing nodes, each of which is used to perform task volume calculation and data preprocessing on the multimodal data of a highway section, and upload the multimodal data of the highway section whose task volume is greater than the edge task volume threshold to the cloud computing terminal for cloud processing, and perform local processing on other multimodal data;

[0012] The information extraction module includes an extraction unit, an encryption unit and a transmission unit;

[0013] The extraction unit extracts information from the multimodal data after preprocessing of each highway section data to obtain key highway information of each highway section;

[0014] The encryption unit encrypts the key highway information of each highway section;

[0015] The transmission unit transmits the encrypted key highway information to the early warning module;

[0016] The early warning module performs data analysis based on key highway information of each highway section to locate highway sections with abnormalities.

[0017] Furthermore, different types of data collection equipment are arranged in each of several highway sections, and the process of collecting multimodal data corresponding to each highway section includes:

[0018] Divide the highway into several highway sections, each highway section is associated with a section number, obtain the GPS location information of the part of the highway corresponding to each highway section, and combine the section number and GPS location information of each highway section as its identity certificate;

[0019] Several equipment points are set up at equal distances on each highway section, and different types of data collection equipment are arranged at the equipment points. The data collection equipment includes panoramic cameras, geomagnetic sensors, radar devices, and weighing sensors;

[0020] Collect live traffic video sets of each highway area through panoramic cameras;

[0021] The traffic volume of each highway section is collected through geomagnetic sensors;

[0022] The radar device includes laser radar and sonic radar. The laser radar collects the vehicle's speed, and the sonic radar identifies the internal damage status of the road.

[0023] The weighing status of the highway is monitored by a weighing sensor;

[0024] All data collected by different data collection devices in each highway section are integrated as multimodal data corresponding to the corresponding highway section.

[0025] Furthermore, a plurality of computing nodes are provided, and each computing node is used to calculate the workload of multimodal data of a highway section and perform data preprocessing, and the process includes:

[0026] A number of computing nodes are numbered and recorded as i, i = 1, 2, 3, ..., n, where n is a natural number greater than 0. The initial state of the computing node is an idle state. When the computing node receives multimodal data of a highway section to calculate the task volume, the computing node is changed to an occupied state, and the computing node splits the multimodal data it receives into single modal data of different modal types;

[0027] Different modal types include picture type, video type, and text type;

[0028] Set the task volume indicators for computing nodes to analyze multimodal data. The task volume indicators include data byte length, data complexity, processing time, and required computing resources. Set the scoring weights corresponding to different task volume indicators. The scoring weights of data byte length, data complexity, processing time, and required computing resources are recorded as α, β, γ, and δ respectively.

[0029] Among them, α, β, γ and δ are all greater than 0;

[0030] The computing node obtains the task volume of the multimodal data calculated by itself, task volume = data byte length × α + data complexity × β + processing time × γ + required computing resources × δ;

[0031] Data preprocessing of multimodal data includes data cleaning, data standardization, and data compression.

[0032] Furthermore, the multimodal data of the highway section with a task volume greater than the edge task volume threshold is uploaded to the cloud computing terminal for cloud processing, and the process of local processing of other multimodal data includes:

[0033] Set the edge task volume threshold, denoted as RWL 边缘 ;

[0034] The task volume of the highway section corresponding to the multimodal data is denoted as RWL<ID,GPS> ;

[0035] Wherein, ID is the section number of each road section, and GPS is the GPS location information of each road section;

[0036] When RWL<ID,GPS> >RWL 边缘 When the multimodal data of the corresponding highway section is locally processed, the multimodal data of the corresponding highway section is decomposed into a plurality of data segments to be extracted through the local processing;

[0037] When RWL<ID,GPS> ≤RWL 边缘When the multimodal data of the corresponding highway section is uploaded to the cloud computing terminal for cloud processing; through cloud processing, the multimodal data of the corresponding highway section is decomposed into a number of data segments to be extracted.

[0038] Furthermore, the process of extracting information from the multimodal data after preprocessing of each highway section data and obtaining key highway information of each highway section includes:

[0039] In the extraction unit, a data mining model is constructed by deep learning technology to extract information from the multimodal data of each highway section. When the data mining model extracts information from the multimodal data composed of several data segments to be extracted, the model computing power configured for the data mining model is computing power SL1. When the data mining model extracts information from the multimodal data composed of several data segments to be extracted, the model computing power configured for the data mining model is computing power SL2.

[0040] Among them, 0<SL1<SL2;

[0041] When the model computing power configuration of the data mining model is completed, the data mining model extracts the key highway information corresponding to each highway section. The key highway information includes pedestrian traffic event information, vehicle traffic event information and highway status information.

[0042] Furthermore, the process of encrypting the key highway information of each highway section includes:

[0043] Set the data encryption period of the encryption unit and record it as T 加密 , T 加密 =[t1, t2], t1 represents the starting time point of the data encryption period, and t2 represents the ending time point of the data encryption period;

[0044] At the start time of the data encryption period, the key highway information of each highway section is divided into a head information segment, a middle information segment and a tail information segment, and symmetric encryption, asymmetric encryption and hybrid encryption are used respectively;

[0045] Then generate encryption keys for the header information segment, the middle information segment and the tail information segment respectively, configure decryption keys for the header information segment, the middle information segment and the tail information segment respectively, integrate the encryption keys of the header information segment, the middle information segment and the tail information segment, and then generate an encryption key set; integrate the decryption keys of the header information segment, the middle information segment and the tail information segment, and then generate a decryption key set.

[0046] Furthermore, the process of transmitting the encrypted key highway information to the early warning module includes:

[0047] At the end time point of the data encryption period, the encryption of the key highway information is completed, and the key highway information is pushed to the transmission unit, the transmission unit receives the encrypted key highway information, establishes a data communication channel between the transmission unit and the early warning module, and transmits the encrypted key highway information to the early warning module through the data communication channel;

[0048] When encrypted key highway information is transmitted in the data communication channel, the channel environment and transmission parameters of the data communication channel are monitored. When there is an abnormality in the channel environment or the transmission parameters do not meet the preset parameter range, a corresponding early warning prompt is generated for the data communication channel, and the relevant channel maintenance personnel adjust the data communication channel; otherwise, no operation is performed.

[0049] Furthermore, the process of performing data analysis based on the key highway information of each highway section and locating the highway section with abnormalities includes:

[0050] The early warning module receives the encrypted highway key information of each highway section, decrypts the encrypted highway key information through the decryption key set, and splits the decrypted highway key information into pedestrian traffic event information, vehicle traffic event information and highway status information;

[0051] Set the judgment condition sets for pedestrian traffic incident information, vehicle traffic incident information and highway status information respectively, and perform data analysis on key highway information of the highway section through the judgment condition sets, and the judgment condition sets are used to judge whether there are abnormalities in the pedestrian traffic accident information, vehicle traffic incident information and highway status information respectively;

[0052] If any of the pedestrian traffic event information, vehicle traffic event information, and highway status information is abnormal, the corresponding highway section is judged to have an abnormality; otherwise, the corresponding highway section is judged to have no abnormality, and then several highway sections with abnormalities are located, and the relevant road administration personnel are notified to handle the abnormal highway sections.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. By arranging different types of data acquisition equipment in multiple highway sections, the system can collect comprehensive multimodal data and perform preliminary data preprocessing through the edge computing module, effectively reducing the processing burden of the cloud computing terminal and improving data processing efficiency; the introduction of the edge computing module enables the system to flexibly choose to process the data at the computing node of the edge computing module or upload it to the cloud computing terminal for processing according to the size of the data task, thereby optimizing the use of computing resources, reducing latency, and improving the response speed of the system.

[0055] 2. The information extraction module uses deep learning technology to analyze the multimodal data after data preprocessing, which can automatically identify key highway information and improve the accuracy and automation of information extraction; the encryption unit and the transmission unit complete the encryption processing of key highway information, and ensure the security of the encrypted multimodal data during the transmission process, thereby preventing the risk of information leakage; the early warning module can quickly locate abnormal highway sections based on the analyzed key highway information, and take early warning, providing strong support for highway safety management. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of the present invention. DETAILED DESCRIPTION

[0057] like Figure 1 As shown, a highway key information intelligent extraction system based on multimodal deep learning includes a cloud computing terminal, and the cloud computing terminal is communicatively connected with a data acquisition module, an edge computing module, an information extraction module and an early warning module;

[0058] The data acquisition module is used to arrange different types of data acquisition equipment in a plurality of highway sections, thereby acquiring multimodal data corresponding to each highway section;

[0059] The edge computing module is provided with a number of computing nodes, each of which is used to perform task volume calculation and data preprocessing on the multimodal data of a highway section, and upload the multimodal data of the highway section whose task volume is greater than the edge task volume threshold to the cloud computing terminal for cloud processing, and perform local processing on other multimodal data;

[0060] The information extraction module includes an extraction unit, an encryption unit and a transmission unit;

[0061] The extraction unit extracts information from the multimodal data after preprocessing of each highway section data to obtain key highway information of each highway section;

[0062] The encryption unit encrypts the key highway information of each highway section;

[0063] The transmission unit transmits the encrypted key highway information to the early warning module;

[0064] The early warning module performs data analysis based on key highway information of each highway section to locate highway sections with abnormalities.

[0065] It should be further explained that, in the specific implementation process, different types of data collection equipment are arranged in several highway sections respectively, and the process of collecting multimodal data corresponding to each highway section includes:

[0066] Divide the highway into several highway sections. Each highway section is associated with a unique section number consisting of digits from 0 to 9. Obtain the GPS location information of the portion of the highway corresponding to each highway section. The GPS location information includes the starting GPS coordinates and the ending GPS coordinates of the portion of the highway corresponding to the highway section.

[0067] The section number and GPS location information of each highway section are combined as its identity certificate;

[0068] Several equipment points are set up at equal distances on each highway section, and different types of data collection equipment are arranged at the equipment points. The different types of data collection equipment include panoramic cameras, geomagnetic sensors, radar devices, and weighing sensors;

[0069] The live traffic video between two equipment points on each highway section is collected by a panoramic camera, and the live traffic video between all equipment points on each highway section is integrated as the live traffic video set of the corresponding highway section;

[0070] The presence of vehicles on the corresponding road of each road section is collected by geomagnetic sensors to obtain the traffic volume;

[0071] Radar devices include laser radar and sonic radar;

[0072] Collect the vehicle's speed through LiDAR;

[0073] Identify the internal damage status of the road through acoustic radar;

[0074] The weighing status of the highway is monitored by a weighing sensor;

[0075] All data collected by different data collection devices in each highway section are integrated as multimodal data corresponding to the corresponding highway section, thereby completing the collection of multimodal data corresponding to each highway section.

[0076] It should be further explained that, in the specific implementation process, a number of computing nodes are set up, and each computing node is used to calculate the task volume and preprocess the data of the multimodal data of a highway section. The process includes:

[0077] A number of computing nodes set in the edge computing module are numbered and recorded as i, i=1, 2, 3, ..., n, where n is a natural number greater than 0, that is, the number of computing nodes in the edge computing module are: computing node 1, computing node 2, ..., computing node n;

[0078] The initial state of the computing node is an idle state;

[0079] When the computing node receives multimodal data of a highway section for task volume calculation, the computing node is changed to an occupied state, and the computing node splits the multimodal data it receives into single modal data of different modal types;

[0080] Different modal types include picture type, video type, and text type;

[0081] Set the task load indicators for computing nodes to analyze multimodal data. The task load indicators include data byte length, data complexity, processing time, and required computing resources.

[0082] Set the scoring weights corresponding to different task volume indicators, where the scoring weights of data byte length, data complexity, processing time, and required computing resources are denoted as α, β, γ, and δ respectively;

[0083] Among them, α, β, γ and δ are all greater than 0;

[0084] Then the computing node obtains the task volume of the multimodal data calculated by itself. The expression formula of the task volume is as follows:

[0085] Task volume = data byte length × α + data complexity × β + processing time × γ + required computing resources × δ;

[0086] Data preprocessing of multimodal data includes data cleaning, data standardization, and data compression;

[0087] Data cleaning is used to initially remove invalid, erroneous or redundant data from multimodal data;

[0088] Convert multimodal data into a unified data format through data standardization;

[0089] Data compression is used to shorten the length of data bytes corresponding to multimodal data, ensuring that multimodal data can be transmitted faster.

[0090] It should be further explained that, in the specific implementation process, the multimodal data of the highway section with a task volume greater than the edge task volume threshold is uploaded to the cloud computing terminal for cloud processing, and the process of local processing of other multimodal data includes:

[0091] Set the edge task volume threshold and record it as RWL 边缘 ;

[0092] The task volume of the highway section corresponding to the multimodal data is denoted as RWL<ID,GPS> ;

[0093] Wherein, ID is the section number corresponding to each road section, and GPS is the GPS location information corresponding to each road section;

[0094] Compare the task volume of the multimodal data corresponding to each highway section with the value of the edge task volume threshold, and perform different operations according to the comparison results; the details are as follows:

[0095] When RWL<ID,GPS> >RWL 边缘 When the multimodal data of the corresponding highway section is locally processed; the multimodal data of the corresponding highway section is decomposed into a plurality of data segments to be extracted through local processing;

[0096] When RWL<ID,GPS> ≤RWL 边缘 When the multimodal data of all corresponding highway sections are uploaded to the cloud computing terminal, the cloud computing terminal performs cloud processing on all received multimodal data; through cloud processing, the multimodal data of the corresponding highway section is disassembled into a number of data segments to be extracted.

[0097] It should be further explained that, in the specific implementation process, the process of extracting information from the multimodal data after preprocessing the data of each highway section and obtaining the key highway information of each highway section includes:

[0098] In the extraction unit, a data mining model is constructed by deep learning technology, and the data mining model is used to extract information from the multimodal data of each highway section. When the data mining model extracts information from the multimodal data composed of several first data segments to be extracted, the model computing power configured for the data mining model is computing power SL1, and when the data mining model extracts information from the multimodal data composed of several second data segments to be extracted, the model computing power configured for the data mining model is computing power SL2;

[0099] 0<SL1<SL2;

[0100] When the model computing power configuration of the data mining model is completed, the data mining model extracts the key highway information corresponding to each highway section, and the key highway information includes pedestrian traffic event information, vehicle traffic event information and highway status information;

[0101] The pedestrian traffic event information is used to describe the traffic events of pedestrians on the corresponding roads in different road sections, and the traffic events specifically include pedestrian walking routes, pedestrian crossing behaviors, and pedestrian-vehicle interaction behaviors;

[0102] The driving traffic event information is used to describe the traffic events on the corresponding highways in different highway sections, and the traffic events specifically include the driving path, lane change behavior, following distance and speed change behavior;

[0103] The highway status information is used to describe the status of the highway itself corresponding to different highway sections, and is used to describe the driving status on the highway corresponding to different highway sections. The specific status of the highway itself includes the internal damage status of the highway and the external damage status of the highway; the driving status on the highway includes the congested driving status, the unimpeded driving status and the normal driving status;

[0104] When the number of vehicles on the highway is greater than the traffic congestion threshold set for the highway, the current highway traffic state is judged to be a traffic congestion state; when the number of vehicles on the highway is less than the traffic unblocking threshold set for the highway, the current highway traffic state is judged to be a traffic unblocking state;

[0105] When the number of vehicles on the highway is greater than or equal to the smooth traffic threshold set for the highway, and less than or equal to the traffic congestion threshold set for the highway, it is determined that the current driving state of the highway is a normal driving state.

[0106] It should be noted that pedestrian traffic event information is obtained through deep learning and data analysis of the real-time traffic video set collected by the panoramic camera, vehicle traffic event information is obtained through analysis of the real-time traffic video set and the data collected by the lidar, and highway status information is obtained through analysis of the data collected by the panoramic camera, sonic radar and weighing sensor.

[0107] It should be further explained that, in the specific implementation process, the process of encrypting the key highway information of each highway section and transmitting the encrypted key highway information to the early warning module includes:

[0108] Set the data encryption period of the encryption unit and record the data encryption period as T 加密 ;

[0109] Among them, T 加密 =[t1, t2], t1 represents the starting time point of the data encryption period, and t2 represents the ending time point of the data encryption period;

[0110] At the start time of the data encryption period, the key highway information of each highway section is divided into a head information segment, a middle information segment and a tail information segment, and symmetric encryption, asymmetric encryption and mixed encryption are respectively used for the head information segment, the middle information segment and the tail information segment;

[0111] Then generate encryption keys for the header information segment, the middle information segment, and the tail information segment respectively;

[0112] Configure the decryption keys for the header information segment, the middle information segment, and the tail information segment;

[0113] The encryption keys of the header information segment, the middle information segment and the tail information segment are integrated to generate an encryption key set; the decryption keys of the header information segment, the middle information segment and the tail information segment are integrated to generate a decryption key set.

[0114] At the end time point of the data encryption period, the encryption of the key highway information is completed and the key highway information is pushed to the transmission unit;

[0115] The transmission unit receives the encrypted key highway information, establishes a data communication channel between the transmission unit and the early warning module, and transmits the encrypted key highway information to the early warning module through the data communication channel;

[0116] When encrypted key highway information is transmitted in the data communication channel, the channel environment and transmission parameters of the data communication channel are monitored. When there is an abnormality in the channel environment or the transmission parameters do not meet the preset parameter range, a corresponding early warning prompt is generated for the data communication channel, and the relevant channel maintenance personnel adjust the data communication channel; otherwise, no operation is performed.

[0117] It should be further explained that, in the specific implementation process, the process of performing data analysis based on the key highway information of each highway section and locating the highway section with abnormalities includes:

[0118] The early warning module receives the encrypted highway key information of each highway section, decrypts the encrypted highway key information through the decryption key set, and splits the decrypted highway key information into pedestrian traffic event information, vehicle traffic event information and highway status information;

[0119] Set the judgment condition sets for pedestrian traffic incident information, vehicle traffic incident information and highway status information respectively, and perform data analysis on key highway information of the highway section through the judgment condition sets, and the judgment condition sets are used to judge whether there are abnormalities in the pedestrian traffic accident information, vehicle traffic incident information and highway status information respectively;

[0120] If any of the pedestrian traffic event information, vehicle traffic event information, and highway status information is abnormal, the corresponding highway section is judged to have an abnormality; otherwise, the corresponding highway section is judged to have no abnormality, and then several highway sections with abnormalities are located, and the relevant road administration personnel are notified to handle the abnormal highway sections.

[0121] It should be noted that the judgment condition sets of pedestrian traffic event information, vehicle traffic event information and highway status information are as follows:

[0122] The judgment condition set of pedestrian traffic event information includes the walkable area corresponding to the pedestrian walking route, the street crossing behavior norms corresponding to the pedestrian crossing behavior, and the interactive behavior norms of the pedestrian and vehicle interaction behavior; if the pedestrian walking route is not in the walkable area, or the pedestrian crossing behavior does not comply with the street crossing behavior norms, or the pedestrian and vehicle interaction behavior does not comply with the interactive behavior norms, then it is judged that the pedestrian traffic event information is abnormal;

[0123] The judgment condition set of the traffic event information includes the drivable area corresponding to the driving path, the lane change rule corresponding to the lane change behavior, the safety distance range corresponding to the following distance, and the speed change rule corresponding to the speed change behavior;

[0124] If the driving path of the vehicle is not in the drivable area, or the lane change behavior of the vehicle violates the lane change rules, or the following distance of the vehicle is not within the safe distance range, or the speed change behavior of the vehicle violates the speed change rules, then it is judged that the traffic event information is abnormal;

[0125] The judgment condition set of the highway status information includes the upper limit of the internal damage degree corresponding to the internal damage state of the highway, and the upper limit of the external damage degree corresponding to the external damage state of the highway. If the damage degree corresponding to the internal damage state of the highway exceeds the upper limit of the internal damage degree, or the damage degree corresponding to the external damage state of the highway exceeds the upper limit of the external damage degree, or the driving state of the highway is in a congested state, then it is judged that the highway status information is abnormal.

[0126] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A highway key information intelligent extraction system based on multimodal deep learning, including a cloud computing terminal, characterized in that: The cloud computing terminal is communicatively connected with a data acquisition module, an edge computing module, an information extraction module and an early warning module; The data acquisition module is used to arrange different types of data acquisition equipment in each of several highway sections, thereby collecting multimodal data corresponding to each highway section; The edge computing module is provided with a number of computing nodes, each of which is used to perform task volume calculation and data preprocessing on the multimodal data of a highway section, and upload the multimodal data of the highway section whose task volume is greater than the edge task volume threshold to the cloud computing terminal for cloud processing, and perform local processing on other multimodal data; The information extraction module includes an extraction unit, an encryption unit and a transmission unit; The extraction unit extracts information from the multimodal data after preprocessing of each highway section data to obtain key highway information of each highway section; The encryption unit encrypts the key highway information of each highway section; The transmission unit transmits the encrypted key highway information to the early warning module; The early warning module performs data analysis based on key highway information of each highway section to locate highway sections with abnormalities.

2. According to claim 1, a highway key information intelligent extraction system based on multimodal deep learning is characterized in that: The process of arranging different types of data collection equipment in several highway sections and then collecting multimodal data corresponding to each highway section includes: Divide the highway into several highway sections, each highway section is associated with a section number, obtain the GPS location information of the part of the highway corresponding to each highway section, and combine the section number and GPS location information of each highway section as its identity certificate; Several equipment points are set up at equal distances on each highway section, and different types of data collection equipment are arranged at the equipment points. The data collection equipment includes panoramic cameras, geomagnetic sensors, radar devices, and weighing sensors; Collect live traffic video sets of each highway area through panoramic cameras; The traffic volume of each highway section is collected through geomagnetic sensors; The radar device includes laser radar and sonic radar. The laser radar collects the vehicle's speed, and the sonic radar identifies the internal damage status of the road. The weighing status of the highway is monitored by a weighing sensor; All data collected by different data collection devices in each highway section are integrated as multimodal data corresponding to the corresponding highway section.

3. According to claim 2, a highway key information intelligent extraction system based on multimodal deep learning is characterized in that: Several computing nodes are set up, and each computing node is used to calculate the task volume and preprocess the data of a highway section. The process includes: A number of computing nodes are numbered and recorded as i, i = 1, 2, 3, ..., n, where n is a natural number greater than 0. The initial state of the computing node is an idle state. When the computing node receives multimodal data of a highway section to calculate the task volume, the computing node is changed to an occupied state, and the computing node splits the multimodal data it receives into single modal data of different modal types; Different modal types include picture type, video type, and text type; Set the task volume indicators for computing nodes to analyze multimodal data. The task volume indicators include data byte length, data complexity, processing time, and required computing resources. Set the scoring weights corresponding to different task volume indicators. The scoring weights of data byte length, data complexity, processing time, and required computing resources are recorded as α, β, γ, and δ respectively. Among them, α, β, γ and δ are all greater than 0; The computing node obtains the task volume of the multimodal data calculated by itself, task volume = data byte length × α + data complexity × β + processing time × γ + required computing resources × δ; Data preprocessing of multimodal data includes data cleaning, data standardization, and data compression.

4. According to claim 3, a highway key information intelligent extraction system based on multimodal deep learning is characterized in that: The multimodal data of the highway section with a task volume greater than the edge task volume threshold is uploaded to the cloud computing terminal for cloud processing. The process of local processing of other multimodal data includes: Set the edge task volume threshold, denoted as RWL 边缘 ; The task volume of the highway section corresponding to the multimodal data is denoted as RWL<ID,GPS> ; Wherein, ID is the section number of each road section, and GPS is the GPS location information of each road section; When RWL<ID,GPS> >RWL 边缘 When the multimodal data of the corresponding highway section is locally processed, the multimodal data of the corresponding highway section is decomposed into a plurality of data segments to be extracted through the local processing; When RWL<ID,GPS> ≤RWL 边缘 When the multimodal data of the corresponding highway section is uploaded to the cloud computing terminal for cloud processing; through cloud processing, the multimodal data of the corresponding highway section is decomposed into a number of data segments to be extracted.

5. According to claim 4, a highway key information intelligent extraction system based on multimodal deep learning is characterized in that: The process of extracting information from the multimodal data after preprocessing of each highway section data and obtaining key highway information of each highway section includes: In the extraction unit, a data mining model is constructed by deep learning technology to extract information from the multimodal data of each highway section. When the data mining model extracts information from the multimodal data composed of several data segments to be extracted, the model computing power configured for the data mining model is computing power SL1. When the data mining model extracts information from the multimodal data composed of several data segments to be extracted, the model computing power configured for the data mining model is computing power SL2. Among them, 0<SL1<SL2; When the model computing power configuration of the data mining model is completed, the data mining model extracts the key highway information corresponding to each highway section. The key highway information includes pedestrian traffic event information, vehicle traffic event information and highway status information.

6. The intelligent highway key information extraction system based on multimodal deep learning according to claim 5 is characterized in that: The process of encrypting key highway information for each highway section includes: Set the data encryption period of the encryption unit and record it as T 加密 , T 加密 =[t1, t2], t1 represents the starting time point of the data encryption period, and t2 represents the ending time point of the data encryption period; At the start time of the data encryption period, the key highway information of each highway section is divided into a head information segment, a middle information segment and a tail information segment, and symmetric encryption, asymmetric encryption and hybrid encryption are used respectively; Then generate encryption keys for the header information segment, the middle information segment and the tail information segment respectively, configure decryption keys for the header information segment, the middle information segment and the tail information segment respectively, integrate the encryption keys of the header information segment, the middle information segment and the tail information segment, and then generate an encryption key set; integrate the decryption keys of the header information segment, the middle information segment and the tail information segment, and then generate a decryption key set.

7. The intelligent highway key information extraction system based on multimodal deep learning according to claim 6 is characterized in that: The process of transmitting encrypted key highway information to the early warning module includes: At the end time point of the data encryption period, the encryption of the key highway information is completed, and the key highway information is pushed to the transmission unit, the transmission unit receives the encrypted key highway information, establishes a data communication channel between the transmission unit and the early warning module, and transmits the encrypted key highway information to the early warning module through the data communication channel; When encrypted key highway information is transmitted in the data communication channel, the channel environment and transmission parameters of the data communication channel are monitored. When there is an abnormality in the channel environment or the transmission parameters do not meet the preset parameter range, a corresponding early warning prompt is generated for the data communication channel, and the relevant channel maintenance personnel adjust the data communication channel; otherwise, no operation is performed.

8. The intelligent highway key information extraction system based on multimodal deep learning according to claim 7 is characterized in that: The process of analyzing the key information of each highway section and locating the abnormal highway section includes: The early warning module receives the encrypted highway key information of each highway section, decrypts the encrypted highway key information through the decryption key set, and splits the decrypted highway key information into pedestrian traffic event information, vehicle traffic event information and highway status information; Set the judgment condition sets for pedestrian traffic incident information, vehicle traffic incident information and highway status information respectively, and perform data analysis on key highway information of the highway section through the judgment condition sets, and the judgment condition sets are used to judge whether there are abnormalities in the pedestrian traffic accident information, vehicle traffic incident information and highway status information respectively; If any of the pedestrian traffic event information, vehicle traffic event information, and highway status information is abnormal, the corresponding highway section is judged to have an abnormality; otherwise, the corresponding highway section is judged to have no abnormality, and then several highway sections with abnormalities are located, and the relevant road administration personnel are notified to handle the abnormal highway sections.

Images