Road data processing method, device and equipment, computer readable storage medium and computer program product

Abstract

The invention provides a road data processing method, device and equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining video data collected for an expressway, performing feature extraction and time sequence analysis on the video data to obtain time sequence data, and determining a first congestion prediction result based on the time sequence data; obtaining a plurality of information source data for the expressway, determining a first weight of each piece of information source data, and determining a first weight matrix based on each first weight; determining the membership degree of each piece of information source data under the evaluation level, and determining a first evaluation matrix based on each membership degree; and based on the first weight matrix and the first evaluation matrix, determining a second congestion prediction result, and based on the first congestion prediction result and the second congestion prediction result, determining an opening processing decision of the emergency lane in the expressway. According to the invention, the accuracy of emergency lane opening decision can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a road data processing method, device, equipment, computer-readable storage medium and computer program product. Background Art

[0003] In the related technology, in the process of emergency lane control on highways, traffic flow data is predicted based on historical traffic flow data, the road congestion index is calculated, and the opening and closing of the emergency lane is controlled based on the traffic congestion index and the set congestion index threshold. Due to factors such as the camera angle and position, there are errors in the traffic flow and speed calculations based on target detection, which affects the accuracy of the emergency lane opening decision. In addition, the comprehensive factors such as the use of the emergency lane, the operation of the emergency vehicle, and the weather conditions at the road end are not considered, which affects the opening of the emergency lane, resulting in a low accuracy rate in the emergency lane opening decision. Summary of the invention

[0004] The embodiments of the present application provide a road data processing method, device, equipment, computer-readable storage medium and computer program product, which can improve the accuracy of emergency lane opening decisions.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a road data processing method, the method comprising:

[0007] Acquire video data collected from the expressway, perform feature extraction and time series analysis on the video data to obtain time series data, and determine a first congestion prediction result based on the time series data;

[0008] Acquire a plurality of information source data for the expressway, and respectively determine a first weight for each of the information source data, and determine a first weight matrix based on each of the first weights;

[0009] Determine the membership of each of the information source data under the evaluation level, and determine a first evaluation matrix based on each of the membership;

[0010] Based on the first weight matrix and the first evaluation matrix, a second congestion prediction result is determined, and based on the first congestion prediction result and the second congestion prediction result, an opening processing decision of the emergency lane in the expressway is determined.

[0011] The present application provides a road data processing device, including:

[0012] A first determination module is used to obtain video data collected for the expressway, perform feature extraction and time series analysis on the video data to obtain time series data, and determine a first congestion prediction result based on the time series data;

[0013] A second determination module is used to obtain a plurality of information source data for the expressway, and respectively determine a first weight of each of the information source data, and determine a first weight matrix based on each of the first weights;

[0014] A third determination module is used to determine the membership of each of the information source data under the evaluation level, and determine a first evaluation matrix based on each of the membership;

[0015] The fourth determination module is used to determine a second congestion prediction result based on the first weight matrix and the first evaluation matrix, and to determine an opening processing decision of the emergency lane in the highway based on the first congestion prediction result and the second congestion prediction result.

[0016] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0017] A memory for storing computer executable instructions or computer programs;

[0018] The processor is used to implement the road data processing method provided in the embodiment of the present application when executing the computer executable instructions or computer programs stored in the memory.

[0019] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or a computer program for implementing the road data processing method provided in the embodiment of the present application when executed by a processor.

[0020] An embodiment of the present application provides a computer program product, including computer executable instructions or a computer program. When the computer executable instructions or the computer program are executed by a processor, the road data processing method provided in the embodiment of the present application is implemented.

[0021] The embodiments of the present application have the following beneficial effects:

[0022] By using the embodiment of the present application, the video data collected for the expressway is obtained, and the video data is subjected to feature extraction and time series analysis to obtain time series data, which has time features and space features. Then, based on the time series data, the first congestion prediction result is determined, and the prediction result of the emergency lane opening decision is determined based on the spatiotemporal features of the expressway. Then, multiple information source data for the expressway are obtained, and the first weight of each information source data is determined respectively, and the first weight matrix is ​​determined based on each first weight, and then the membership of each information source data under the evaluation level is determined, and the first evaluation matrix is ​​determined based on each membership, and then the second congestion prediction result is determined based on the first weight matrix and the first evaluation matrix, and the prediction result of the emergency lane opening decision is determined comprehensively based on multiple information source data of the expressway. And based on the first congestion prediction result and the second congestion prediction result, the emergency lane opening processing decision in the expressway is determined, so that the spatiotemporal features of the expressway and the influence of multiple information source data on the emergency lane opening are fully considered, and the real-time and accuracy of the emergency lane opening decision are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of an application mode of the road data processing method provided in an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0025] Figure 3A is a first flow chart of the road data processing method provided in an embodiment of the present application;

[0026] Figure 3B is a second flow chart of the road data processing method provided in an embodiment of the present application;

[0027] Figure 3C is a third flow chart of the road data processing method provided in an embodiment of the present application;

[0028] Figure 3D is a fourth flow chart of the road data processing method provided in an embodiment of the present application;

[0029] Figure 3E is a fifth flow chart of the road data processing method provided in an embodiment of the present application;

[0030] Figure 4 It is a structural diagram of the training process of the congestion prediction model provided in an embodiment of the present application;

[0031] Figure 5 It is a structural diagram of the congestion prediction process of the congestion prediction model provided in an embodiment of the present application.

[0032] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0034] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0035] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0036] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0037] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0038] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0039] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0040] 1) Emergency lane: It is a special lane set up on the highway for vehicles to pass in emergency situations.

[0041] 2) MobileNet: It is a lightweight convolutional neural network architecture designed for mobile and embedded devices for low-latency and high-throughput applications. MobileNetV2 is an improved network based on MobileNet.

[0042] 3) Long Short-Term Memory (LSTM): It is a special recurrent neural network structure that can learn long-term dependency information and is used for sequence processing tasks such as speech recognition, language models, and time series prediction.

[0043] 4) Cross-entropy loss: It is a loss function commonly used in classification problems. It measures the difference between the predicted probability distribution and the true label probability distribution. In machine learning, cross-entropy loss is widely used to evaluate the match between model output and true label.

[0044] 5) Membership degree: It is a core concept in fuzzy logic and fuzzy mathematics. It indicates the degree to which an element belongs to a fuzzy set, and its value range is between 0 and 1.

[0045] 6) Fuzzy set theory: A mathematical tool for dealing with uncertainty and ambiguity, which expands the concept of traditional set theory and allows elements to exist in a set with different degrees of belonging.

[0046] In order to better understand the road data processing method provided in the embodiment of the present application, the data processing method applied to the emergency lane of the highway in the related art and the existing shortcomings are first explained.

[0047] The decision to open the emergency lane on the highway is an important technical problem that needs to be solved in the work of intelligent expansion of highways. At present, the main technical idea of ​​the decision to open the emergency lane is to predict the traffic flow data based on the historical traffic flow data, calculate the traffic congestion index, and control the opening and closing of the emergency lane based on the traffic congestion index and the set congestion index threshold. Specifically, the historical traffic parameter data is brought into a specific traffic flow prediction model to predict the traffic status after a period of time. Combined with the prediction results, the traffic congestion index is calculated, and the obtained traffic congestion index is compared with the set congestion index threshold. When the traffic congestion index is greater than the set congestion index threshold, the emergency lane is opened.

[0048] The related technologies have the following problems in the emergency lane control process:

[0049] (1) Due to factors such as the camera angle and position, there are errors in the traffic flow and speed calculation based on target detection, which affects the accuracy of the emergency lane opening decision. In addition, it involves multi-step data processing and cannot meet the real-time requirements of the emergency lane opening decision.

[0050] (2) The impact of comprehensive factors such as emergency lane usage, emergency vehicle operation, and roadside weather conditions on the opening of emergency lanes was not considered, resulting in low accuracy in emergency lane opening decisions.

[0051] The embodiments of the present application provide a road data processing method, device, equipment, computer-readable storage medium and computer program product, which can improve the accuracy of emergency lane opening decisions.

[0052] The following describes an exemplary application of the electronic device provided by the embodiment of the present application. The electronic device provided by the embodiment of the present application can be implemented as various types of terminals such as a laptop computer, a tablet computer, a desktop computer, a set-top box, a smart phone, a smart speaker, a smart watch, a smart TV, and a vehicle-mounted terminal, and can also be implemented as a server. The following describes an exemplary application when the device is implemented as a server.

[0053] See also Figure 1 , Figure 1 This is a schematic diagram of an application mode of the road data processing method provided in an embodiment of the present application, for example, Figure 1 The server 200, the network 300 and the terminal 400 are involved. The terminal 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0054] In the process of road data processing, the terminal 400 receives the road data processing instruction, and in response to the road data processing instruction, sends the road data processing request to the server 200. In response to the road data processing request, the server 200 obtains the video data collected for the expressway, obtains multiple information source data for the expressway, performs feature extraction and time series analysis on the video data, obtains time series data, and determines the first congestion prediction result based on the time series data, and determines the first weight of each information source data respectively, and determines the first weight matrix based on each first weight; determines the membership of each information source data under the evaluation level, and determines the first evaluation matrix based on each membership; determines the second congestion prediction result based on the first weight matrix and the first evaluation matrix, and determines the opening processing decision of the emergency lane in the expressway based on the first congestion prediction result and the second congestion prediction result, and returns the road data processing result to the terminal 400. The terminal 400 receives the road data processing result and displays it.

[0055] In some embodiments, the server (e.g., server 200) may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. Terminal 400 may be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, vehicle-mounted terminal, aircraft, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0056] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, the electronic device may be a terminal or a server, Figure 2 The electronic device 500 shown includes: at least one processor 410, a memory 450, and at least one network interface 420. The various components in the electronic device 500 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0057] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0058] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0059] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0060] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0061] The operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic businesses and process hardware-based tasks.

[0062] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include: Bluetooth, Wireless Authentication (WiFi), and Universal Serial Bus (USB).

[0063] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 A road data processing device 455 stored in a memory 450 is shown, which may be software in the form of a program or plug-in, including the following software modules: a first determination module 4551, a second determination module 4552, a third determination module 4553, and a fourth determination module 4554. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0064] In some embodiments, the terminal or server can implement the road data processing method provided in the embodiments of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be microprogram-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a native application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as a cloud computing APP or an instant messaging APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0065] The road data processing method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server device provided in the embodiment of the present application.

[0066] Next, the road data processing method provided by the embodiment of the present application is described. As mentioned above, the electronic device that implements the road data processing method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Next, the road data processing method provided by the embodiment of the present application is described by taking the electronic device as a server as an example. Figure 3A , Figure 3A is a first flow chart of the road data processing method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0067] In step 301, video data collected from a highway is obtained, feature extraction and time series analysis are performed on the video data to obtain time series data, and a first congestion prediction result is determined based on the time series data.

[0068] Here, the first congestion prediction result is a prediction result of the decision on the opening of the emergency lane in the highway based on the spatial and temporal characteristics of the highway. Video data from the highway monitoring camera is obtained, and based on the camera point topology structure, the video data is divided into different data groups, one camera point data corresponds to one data group, and each data group corresponds to a different spatial position of the highway. Spatial features are extracted from the video data of different data groups to obtain a feature map with spatial features, and time series analysis is performed on the feature map to obtain time series data, and the first congestion prediction result is determined based on the time series data.

[0069] In some embodiments, the video data includes multiple image frames. The road data processing method provided in the embodiment of the present application can be implemented by using a trained congestion prediction model, see Figure 3B , Figure 3B It is a second flow chart of the road data processing method provided in an embodiment of the present application. Figure 3A In step 301 shown in the figure, "feature extraction and time series analysis of video data to obtain time series data" can be performed by Figure 3B Steps 3011 to 3013 are implemented as described in detail below.

[0070] In step 3011, the trained congestion prediction model is obtained.

[0071] Here, the congestion prediction model is an improved network based on the lightweight convolutional neural network MobileNetV2. The congestion prediction model includes the MobileNetV2 network and the long short-term memory recurrent neural network (Long Short-Term Memory, LSTM).

[0072] In step 3012, the convolutional neural network layer in the trained congestion prediction model is called to extract features from each image frame, and a corresponding feature vector of each image frame is obtained.

[0073] Here, the convolutional neural network layer refers to the MobileNetV2 network. Preprocess multiple image frames in the video data, adjust the size of each image frame to the input size required by the MobileNetv2 network, such as a size of 224*224 pixels, and normalize each image frame to ensure that the pixel values ​​of the image frame are within an appropriate range. The preprocessed image frames are input into the MobileNetv2 network. The core of the MobileNetv2 network is the depthwise separable convolution and linear bottleneck structure. The network structure of the depthwise separable convolution includes an initial standard convolution layer. The size of the standard convolution layer can be set to 3*3, followed by multiple layer-by-layer convolution layers and residual blocks. The residual blocks further optimize the feature extraction of the image frames through the linear bottleneck structure to obtain the feature map of the image frame. Remove the last fully connected layer in the MobileNetv2 network, retain only the last convolution layer, process the feature map of the image frame through the last convolution layer, and output the feature vector of each image frame. For example, the feature vector F of the image frame t =MobileNetv2(X t ), where X t Indicates the image frame input at the current time.

[0074] In step 3013, time series data is constructed based on the feature vectors of each image frame and a preset time window length.

[0075] Here, the preset time window length is the length of each time series sample in the time series data, and each time series sample includes the feature vectors of a preset number of consecutive image frames. Using the sliding window technology, the time series data is constructed based on the feature vectors of each image frame and the preset time window length. For example, the time series data is {F t , F t-1 , ..., F t-N+1}, the N in the subscript is the preset time window length.

[0076] In an embodiment of the present application, by calling a lightweight convolutional neural network MobileNetV2, feature extraction and time series analysis are performed on multiple image frames in the video data to obtain time series data, thereby reducing the computational complexity while maintaining a high feature extraction capability, thereby facilitating the subsequent determination of the first congestion prediction result based on the time series data.

[0077] In some embodiments, see Figure 3C , Figure 3C This is a third flow chart of the road data processing method provided in an embodiment of the present application. Figure 3A In step 301 shown in the figure, "determine a first congestion prediction result based on the time series data", which can be achieved by Figure 3C Steps 3014 to 3017 are implemented as described in detail below.

[0078] In step 3014, the time series data is expanded to obtain updated time series data.

[0079] Here, by increasing the preset time window length, the extended time window length is obtained, and the time series data is extended based on the extended time window length to obtain updated time series data. For example, the shape of the updated time series data is (T, D), where T = N + M, which is the extended time window length, N and M represent the preset time window length and the additional extended time window length, respectively, and D is the dimension of the feature vector of the image frame.

[0080] In step 3015, the timestamp corresponding to each feature vector in the updated time series data is obtained.

[0081] The starting timestamp of each image frame is determined based on the acquisition time of each image frame, and the time interval between adjacent image frames is determined based on the sampling frequency of each image frame. The timestamp corresponding to the feature vector of each image frame is calculated based on the starting timestamp and the time interval, thereby obtaining the timestamp corresponding to each feature vector in the updated time series data.

[0082] In step 3016, the recurrent neural network layer in the trained congestion prediction model is called to predict the updated time series data and each timestamp to obtain the congestion prediction result of the first current moment, the congestion prediction result of the first future moment, and the congestion prediction result of the second future moment.

[0083] Here, the first current moment is the moment when the video data collection ends, and the first future moment is before the second future moment. The recurrent neural network layer refers to an LSTM network, and the updated time series data and each timestamp are used as the input of the LSTM network. A random deletion layer is added to the LSTM network to prevent overfitting, and a batch normalization layer is used to stabilize the training process. The LSTM network is designed as a multi-output structure, and the congestion prediction result at each moment is output through an independent fully connected layer. The activation function sigmoid is used to convert the output into the congestion prediction result of the first current moment, the congestion prediction result of the first future moment, and the congestion prediction result of the second future moment. Among them, the congestion prediction result of the first current moment is used to characterize whether the highway is congested at the first current moment. For example, the congestion prediction result of the first current moment can be 0 or 1. When the congestion prediction result of the first current moment is 0, it indicates that the highway is not congested at the first current moment. When the congestion prediction result of the first current moment is 1, it indicates that the highway is congested at the first current moment. Similarly, the congestion prediction result at the first future moment is used to characterize whether the expressway is congested at the first future moment, and the congestion prediction result at the second future moment is used to characterize whether the expressway is congested at the second future moment.

[0084] In step 3017, a first congestion prediction result is determined based on the congestion prediction result at the first current moment, the congestion prediction result at the first future moment, and the congestion prediction result at the second future moment.

[0085] Here, if the first congestion prediction result at the current moment indicates that the highway is congested at the current moment, and the congestion prediction result at the first future moment indicates that the highway is congested at the first future moment, and the congestion prediction result at the second future moment indicates that the highway is not congested at the second future moment, then the first congestion prediction result indicates that the emergency lane is open, and the value of the first congestion prediction result is determined to be 1. In other cases, the first congestion prediction result indicates that the emergency lane is not open, and the value of the first congestion prediction result is determined to be 0.

[0086] In an embodiment of the present application, by performing expansion processing on the time series data to obtain updated time series data, calling the recurrent neural network layer in the trained congestion prediction model, predicting the updated time series data and the timestamps corresponding to each feature vector, and obtaining the congestion prediction result of the first current moment, the congestion prediction result of the first future moment, and the congestion prediction result of the second future moment, the traffic status changes within a longer time span can be captured, thereby improving the accuracy of the congestion prediction results.

[0087] In some embodiments, before obtaining the trained congestion prediction model, the trained congestion prediction model can also be obtained by performing the following steps: obtaining a sample data set collected for the expressway and a congestion prediction model to be trained, the sample data set including sample video data and annotation information corresponding to the sample video data; calling the convolutional neural network layer in the congestion prediction model to be trained, performing feature extraction and time series analysis on the sample video data, and obtaining sample time series data; calling the recurrent neural network layer in the congestion prediction model to be trained, predicting the sample time series data, and obtaining a first sample congestion prediction result at a second current moment, a second sample congestion prediction result at a third future moment, and a third sample congestion prediction result at a fourth future moment, wherein the third future moment is before the fourth future moment, and the second current moment is the moment when the collection of the sample video data ends; based on the first sample congestion prediction result, the second sample congestion prediction result, the third sample congestion prediction result and the annotation information, the congestion prediction model to be trained is trained to obtain the trained congestion prediction model.

[0088] Here, the sample video data is road monitoring video data of the open section of the emergency lane of the expressway within 1 to 2 years. The sample video data includes sample image frames in the state of traffic congestion and non-congestion. Each sample image frame is labeled with a "congested" label or a "non-congested" label to obtain the labeling information corresponding to the sample video data. The sample video data is divided into multiple time windows, each of which includes a fixed number of continuous sample image frames. The multiple sample image frames in the sample video data are scaled and normalized for preprocessing, and the convolutional neural network layer MobileNetV2 network in the congestion prediction model to be trained is called to extract features and perform time series analysis on the preprocessed sample image frames to obtain sample time series data. The recurrent neural network layer LSTM network in the congestion prediction model to be trained is called to predict the sample time series data. The sample congestion prediction results at each moment are output through an independent fully connected layer, and the activation function sigmoid is used to convert the output into the first sample congestion prediction result at the second current moment, the second sample congestion prediction result at the third future moment, and the third sample congestion prediction result at the fourth future moment.

[0089] In an embodiment of the present application, based on the first sample congestion prediction result, the second sample congestion prediction result, the third sample congestion prediction result and the annotation information, the congestion prediction model to be trained is trained to obtain a trained congestion prediction model, and then when congestion prediction is performed using the trained congestion prediction model, the accuracy of the congestion prediction result is improved.

[0090] In some embodiments, the congestion prediction model to be trained can be trained based on the first sample congestion prediction result, the second sample congestion prediction result, the third sample congestion prediction result and the annotation information to obtain the trained congestion prediction model by executing the following steps: determining a first loss value based on the first sample congestion prediction result and the annotation information, determining a second loss value based on the second sample congestion prediction result and the annotation information, and determining a third loss value based on the third sample congestion prediction result and the annotation information; determining a total loss value of the congestion prediction model to be trained based on the first loss value, the second loss value and the third loss value; updating the model parameters of the congestion prediction model to be trained based on the total loss value to obtain the trained congestion prediction model.

[0091] Here, based on the first sample congestion prediction result and the annotation information, the first loss value of the first sample congestion prediction result is determined. Based on the second sample congestion prediction result and the annotation information, the second loss value of the second sample congestion prediction result is determined. Based on the third sample congestion prediction result and the annotation information, the third loss value of the third sample congestion prediction result is determined. The first loss value, the second loss value, and the third loss value can be calculated by a cross entropy loss function. Different weights can be set for the first loss value, the second loss value, and the third loss value, and the first loss value, the second loss value, and the third loss value are weighted and summed to obtain the total loss value of the congestion prediction model to be trained. Based on the total loss value, the model parameters of the congestion prediction model to be trained are updated to obtain the trained congestion prediction model.

[0092] In an embodiment of the present application, based on the first loss value of the first sample congestion prediction result, the second loss value of the second sample congestion prediction result, and the third loss value of the third sample congestion prediction result, the total loss value of the congestion prediction model to be trained is determined, and based on the total loss value, the model parameters of the congestion prediction model to be trained are updated to obtain the trained congestion prediction model, and then when congestion prediction is performed using the trained congestion prediction model, the accuracy of the congestion prediction results is improved.

[0093] Continue to refer Figure 3A In step 302, a plurality of information source data for the highway is obtained, and a first weight of each information source data is determined respectively, and a first weight matrix is ​​determined based on each first weight.

[0094] Here, the multiple information source data are data from different business domains such as highway dispatching domain, toll collection domain, and maintenance domain. The first weight refers to the weight of each information source data, and the first weight matrix is ​​a matrix composed of the weight of each information source data. Obtain multiple information source data for the highway, and divide the multiple information source data into different data groups based on the spatial attributes of the information source data and the camera point topological structure. One camera point data corresponds to one data group, and each data group corresponds to a different spatial position of the highway. Determine the first weight of each information source data in different data groups respectively, and determine the first weight matrix based on the first weight of each information source data.

[0095] In some embodiments, see Figure 3D , Figure 3D This is a fourth flow chart of the road data processing method provided in an embodiment of the present application. Figure 3A In step 302 shown in the figure, "determine the first weight of each information source data respectively", which can be achieved by Figure 3D Steps 3021 to 3024 are implemented as described in detail below.

[0096] In step 3021, for each information source data, based on the information source data, a state value of the highway in the scenario to which the information source data belongs is determined.

[0097] Here, the state value is used to characterize the performance of the highway in the scenario to which the information source data belongs.

[0098] In some embodiments, the state value of the highway in the scenario to which the information source data belongs can be determined based on the information source data by executing the following steps: based on the information source data, determine the state information of the highway in the scenario to which the information source data belongs; when the state information represents a positive state, determine the state value as a first value; when the state information represents a negative state, determine the state value as a second value, and the first value is greater than the second value; when the state information represents an uncertain state, determine the state value as a third value, and the third value is greater than the second value and less than the first value.

[0099] Here, the state information is used to characterize the actual state of the highway in the scenario to which the information source data belongs, and the state information includes a positive state, a negative state, and an uncertain state. Based on each information source data, the actual state of the highway in the scenario to which the information source data belongs is determined, and the state information containing positive performance is determined as a positive state, the state information containing negative performance is determined as a negative state, and the state information containing uncertain performance is determined as an uncertain state. In the case where the state information represents a positive state, the state value is determined as a first value, and the first value can be set to 1. In the case where the state information represents a negative state, the state value is determined as a second value, the first value is greater than the second value, and the second value can be set to 0. In the case where the state information represents an uncertain state, the state value is determined as a third value, the third value is greater than the second value and less than the first value, and the third value can be set to 0.5.

[0100] For example, the information source data may be road sensor data. Based on the road sensor data, the state information of the highway in terms of road conditions is determined, and the state information of good road conditions is determined as a positive state, the state information of poor road conditions is determined as a negative state, and the state information of uncertain road conditions is determined as an uncertain state. In the case where the state information represents a positive state, the state value is determined as a first value, and the first value can be set to 1. In the case where the state information represents a negative state, the state value is determined as a second value, and the second value can be set to 0. In the case where the state information represents an uncertain state, the state value is determined as a third value, and the third value can be set to 0.5.

[0101] In an embodiment of the present application, the state information of the highway in the scenario to which each information source data belongs is determined. When the state information represents a positive state, the state value is determined as a first value; when the state information represents a negative state, the state value is determined as a second value; when the state information represents an uncertain state, the state value is determined as a third value, wherein the first value is greater than the second value, and the third value is greater than the second value and less than the first value, so that the weight of each information source data can be determined subsequently based on the state value.

[0102] In step 3022, the reliability weight of each information source data is determined based on the state value and the number of historical opening processing decision events of the emergency lane.

[0103] Here, for each information source data, the number of times the state value of the highway in the scenario to which the information source data belongs is the first value is counted, and the number of times the first value is divided by the number of historical opening and processing decision events of the emergency lane is divided, and the obtained ratio result is used as the reliability weight of each information source data, that is, the frequency of the success rate of the information source data in the historical opening and processing decision events of the emergency lane. The value range of the reliability weight is [0,1], and the reliability weight of 1 indicates that the historical performance of the information source data is the most reliable.

[0104] For example, the reliability weight of each information source data can be determined by formula (4):

[0105]

[0106] Among them, r i represents the reliability weight, f i represents the number of times the state value is the first value, and F represents the number of historical opening processing decision events of the emergency lane.

[0107] In step 3023, based on the update frequency of each information source data and a preset time period, the timeliness weight of each information source data is determined.

[0108] Here, for each information source data, the update frequency is the number of times the information source data is updated per unit time, and the preset time period is the total time period, such as one day or one hour. The update frequency of each information source data is divided by the preset time period, and the obtained ratio result is used as the timeliness weight of each information source data, that is, the update rate of the information source data. The value range of the timeliness weight is [0,1]. The higher the update frequency of the information source data within the preset time period, the higher the timeliness weight of the information source data.

[0109] For example, the timeliness weight of each information source data can be determined by formula (5):

[0110]

[0111] Among them, t i represents the timeliness weight, U i Indicates the update frequency of the information source data, and T indicates the preset time period.

[0112] In step 3024, based on the reliability weight and the timeliness weight, the first weight of each information source data is determined respectively.

[0113] Here, for each information source data, a time adjustment coefficient is set according to the actual scenario requirements, and the number of information source data categories is counted. The time adjustment coefficient is a coefficient that affects the time decay speed. The first weight of each information source data is determined by using the time adjustment coefficient and the number of information source data categories, combined with the reliability weight and timeliness weight of the information source data.

[0114] For example, the first weight of each information source data can be determined by formula (6):

[0115]

[0116] Among them, ωi represents the first weight of the i-th information source data, r i Represents the reliability weight of the information source data, t i represents the timeliness weight of the information source data, α represents the time adjustment coefficient, and n represents the number of information source data categories.

[0117] In the embodiment of the present application, the reliability weight of each information source data is determined based on the state value of the highway in the scenario to which the information source data belongs and the number of historical opening and processing decision events of the emergency lane. The timeliness weight of each information source data is determined based on the update frequency and preset time period of each information source data, and then the weight of each information source data is determined by combining the reliability weight and the timeliness weight, which fully considers the historical performance and real-time performance of different information source data, and improves the accuracy of emergency lane opening decisions.

[0118] Continue to refer Figure 3A In step 303, the membership of each information source data under the evaluation level is determined, and based on each membership, a first evaluation matrix is ​​determined.

[0119] Here, the membership degree is the degree to which each information source data belongs to a certain evaluation level. The first evaluation matrix is ​​a matrix composed of the membership degree of each information source data. According to the membership degree of each information source data under different evaluation levels, the first evaluation matrix is ​​established.

[0120] In some embodiments, the evaluation level includes a first evaluation level, a second evaluation level, and a third evaluation level, see Figure 3E , Figure 3E This is the fifth flow chart of the road data processing method provided in the embodiment of the present application. Figure 3A In step 303 shown in the figure, "determining the degree of membership of each information source data under the evaluation level" can be achieved by Figure 3E Steps 3031 to 3033 are implemented as described in detail below.

[0121] In step 3031, based on the upper limit value of the first evaluation level, the lower limit value of the first evaluation level and the state value, the membership degree of each information source data under the first evaluation level is determined.

[0122] Here, the first evaluation level is an evaluation level used to characterize that the emergency lane is not open. According to the actual scene requirements, the upper limit value of the first evaluation level and the lower limit value of the first evaluation level are preset. Based on the commonality of different information source data, the first membership function of the first evaluation level is constructed using a trapezoidal distribution function. For each information source data, based on the upper limit value of the first evaluation level, the lower limit value of the first evaluation level and the state value, the value of the first membership function is determined as the membership of the information source data under the first evaluation level.

[0123] For example, the membership degree of each information source data at the first evaluation level can be determined by formula (7):

[0124]

[0125] Among them, x represents the state value of the information source data, a L Indicates the lower limit of the first evaluation level, b L Indicates the upper limit value of the first evaluation level.

[0126] In step 3032, based on the upper limit value of the first evaluation level, the upper limit value of the second evaluation level, the preset state value and the state value of the second evaluation level, the membership degree of each information source data at the second evaluation level is determined.

[0127] Here, the second evaluation level is an evaluation level used to characterize the partial opening of the emergency lane. According to the actual scene requirements, the upper limit value of the second evaluation level and the preset state value of the first evaluation level are preset. Based on the commonality of different information source data, the second membership function of the second evaluation level is constructed using a trapezoidal distribution function. For each information source data, based on the upper limit value of the first evaluation level, the upper limit value of the second evaluation level, the preset state value and the state value of the second evaluation level, the value of the second membership function is determined as the membership of the information source data under the second evaluation level.

[0128] For example, the membership degree of each information source data at the second evaluation level can be determined by formula (8):

[0129]

[0130] Among them, x represents the state value of the information source data, b L represents the upper limit of the first evaluation level, c M Indicates the preset state value of the second evaluation level, d M Indicates the upper limit value of the second evaluation level.

[0131] In step 3033, based on the upper limit value of the second evaluation level, the upper limit value of the third evaluation level and the state value, the membership degree of each information source data under the third evaluation level is determined.

[0132] Here, the third evaluation level is an evaluation level used to characterize the fully open emergency lane. The upper limit value of the third evaluation level is preset according to the actual scene requirements. Based on the commonality of different information source data, the third membership function of the third evaluation level is constructed using a trapezoidal distribution function. For each information source data, based on the upper limit value of the second evaluation level, the upper limit value of the third evaluation level and the state value, the value of the third membership function is determined as the membership of the information source data under the third evaluation level.

[0133] For example, the membership degree of each information source data at the third evaluation level can be determined by formula (9):

[0134]

[0135] Among them, x represents the state value of the information source data, d M represents the upper limit of the second evaluation level, d H Indicates the upper limit value of the third evaluation level.

[0136] In the embodiment of the present application, the evaluation levels include a first evaluation level, a second evaluation level and a third evaluation level. By determining the membership of each information source data at different evaluation levels, an evaluation matrix for multiple information source data can be established based on each membership.

[0137] Continue to refer Figure 3A In step 304, based on the first weight matrix and the first evaluation matrix, a second congestion prediction result is determined, and based on the first congestion prediction result and the second congestion prediction result, an opening processing decision of the emergency lane in the highway is determined.

[0138] In some embodiments, the second congestion prediction result can be determined based on the first weight matrix and the first evaluation matrix by executing the following steps: multiplying the first weight matrix and the first evaluation matrix to obtain a second evaluation matrix; transposing the second evaluation matrix to obtain a transposed second evaluation matrix; obtaining a score matrix corresponding to the evaluation level, and determining the second congestion prediction result based on the transposed second evaluation matrix and the score matrix.

[0139] Here, the second congestion prediction result is based on multiple information source data of the expressway to determine the prediction result of the emergency lane opening processing decision in the expressway. According to the actual scene requirements, the first evaluation level, the second evaluation level, and the third evaluation level are respectively set with corresponding scores to obtain the score matrix corresponding to the evaluation level. The transposed second evaluation matrix and the score matrix are multiplied and summed to obtain the second congestion prediction result.

[0140] For example, the second evaluation matrix can be determined by formula (12):

[0141]

[0142] Among them, S represents the second evaluation matrix, A represents the first weight matrix, and R represents the first evaluation matrix.

[0143] The scores of the first evaluation level, the second evaluation level, and the third evaluation level are set to 100, 75, and 50, respectively, and the score matrix corresponding to the evaluation level is obtained [100, 75, 50]. The second congestion prediction result can be determined by formula (13):

[0144] S minor_defuzz =S E ·S T ∈[0,100] (13)

[0145] Among them, S minor_defuzz represents the second congestion prediction result, W E represents the score matrix, S T represents the second evaluation matrix after transposition.

[0146] In an embodiment of the present application, the second congestion prediction result is determined based on the first weight matrix and the first evaluation matrix. Through the improved fuzzy comprehensive evaluation method, the data from different information sources are weighted based on timeliness and historical reliability. A defuzzification analysis method that distinguishes between primary and secondary factors is used to achieve a comprehensive decision on the opening of the emergency lane. The impact of data from multiple information sources on the opening of the emergency lane is fully considered, thereby improving the accuracy of the decision on the opening of the emergency lane.

[0147] In some embodiments, the opening processing decision of the emergency lane in the highway can be determined based on the first congestion prediction result and the second congestion prediction result by executing the following steps: determining the second weight of the first congestion prediction result and the third weight of the second congestion prediction result; based on the second weight and the third weight, performing weighted summation of the first congestion prediction result and the second congestion prediction result to obtain a third congestion prediction result, wherein the third congestion prediction result represents an evaluation value of congestion occurring on the highway; based on the third congestion prediction result and a preset threshold, determining the opening processing decision of the emergency lane in the highway.

[0148] Here, according to actual scenario requirements, the second weight of the first congestion prediction result and the third weight of the second congestion prediction result are preset, so as to perform weighted summation on the first congestion prediction result and the second congestion prediction result to obtain the third congestion prediction result.

[0149] For example, the third congestion prediction result can be determined by formula (14):

[0150] S defuzz =μ*100*S main_defuzz +(1-μ)×S minor_defuzz(14)

[0151] Among them, S defuzz represents the third congestion prediction result, S main_defuzz represents the first congestion prediction result, S minor_defuzz represents the second congestion prediction result, μ represents the second weight of the first congestion prediction result, and 1-μ represents the third weight of the second congestion prediction result. μ can be set to 0.7, that is, the first congestion prediction result accounts for 70% of the weight in the third congestion prediction result.

[0152] In an embodiment of the present application, a weighted sum is performed on the first congestion prediction result and the second congestion prediction result to obtain a third congestion prediction result, which fully considers the spatiotemporal characteristics of the expressway and the impact of multi-information source data on the opening of the emergency lane, thereby improving the accuracy of the emergency lane opening decision.

[0153] In some embodiments, the opening processing decision of the emergency lane in the highway can be determined based on the third congestion prediction result and the preset threshold by executing the following steps: when the third congestion prediction result is greater than or equal to the preset threshold, the emergency lane in the highway is opened; when the third congestion prediction result is less than the preset threshold, the emergency lane in the highway remains closed.

[0154] Here, the preset threshold is the emergency lane opening decision threshold. The third congestion prediction result is compared with the preset threshold to determine the emergency lane opening decision. When the third congestion prediction result is greater than or equal to the preset threshold, the emergency lane is determined to be opened; when the third congestion prediction result is less than the preset threshold, the emergency lane remains closed.

[0155] In an embodiment of the present application, when the third congestion prediction result is greater than or equal to a preset threshold, the emergency lane in the highway is opened; when the third congestion prediction result is less than the preset threshold, the emergency lane in the highway remains closed, thereby improving the real-time performance of the emergency lane opening decision.

[0156] Example, reference Figure 4 , Figure 4It is a structural diagram of the training process of the congestion prediction model provided by the embodiment of the present application. In the congestion prediction model 401 to be trained, the convolutional neural network layer 403 is called to perform feature extraction and time series analysis on the sample video data 402 to obtain sample time series data. The recurrent neural network layer 404 is called to predict the sample time series data to obtain the first sample congestion prediction result at the second current moment, the second sample congestion prediction result at the third future moment, and the third sample congestion prediction result at the fourth future moment. The third future moment is located before the fourth future moment, and the second current moment is the acquisition end moment of the sample video data 402. The first loss value is determined based on the first sample congestion prediction result and the annotation information 405, the second loss value is determined based on the second sample congestion prediction result and the annotation information 405, and the third loss value is determined based on the third sample congestion prediction result and the annotation information 405. Based on the first loss value, the second loss value and the third loss value, the total loss value of the congestion prediction model 401 to be trained is determined, and based on the total loss value, the model parameters of the congestion prediction model 401 to be trained are updated to obtain the trained congestion prediction model.

[0157] The following describes the congestion prediction process of the trained congestion prediction model provided in the embodiment of the present application. Figure 5 , Figure 5 It is a structural schematic diagram of the congestion prediction process of the congestion prediction model provided in the embodiment of the present application. In the congestion prediction model 501, the convolutional neural network layer 503 is called to perform feature extraction and time series analysis on the video data 502 collected for the highway to obtain time series data. The recurrent neural network layer 504 is called to predict the time series data to obtain the congestion prediction result of the first current moment, the congestion prediction result of the first future moment, and the congestion prediction result of the second future moment. The first future moment is before the second future moment, and the first current moment is the end time of the video data collection. Based on the congestion prediction result of the first current moment, the congestion prediction result of the first future moment, and the congestion prediction result of the second future moment, the first congestion prediction result is determined.

[0158] In some embodiments, the road data processing method provided by the embodiment of the present application can be applied in the field of autonomous driving. The cloud server obtains the video data collected for the highway, performs feature extraction and time series analysis on the video data, and obtains the time series data. The time series data has time characteristics and spatial characteristics. Then, based on the time series data, the first congestion prediction result is determined, and the prediction result of the emergency lane opening decision is determined based on the spatiotemporal characteristics of the highway. Then, multiple information source data for the highway are obtained, and the first weight of each information source data is determined respectively, and based on each first weight, the first weight matrix is ​​determined, and then the membership of each information source data under the evaluation level is determined, and based on each membership, the first evaluation matrix is ​​determined, and then based on the first weight matrix and the first evaluation matrix, the second congestion prediction result is determined, and the prediction result of the emergency lane opening decision is comprehensively determined based on multiple information source data of the highway. And based on the first congestion prediction result and the second congestion prediction result, the opening processing decision of the emergency lane in the highway is determined, so that the spatiotemporal characteristics of the highway and the influence of multiple information source data on the opening of the emergency lane are fully considered, and the real-time and accuracy of the emergency lane opening decision are improved.

[0159] The following describes an exemplary application of the road data processing method provided in an embodiment of the present application in a highway emergency lane opening scenario.

[0160] In view of the problems existing in the related art, the embodiment of the present application proposes a road data processing method, which includes the following improvements compared with the related art:

[0161] The improved MobileNet-LSTM network (congestion prediction model) is used to predict congestion in road surveillance video images, improving the real-time and accuracy of congestion identification. The weights of different information source data are determined based on timeliness and historical reliability. The improved fuzzy comprehensive evaluation method is combined with the defuzzification analysis method of primary and secondary decision factors to calculate the comprehensive score of congestion prediction (the second congestion prediction result). When the comprehensive score is greater than the preset threshold, the emergency lane is opened. Through the integrated vehicle-road-cloud architecture, the on-board display screen and road information screen can publish emergency lane opening information, further improving the real-time performance of emergency lane management and control.

[0162] For example, first collect basic data and highway congestion recognition datasets. Basic data include historical highway emergency lane opening status data, highway network topology data and camera point data. Based on historical highway emergency lane opening status data, obtain supplementary data such as historical emergency lane opening duration, opening reasons, traffic flow, emergency lane occupancy, etc. Based on highway network topology data and camera point data, obtain camera point topology structure, that is, road connection relationship and driving direction of camera points. Obtain highway congestion recognition dataset through 1-2 years of road monitoring video data of highway emergency lane open section. Congestion recognition dataset includes image frames in traffic congestion and non-congestion state, and each image frame carries the label of "congestion" or "non-congestion". Split the monitoring video data into multiple time windows, the time window is set to 15 minutes, and each time window contains a fixed number of continuous image frames. The image frames are scaled, normalized and pre-processed for image enhancement for model training and verification. Based on the congestion recognition dataset, train a congestion recognition network based on spatiotemporal fusion features.

[0163] The congestion identification network based on spatiotemporal fusion features is the MobileNet-LSTM network. The original MobileNet network is supplemented with a modified LSTM network. The MobileNet-LSTM network includes the MobileNetV2 network (spatial feature extraction network) and the LSTM network (temporal feature analysis network). The MobileNet-LSTM network can classify congestion based on the spatial features of the image, consider the temporal dependency between image frames, identify the current congestion status and predict the future t s and t l Two time points (t s <t l )’s congestion status, the current congestion status (the first current moment’s congestion prediction result) and the congestion status at future time points (the first future moment’s congestion prediction result, the second future moment’s congestion prediction result) will serve as the main decision basis for opening the emergency lane.

[0164] Use the MobileNetv2 network as the basic convolutional neural network. The MobileNetv2 network is an efficient lightweight network. Its core is the depth-separable convolution and linear bottleneck structure, which reduces the computational complexity while maintaining a high feature extraction capability. Specifically, the MobileNetv2 network structure includes an initial standard convolution layer (3x3), multiple layer-by-layer convolution layers and residual blocks. The residual block further optimizes the feature extraction process through the linear bottleneck structure while reducing the computational overhead. Delete the last fully connected layer in the MobileNetv2 network, retain only the last convolution layer, and finally output a feature vector F with a dimension of D.t , where t represents the current time frame, i.e. F t =MobileNetv2(X t ), where X t Represents the image data input at the current time frame.

[0165] In the LSTM network, the feature maps from the MobileNetv2 network are flattened into time series data {F t , F t-1 ,......,F t-N+1}, where N in the subscript is the length of the fixed time window. Each time window contains a fixed number of continuous image frame features. The time series data is expanded to increase the number of video image frames to reflect traffic changes in a longer time span. The shape of the expanded time series data is (T, D), where T = N + M, which is the length of the expanded time window, N and M represent the fixed time window length and the additional extended time window length, respectively, and D is the dimension of the feature vector. The feature vector of each time frame is extracted through the MobileNetv2 network, and the LSTM network processes the expanded time series data {F t , F t-1 ,......,F t-(N+M-1) In addition, in order to further improve the sensitivity of the LSTM network to time information, the timestamp is used as an additional input of the LSTM network to allow the LSTM network to distinguish traffic status changes in different time periods. Set the timestamp feature to T t , then the input of the LSTM network is ([F t , T t ], H t-1 ), where H t-1 is the hidden state of the LSTM network at the previous time step.

[0166] The LSTM network is modeled based on extended time series data containing timestamps to capture long-term dependencies between time steps. To improve the generalization ability of the model, a random dropout layer is added to the LSTM network to prevent overfitting, and batch normalization is used. n ) layer to stabilize the training process. The LSTM network is designed to output three key results to comprehensively judge the congestion status and the emergency lane opening strategy. Specifically, the output of the LSTM network includes the current congestion status, the future t s The congestion status of the minute and the future l The current congestion status reflects the traffic congestion at the current time point. sThe congestion status of the minute is used to evaluate the congestion in a short period of time (since the decision and implementation of the opening of the emergency lane requires a certain amount of human and material resources, when congestion is predicted in a short period of time, the emergency lane will not be considered for opening to avoid unnecessary waste of resources). l The congestion status of each minute is used to evaluate the congestion situation over a longer period of time (the long-term opening of the emergency lane may make it unable to handle emergencies and affect its emergency function. When long-term congestion is predicted, the emergency lane will not be considered for opening to avoid the impact of the long-term opening of the emergency lane on the emergency handling capacity).

[0167] In order to predict the congestion status at future moments, the LSTM network is designed as a multi-output structure. The prediction at each moment is obtained through an independent fully connected layer. Specifically, the output H of the LSTM network t is mapped to the current time y t , Future Moments and The congestion probability at three moments can be determined by formula (1):

[0168]

[0169] Among them, W t , Represents the weight matrix of the current moment and the next two moments, b t , is the bias term, and σ is the activation function sigmoid.

[0170] In the MobileNet-LSTM network training phase, forward propagation is first performed to input the video frame image into the MobileNetv2 network. The MobileNetv2 network extracts spatial features from each frame image to obtain multiple feature maps, and organizes these feature maps into time series data. The LSTM network processes the time series data, and the LSTM network output is mapped to the congestion probability at the current moment and the next two moments through the fully connected layer. During the training process, the cross entropy loss function is used to calculate the error between the MobileNet-LSTM network output and the true label. The design of the loss function includes the current moment, the future t s Minutes, future l The weighted sum of the predicted losses at each minute (based on the first loss value, the second loss value and the third loss value, the total loss value of the congestion prediction model to be trained is determined) to comprehensively consider the accuracy of short-term and long-term predictions. The weighted sum of the predicted losses at each moment can be determined by formula (2):

[0171]

[0172] Among them, Loss current , Respectively represent the current time and the future time s Minutes, future i The prediction loss of the minute. α, β, and γ represent the weight coefficients of the loss function, which are used to balance the prediction accuracy at different moments. α, β, and γ can be set to 0.5, 0.3, and 0.2 respectively.

[0173] The back propagation algorithm is used to calculate the gradient and update the network weights. The optimizer selects Adaptive Moment Estimation to minimize the loss function. During the training process, the mini-batch training method is used to accelerate the training process and improve stability. During the MobileNet-LSTM network verification and testing phase, the validation set is used to evaluate the network performance. The evaluation indicators include accuracy and F1 score to ensure the effectiveness and reliability of the MobileNet-LSTM network in practical applications.

[0174] The operation process of the MobileNet-LSTM network includes: first, receiving real-time image data from highway monitoring cameras. Based on the camera point topology, the image data is divided into different data groups, one camera point data corresponds to one data group, which is used for congestion identification at different spatial locations and subsequent emergency lane opening decisions. The grouped image data is preprocessed, the image data is adjusted to the input size required by the MobileNetv2 network (such as 224x224 pixels), and normalized to ensure that the pixel value is within the appropriate range. The preprocessed image data is input into the MobileNetv2 network. The MobileNetv2 network uses its efficient convolutional structure to extract feature maps of the image data. These feature maps contain important information about the image content. Then, the extracted feature maps are organized into time series data, and the time series data is input into the LSTM network. The LSTM network is responsible for processing the time series data. Each output of the LSTM network is obtained through an independent fully connected layer. The activation function sigmoid converts the output into congestion judgment results at three moments, namely, whether it is congested at the current moment, after t s Is there congestion after t minutes? l Is there congestion in minutes? If and only if the current congestion status is "congested", the future t s The congestion status will be "congested" after t l When the congestion status is "not congested" after 1 minute, the final output result of the network (the first congestion prediction result) is: F MobileNet-LSTM(x) = 1, that is, it is recommended to open the emergency lane on this road section. In other cases, the final output of the network is: F MobileNet-LSTM (x)=0, that is, it is not recommended to open an emergency lane on this road section.

[0175] The multi-source data fusion strategy in the embodiment of the present application aims to optimize the open control strategy of the emergency lane by integrating data from multiple information sources as auxiliary decision-making factors for the emergency lane, so as to make up for the defect that congestion identification does not fully consider the actual scene conditions. Receive and integrate data from different information sources (from different business domains such as highway scheduling, toll collection, maintenance, etc.), convert data from different information sources into floating point form of [0, 1], and obtain the performance value (state value) of data from different information sources. Based on the spatial attributes of the information and the topological structure of the camera point, the data from different information sources are divided into different data groups, and one camera point data corresponds to one data group, ensuring the accuracy and effectiveness of the decision space.

[0176] The information source data may be traffic incident report data, and the traffic incident report data is used to confirm whether a traffic accident has occurred on the main road section. If the traffic incident report data indicates that a traffic accident has occurred, the performance value of the information source data to which the traffic incident report data belongs is 1; if the traffic incident report data indicates that no accident has occurred, the performance value of the information source data to which the traffic incident report data belongs is 0; if the traffic incident report data indicates that the accident situation is uncertain, the performance value of the information source data to which the traffic incident report data belongs is an intermediate value (which can be set to 0.5). At the same time, the traffic incident report data is also used to ensure whether there is a vehicle parked in the emergency lane. If the traffic incident report data indicates that there is a vehicle parked in the emergency lane, the performance value of the information source data to which the traffic incident report data belongs is 0; if the traffic incident report data indicates that there is no vehicle parked in the emergency lane, the performance value of the information source data to which the traffic incident report data belongs is 1; if the traffic incident report data indicates that the parking situation of the vehicle in the emergency lane is uncertain, the performance value of the information source data to which the traffic incident report data belongs is an intermediate value (which can be set to 0.5).

[0177] The information source data may be traffic incident alarm data, which is used to confirm whether there is an emergency vehicle that needs to use the emergency lane at present. If the traffic incident alarm data indicates that there is a demand for use, then the performance value of the information source data to which the traffic incident alarm data belongs is 0; if the traffic incident alarm data indicates that there is no demand for use, then the performance value of the information source data to which the traffic incident alarm data belongs is 1; if the traffic incident alarm data indicates that the demand situation is uncertain, then the performance value of the information source data to which the traffic incident alarm data belongs is an intermediate value (which can be set to 0.5).

[0178] The information source data may be weather station data, which provides information about the weather conditions of a road section. If the weather station data indicates that there is bad weather on the road section, the performance value of the information source data to which the weather station data belongs is 0; if the weather station data indicates that the weather conditions on the road section are good, the performance value of the information source data to which the weather station data belongs is 1; if the weather station data indicates that the weather conditions on the road section are uncertain, the performance value of the information source data to which the weather station data belongs is an intermediate value (which can be set to 0.5).

[0179] The information source data may be construction information data, which is used to provide the construction status of the road section. If the construction information data indicates that the road section is under construction, then the performance value of the information source data to which the construction information data belongs is 0; if the construction information data indicates that the road section is not under construction, then the performance value of the information source data to which the construction information data belongs is 1; if the construction information data indicates that the construction status of the road section is uncertain, then the performance value of the information source data to which the construction information data belongs is an intermediate value (which can be set to 0.5).

[0180] The information source data may be service area passenger flow data, which is used to confirm whether there is traffic congestion at the service area exit on the road section. If the service area passenger flow data indicates that there is traffic congestion, the performance value of the information source data to which the service area passenger flow data belongs is 0; if the service area passenger flow data indicates that there is no traffic congestion, the performance value of the information source data to which the service area passenger flow data belongs is 1; if the service area passenger flow data indicates that the congestion situation is uncertain, the performance value of the information source data to which the service area passenger flow data belongs is an intermediate value (which can be set to 0.5).

[0181] The information source data may be road surface sensor data, which provides information on the road condition of the road section to which it belongs. If the road surface sensor data indicates that the road condition is poor, then the performance value of the information source data to which the road surface sensor data belongs is 0; if the road surface sensor data indicates that the road condition is good, then the performance value of the information source data to which the road surface sensor data belongs is 1; if the road surface sensor data indicates that the road condition is uncertain, then the performance value of the information source data to which the road surface sensor data belongs is an intermediate value (which can be set to 0.5).

[0182] The information source data may be emergency lane management data, which is used to confirm whether the time from the last emergency lane opening is more than 15 minutes. If the emergency lane management data indicates that the emergency lane is still open, the performance value of the information source data to which the emergency lane management data belongs is 0; if the emergency lane management data indicates that the time interval is more than 15 minutes, the performance value of the information source data to which the emergency lane management data belongs is 1; if the emergency lane management data indicates that the time interval is between 0 and 15 minutes, the performance value of the information source data to which the emergency lane management data belongs can be determined by formula (3):

[0183]

[0184] Where T is the time interval (in minutes) since the last emergency lane opening.

[0185] In order to further optimize the decision-making of emergency lane opening, the information source data can also be historical congestion pattern data, which can be used to determine whether it is currently in the morning and evening peak periods. If the historical congestion pattern data indicates that it is not in the peak period, the performance value of the information source data to which the historical congestion pattern data belongs is 0; if the historical congestion pattern data indicates that it is in the peak period, the performance value of the information source data to which the historical congestion pattern data belongs is 1. The information source data can also be traffic warning information data, which can help identify whether it is in the period of large-scale events or holidays. If the traffic warning information data indicates that it is not in the period of large-scale events or holidays, the performance value of the information source data to which the traffic warning information data belongs is 0; if the traffic warning information data indicates that it is in the period of large-scale events or holidays, the performance value of the information source data to which the traffic warning information data belongs is 1.

[0186] In order to ensure the scientificity and accuracy of the emergency lane opening decision, the reliability and timeliness of each information source data are evaluated. Reliability and timeliness consider the two main factors of the historical performance and real-time performance of the information source data respectively. It is necessary to calculate the reliability weight and timeliness weight of the information source data respectively. The reliability weight reflects the stability and accuracy of the information source data in the historical lane opening decision process. Using statistical methods, by quantifying the performance of the information source data in historical emergency lane opening decision events (such as 50 emergency lane opening events), the reliability weight of each information source data can be determined by formula (4):

[0187]

[0188] Among them, r i represents the reliability weight, f i represents the number of times the performance value of the information source data is 1, and F represents the number of historical emergency lane opening decision events. Formula (4) calculates the frequency of the success rate of the information source data in historical emergency lane opening decision events. i The value range is [0, 1], r i =1 indicates that the historical performance of the information source data is the most reliable.

[0189] The timeliness weight reflects the update frequency and real-time nature of the information source data. By quantifying the update frequency of the information source data, the timeliness weight of each information source data can be determined by formula (5):

[0190]

[0191] Among them, t i represents the timeliness weight, U i represents the update frequency of the information source data (the number of times the information source data is updated per unit time), and T represents the total time period (for example, one day or one hour). Formula (5) calculates the update rate of the information source data. The higher the update frequency of the information source data in the total time period, the higher the timeliness weight of the information source data, and the timeliness weight t i The value range is [0, 1].

[0192] In order to comprehensively quantify the two factors of timeliness and reliability, the weight of each information source data can be determined by formula (6):

[0193]

[0194] Among them, ω i represents the weight of the i-th information source data, r i Represents the reliability weight of the information source data, t i represents the timeliness weight of the information source data, α represents the adjustment factor, which affects the speed of time decay, and n represents the total number of information source data categories.

[0195] Combine the weights of each information source data to form a weight matrix (first weight matrix) [ω1 ω2 ......ω n ] A fuzzy evaluation matrix (first evaluation matrix) based on fuzzy set theory is constructed. The fuzzy evaluation matrix is ​​based on three evaluation levels: emergency lane not open, emergency lane partially open, and emergency lane fully open (first evaluation level, second evaluation level, and third evaluation level), and describes the membership of each information source data at different evaluation levels. Using the membership function μ L (x), μ M (x) and μ H (x) represents the membership degree of each information source data under the three evaluation levels of emergency lane not open, emergency lane partially open, and emergency lane fully open. Substitute the performance values ​​of different information source data into the membership function to obtain the membership degrees of different information source data under different evaluation levels.

[0196] The membership degree of the information source data under the emergency lane non-opening evaluation level can be determined by formula (7):

[0197]

[0198] Among them, x represents the performance value of the corresponding information source data, a LIndicates the lower limit of the evaluation level of emergency lane not being open, which means the worst performance, b L It indicates the upper limit of the evaluation level when the emergency lane is not open, and it is also the lower limit of the evaluation level when the emergency lane is partially open, which means the medium minimum standard performance.

[0199] The membership of the information source data under the evaluation level of partial opening of the emergency lane can be determined by formula (8):

[0200]

[0201] Among them, x represents the performance value of the corresponding information source data, b L Indicates the upper limit of the emergency lane closed evaluation level, c M Indicates the typical performance value (preset state value) of the partial opening evaluation level of the emergency lane, d M It represents the upper limit of the evaluation grade for partial opening of the emergency lane, and is also the lower limit of the evaluation grade for full opening of the emergency lane, which means it is close to excellent but has not yet reached the high-level standard performance.

[0202] The membership degree of the information source data under the emergency lane fully open evaluation level can be determined by formula (9):

[0203]

[0204] Among them, x represents the performance value of the corresponding information source data, d M Indicates the upper limit of the evaluation level of partial opening of the emergency lane, d H It indicates the upper limit of the evaluation level of the emergency lane being fully open, which indicates the best performance.

[0205] According to the membership degree of each information source data at different evaluation levels, a fuzzy evaluation matrix is ​​established. The fuzzy evaluation matrix can be determined by formula (10):

[0206]

[0207] Among them, R represents the fuzzy evaluation matrix, x i represents the performance value of the i-th information source data, μ L (x i ), μ M (x i ) and μ H (x i ) respectively represent the membership degree of the information source data under the three evaluation levels of emergency lane not open, emergency lane partially open, and emergency lane fully open.

[0208] The output of the MobileNet-LSTM network is used as the score of the main decision factor, and the score of the main decision factor can be determined by formula (11):

[0209] S main_defuzz =F MobileNet-LSTM (x)∈{0,1} (11)

[0210] Among them, S main_defuzz Indicates the score of the main decision factor, F MobileNet-LSTM (x) represents the output result of the MobileNet-LSTM network.

[0211] The fuzzy evaluation matrix and the weight matrix are multiplied to obtain the defuzzified scoring matrix (the second evaluation matrix). The defuzzified scoring matrix can be determined by formula (12):

[0212]

[0213] Among them, S represents the defuzzified rating matrix, A represents the weight matrix, and R represents the fuzzy evaluation matrix.

[0214] The scores of the three evaluation levels of emergency lane not open, emergency lane partially open, and emergency lane fully open are set to 100, 75, and 50 respectively, and the score matrix corresponding to the evaluation level is obtained [100, 75, 50]. The score matrix is ​​multiplied by the transposed defuzzified score matrix and then summed to obtain the defuzzified value (the second congestion prediction result), and the defuzzified value is used as the score of the auxiliary decision factor. The score of the auxiliary decision factor can be determined by formula (13):

[0215] S minor_defuzz =W E ·S T ∈[0,100] (13)

[0216] Among them, S minor_defuzz represents the score of auxiliary decision-making factors, W E represents the score matrix, S T represents the defuzzified rating matrix after transposition.

[0217] Different weights are applied to the scores of the main decision factors and the scores of the auxiliary decision factors to obtain the comprehensive score of the congestion prediction (the third congestion prediction result). The comprehensive score of the congestion prediction can be determined by formula (14):

[0218] S defuzz =μ*100*S main_defuzz +(1-μ)×S minor_defuzz (14)

[0219] Among them, S defuzzrepresents the comprehensive score, μ represents the weight coefficient, S main_defuzz represents the score of the main decision factor, S minor_defuzz Represents the score of auxiliary decision-making factors.

[0220] For example, μ can be set to 0.7, that is, the main decision factor score accounts for 70% of the influence in the comprehensive score. The comprehensive score is compared with the set decision threshold to determine the opening decision of the emergency lane. If the comprehensive score is greater than or equal to the set decision threshold, the emergency lane is opened; if the comprehensive score is less than the set decision threshold, the emergency lane remains closed.

[0221] In the embodiment of the present application, the emergency lane opening decision system based on multi-source data fusion adopts an integrated vehicle-road-cloud emergency lane control system architecture. The information flow transmission of this architecture is divided into three main levels: edge (road end), cloud end, and terminal device, to achieve efficient traffic congestion detection and emergency lane opening control.

[0222] Edge devices are deployed near traffic surveillance cameras or at key locations on highways. The MobieNetv2-LSTM network is deployed using embedded computing units to perform real-time congestion identification on the video stream from the camera and upload the MobieNetv2-LSTM network output data to the cloud. Due to the limited computing power of edge devices, a lightweight convolutional neural network MobieNetv2 is used to ensure real-time and accuracy when processing large amounts of data. The cloud performs a comprehensive analysis of the data from the edge, and combines multi-source data such as meteorological data, traffic incident reports, and emergency vehicle tracking information to make a comprehensive decision on the opening of emergency lanes. The cloud also has efficient data storage and processing capabilities, which can support long-term data storage and historical data analysis, complete the training of the congestion identification network MobieNetv2-LSTM, and regularly update the weights of multi-source data. Terminal devices usually include emergency lane control equipment, highway roadside information display screens, and networked vehicle information presentation terminals. These terminal devices receive decision results from the cloud and perform corresponding control operations according to the instructions, such as starting the opening of the emergency lane, presenting the emergency lane opening information on the roadside, or directly providing real-time traffic information to the vehicle side, to ensure the effective use of the emergency lane and smooth traffic flow.

[0223] In the above-mentioned application scenario of emergency lane opening on expressways, the long short-term memory network is introduced on the basis of the lightweight overall image classification network to learn the spatiotemporal characteristics of congestion and consider the impact of congestion duration on emergency lane opening, thereby improving the accuracy and real-time performance of congestion identification. The weights of different information source data are determined based on timeliness and historical reliability, and the emergency lane opening decision is determined through the improved fuzzy comprehensive evaluation method combined with the defuzzification analysis method of primary and secondary decision factors, which improves the accuracy of emergency lane opening decision and further improves the real-time performance of emergency lane opening and response.

[0224] The following is a description of an exemplary structure of a road data processing device 455 provided in an embodiment of the present application implemented as a software module. In some embodiments, for example, Figure 2 As shown, the software modules stored in the road data processing device 455 of the memory 450 may include: a first determination module 4551, which is used to obtain video data collected for the expressway, perform feature extraction and time series analysis on the video data to obtain time series data, and determine a first congestion prediction result based on the time series data; a second determination module 4552, which is used to obtain multiple information source data for the expressway, and respectively determine a first weight of each information source data, and determine a first weight matrix based on each first weight; a third determination module 4553, which is used to determine the membership of each information source data under the evaluation level, and determine a first evaluation matrix based on each membership; a second determination module 4554, which is used to determine a second congestion prediction result based on the first weight matrix and the first evaluation matrix, and determine an open processing decision for the emergency lane in the expressway based on the first congestion prediction result and the second congestion prediction result.

[0225] In some embodiments, the video data includes multiple image frames, and the first determination module 4551 is also used to obtain a trained congestion prediction model; call the convolutional neural network layer in the trained congestion prediction model to extract features from each image frame, and obtain a corresponding feature vector of each image frame; based on the feature vector of each image frame and a preset time window length, construct time series data.

[0226] In some embodiments, the first determination module 4551 is also used to perform expansion processing on the time series data to obtain updated time series data; obtain the timestamp corresponding to each feature vector in the updated time series data; call the recurrent neural network layer in the trained congestion prediction model to predict the updated time series data and each timestamp to obtain the congestion prediction result of the first current moment, the congestion prediction result of the first future moment, and the congestion prediction result of the second future moment, the first future moment is before the second future moment, and the first current moment is the end time of video data acquisition; based on the congestion prediction result of the first current moment, the congestion prediction result of the first future moment, and the congestion prediction result of the second future moment, determine the first congestion prediction result.

[0227] In some embodiments, the first determination module 4551 is also used to obtain a sample data set collected for the expressway and a congestion prediction model to be trained before obtaining the trained congestion prediction model, the sample data set including sample video data and annotation information corresponding to the sample video data; call the convolutional neural network layer in the congestion prediction model to be trained to perform feature extraction and time series analysis on the sample video data to obtain sample time series data; call the recurrent neural network layer in the congestion prediction model to be trained to predict the sample time series data to obtain a first sample congestion prediction result at a second current moment, a second sample congestion prediction result at a third future moment, and a third sample congestion prediction result at a fourth future moment, the third future moment being before the fourth future moment, and the second current moment being the moment when the collection of the sample video data ends; based on the first sample congestion prediction result, the second sample congestion prediction result, the third sample congestion prediction result and the annotation information, train the congestion prediction model to be trained to obtain a trained congestion prediction model.

[0228] In some embodiments, the first determination module 4551 is also used to determine a first loss value based on the first sample congestion prediction result and annotation information, determine a second loss value based on the second sample congestion prediction result and annotation information, and determine a third loss value based on the third sample congestion prediction result and annotation information; determine the total loss value of the congestion prediction model to be trained based on the first loss value, the second loss value and the third loss value; based on the total loss value, update the model parameters of the congestion prediction model to be trained to obtain the trained congestion prediction model.

[0229] In some embodiments, the second determination module 4552 is also used to determine, for each information source data, based on the information source data, the state value of the highway in the scenario to which the information source data belongs; determine the reliability weight of each information source data based on the state value and the number of historical opening and processing decision events of the emergency lane; determine the timeliness weight of each information source data based on the update frequency and preset time period of each information source data; and determine the first weight of each information source data based on the reliability weight and the timeliness weight.

[0230] In some embodiments, the second determination module 4552 is also used to determine the status information of the highway in the scenario to which the information source data belongs based on the information source data; when the status information represents a positive state, the status value is determined to be a first value; when the status information represents a negative state, the status value is determined to be a second value, and the first value is greater than the second value; when the status information represents an uncertain state, the status value is determined to be a third value, and the third value is greater than the second value and less than the first value.

[0231] In some embodiments, the evaluation levels include a first evaluation level, a second evaluation level and a third evaluation level, and the third determination module 4553 is further used to determine the membership of each information source data at the first evaluation level based on the upper limit value of the first evaluation level, the lower limit value of the first evaluation level and the status value; determine the membership of each information source data at the second evaluation level based on the upper limit value of the first evaluation level, the upper limit value of the second evaluation level, the preset status value and the status value of the second evaluation level; determine the membership of each information source data at the third evaluation level based on the upper limit value of the second evaluation level, the upper limit value of the third evaluation level and the status value.

[0232] In some embodiments, the fourth determination module 4554 is further used to multiply the first weight matrix and the first evaluation matrix to obtain a second evaluation matrix; transpose the second evaluation matrix to obtain a transposed second evaluation matrix; obtain a score matrix corresponding to the evaluation level, and determine a second congestion prediction result based on the transposed second evaluation matrix and the score matrix.

[0233] In some embodiments, the fourth determination module 4554 is also used to determine the second weight of the first congestion prediction result and the third weight of the second congestion prediction result; based on the second weight and the third weight, the first congestion prediction result and the second congestion prediction result are weightedly summed to obtain a third congestion prediction result, and the third congestion prediction result represents an evaluation value of congestion on the highway; based on the third congestion prediction result and a preset threshold, a decision on the opening of the emergency lane in the highway is determined.

[0234] In some embodiments, the fourth determination module 4554 is also used to open the emergency lane in the highway when the third congestion prediction result is greater than or equal to a preset threshold; when the third congestion prediction result is less than the preset threshold, the emergency lane in the highway remains closed.

[0235] The embodiment of the present application provides a computer program product, which includes computer executable instructions or computer programs, and the computer executable instructions or computer programs are stored in a computer readable storage medium. The processor of the electronic device reads the computer executable instructions or computer programs from the computer readable storage medium, and the processor executes the computer executable instructions or computer programs, so that the electronic device executes the road data processing method described in the embodiment of the present application.

[0236] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the method provided by the present application embodiment, for example, Figure 3A The road data processing method shown.

[0237] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0238] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0239] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0240] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0241] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A road data processing method, characterized in that: The method comprises: Acquire video data collected from the expressway, perform feature extraction and time series analysis on the video data to obtain time series data, and determine a first congestion prediction result based on the time series data; Acquire a plurality of information source data for the expressway, and respectively determine a first weight for each of the information source data, and determine a first weight matrix based on each of the first weights; Determine the membership of each of the information source data under the evaluation level, and determine a first evaluation matrix based on each of the membership; Based on the first weight matrix and the first evaluation matrix, a second congestion prediction result is determined, and based on the first congestion prediction result and the second congestion prediction result, an opening processing decision of the emergency lane in the expressway is determined.

2. The method according to claim 1, characterized in that The video data includes a plurality of image frames, and the feature extraction and time series analysis of the video data are performed to obtain time series data, including: Obtain the trained congestion prediction model; Calling the convolutional neural network layer in the trained congestion prediction model to extract features from each of the image frames, and correspondingly obtaining feature vectors of each of the image frames; The time series data is constructed based on the feature vectors of each of the image frames and a preset time window length.

3. The method according to claim 2, characterized in that The determining a first congestion prediction result based on the time series data includes: Performing expansion processing on the time series data to obtain updated time series data; Obtaining the timestamp corresponding to each feature vector in the updated time series data; Calling the recurrent neural network layer in the trained congestion prediction model to predict the updated time series data and each of the timestamps to obtain a congestion prediction result at a first current moment, a congestion prediction result at a first future moment, and a congestion prediction result at a second future moment, wherein the first future moment is before the second future moment, and the first current moment is the moment when the collection of the video data ends; A first congestion prediction result is determined based on the first congestion prediction result at the current moment, the first congestion prediction result at the future moment, and the congestion prediction result at the second future moment.

4. The method according to claim 2, characterized in that: Before obtaining the trained congestion prediction model, the method further includes: Acquire a sample data set collected for the expressway and a congestion prediction model to be trained, wherein the sample data set includes sample video data and annotation information corresponding to the sample video data; Calling the convolutional neural network layer in the congestion prediction model to be trained to perform feature extraction and time series analysis on the sample video data to obtain sample time series data; Calling the recurrent neural network layer in the congestion prediction model to be trained to predict the sample time series data, to obtain a first sample congestion prediction result at a second current moment, a second sample congestion prediction result at a third future moment, and a third sample congestion prediction result at a fourth future moment, wherein the third future moment is before the fourth future moment, and the second current moment is the moment when the collection of the sample video data ends; Based on the first sample congestion prediction result, the second sample congestion prediction result, the third sample congestion prediction result and the annotation information, the congestion prediction model to be trained is trained to obtain a trained congestion prediction model.

5. The method according to claim 4, characterized in that The step of training the congestion prediction model to be trained based on the first sample congestion prediction result, the second sample congestion prediction result, the third sample congestion prediction result and the annotation information to obtain the trained congestion prediction model includes: Determine a first loss value based on the first sample congestion prediction result and the annotation information, determine a second loss value based on the second sample congestion prediction result and the annotation information, and determine a third loss value based on the third sample congestion prediction result and the annotation information; Determining a total loss value of the congestion prediction model to be trained based on the first loss value, the second loss value, and the third loss value; Based on the total loss value, the model parameters of the congestion prediction model to be trained are updated to obtain a trained congestion prediction model.

6. The method according to claim 1, characterized in that The step of respectively determining the first weight of each of the information source data comprises: For each of the information source data, based on the information source data, determining a state value of the highway in the scenario to which the information source data belongs; Determining the reliability weight of each of the information source data based on the state value and the number of historical opening and processing decision events of the emergency lane; Determine the timeliness weight of each of the information source data based on the update frequency and the preset time period of each of the information source data; Based on the reliability weight and the timeliness weight, a first weight of each information source data is determined respectively.

7. The method according to claim 6, characterized in that The determining, based on the information source data, a state value of the highway in the scenario to which the information source data belongs, includes: Based on the information source data, determining the state information of the highway in the scenario to which the information source data belongs; In a case where the state information represents a positive state, determining the state value as a first value; In the case where the state information represents a negative state, determining the state value as a second value, the first value being greater than the second value; In a case where the state information represents an uncertain state, the state value is determined to be a third value, and the third value is greater than the second value and less than the first value.

8. The method according to claim 7, characterized in that The evaluation levels include a first evaluation level, a second evaluation level, and a third evaluation level, and the determining of the degree of membership of each of the information source data under the evaluation level includes: Determining the membership degree of each of the information source data under the first evaluation level based on the upper limit value of the first evaluation level, the lower limit value of the first evaluation level and the state value; Determining the membership degree of each of the information source data at the second evaluation level based on the upper limit value of the first evaluation level, the upper limit value of the second evaluation level, the preset state value of the second evaluation level, and the state value; Based on the upper limit value of the second evaluation level, the upper limit value of the third evaluation level and the state value, the degree of membership of each of the information source data under the third evaluation level is determined.

9. The method according to claim 1, characterized in that: The determining a second congestion prediction result based on the first weight matrix and the first evaluation matrix includes: Multiplying the first weight matrix and the first evaluation matrix to obtain a second evaluation matrix; Transposing the second evaluation matrix to obtain a transposed second evaluation matrix; A score matrix corresponding to the evaluation level is obtained, and a second congestion prediction result is determined based on the transposed second evaluation matrix and the score matrix.

10. The method according to claim 1, characterized in that The step of determining an opening decision of an emergency lane in the expressway based on the first congestion prediction result and the second congestion prediction result includes: determining a second weight of the first congestion prediction result and a third weight of the second congestion prediction result; Based on the second weight and the third weight, weighted summing the first congestion prediction result and the second congestion prediction result is performed to obtain a third congestion prediction result, wherein the third congestion prediction result represents an evaluation value of congestion occurring on the expressway; Based on the third congestion prediction result and a preset threshold, a decision on opening an emergency lane in the expressway is determined.

11. The method according to claim 10, characterized in that The step of determining the opening decision of the emergency lane in the expressway based on the third congestion prediction result and a preset threshold value includes: When the third congestion prediction result is greater than or equal to the preset threshold, opening the emergency lane in the expressway; When the third congestion prediction result is less than the preset threshold, the emergency lane in the expressway remains closed.

12. A road data processing device, characterized in that: The device comprises: A first determination module is used to obtain video data collected for the expressway, perform feature extraction and time series analysis on the video data to obtain time series data, and determine a first congestion prediction result based on the time series data; A second determination module is used to obtain a plurality of information source data for the expressway, and respectively determine a first weight of each of the information source data, and determine a first weight matrix based on each of the first weights; A third determination module is used to determine the membership of each of the information source data under the evaluation level, and determine a first evaluation matrix based on each of the membership; The fourth determination module is used to determine a second congestion prediction result based on the first weight matrix and the first evaluation matrix, and to determine an opening processing decision of the emergency lane in the highway based on the first congestion prediction result and the second congestion prediction result.

13. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor is used to implement the road data processing method described in any one of claims 1 to 11 when executing the computer executable instructions or computer programs stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the road data processing method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the road data processing method according to any one of claims 1 to 11 is implemented.

Images