Traffic monitoring video stream processing method, device, system and storage medium

Abstract

The present invention discloses a method, device, system and storage medium for processing traffic monitoring video streams, the method comprising: establishing communication connections between multiple roadside units in a preset area and multiple cameras, and updating information tables in multiple roadside units; determining multiple-feature selective transmission strategies and computing resource allocation strategies in an edge server according to the updated information tables and a real-time scheduling algorithm for multi-feature stream transmission; uploading video streams corresponding to preset task information in multiple cameras according to the determined strategy information, and performing online reasoning and analysis on the uploaded video streams in the edge server; performing real-time tracking and prediction of target objects in the preset task information according to the online reasoning and analysis results, and outputting the tracking and prediction results. The present invention proposes an integrated end-edge-cloud architecture for large-scale real-time monitoring video stream processing, providing accurate low-energy response and control services for existing traffic monitoring systems.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic monitoring, and in particular to a method, device, system and storage medium for processing traffic monitoring video streams. Background Art

[0002] Currently, for a traffic monitoring system driven by a series of specific tasks such as vehicle tracking, the traffic control center first sends the task to all cameras in the jurisdiction. The cameras capture the monitoring video stream in real time and upload the compressed and encoded video stream to the roadside unit or server. The roadside unit or server performs basic processing and storage on the collected monitoring video stream for cloud server to call. However, the explosive growth of surveillance video data and limited computing resources and communication resources have brought challenges to the development of traffic monitoring. This traditional traffic monitoring video stream processing method not only transmits a large amount of invalid information, resulting in bandwidth waste, but also increases the processing burden of the cloud server of the traffic control center, because the cloud server needs to analyze the compressed video streams uploaded by all cameras in real time. The traffic monitoring system gradually relies on more communication and computing resources to improve performance, which is undoubtedly a high-energy consumption and low-efficiency method. Therefore, there is an urgent need for an interactive framework for traffic monitoring video stream transmission and processing to improve the processing efficiency of the end-edge cloud real-time monitoring system.

[0003] Therefore, the prior art needs to be improved. Summary of the invention

[0004] The technical problem to be solved by the present invention is that, in view of the defects of the prior art, the present invention provides a traffic monitoring video stream processing method, device, system and storage medium to solve the technical problems of low efficiency and high energy consumption in processing monitoring video streams in the existing traffic monitoring system.

[0005] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0006] In a first aspect, the present invention provides a method for processing a traffic monitoring video stream, the method comprising:

[0007] Establishing communication connections between a plurality of roadside units in a preset area and a plurality of cameras, and updating information tables in the plurality of roadside units;

[0008] Determine a selective transmission strategy and a computing resource allocation strategy for multiple features in the edge server according to the updated information table and the real-time scheduling algorithm for multiple feature stream transmission;

[0009] Upload video streams corresponding to preset task information in multiple cameras according to the determined strategy information, and perform online reasoning and analysis on the uploaded video streams in the edge server;

[0010] Based on the online reasoning and analysis results, the target objects in the preset task information are tracked and predicted in real time, and the tracking and prediction results are output.

[0011] In one implementation, establishing communication connections between a plurality of roadside units in a preset area and a plurality of cameras includes:

[0012] Obtain preset task information and send the preset task information to multiple roadside units in a preset area through the traffic control center.

[0013] In one implementation, establishing communication connections between a plurality of roadside units in a preset area and a plurality of cameras, and updating information tables in the plurality of roadside units, includes:

[0014] Establishing communication connections between the multiple roadside units and the multiple cameras according to the video transmission protocol, and initializing the communication status of the multiple cameras;

[0015] Control each camera to broadcast the current device status to multiple roadside units, and update the corresponding information table in the multiple roadside units;

[0016] The current device status includes: the computing resources required by each camera, the size of the video data, the percentage of used resources, and the location coordinates of the monitored object.

[0017] In one implementation, based on the updated information table and the real-time scheduling algorithm for multi-feature stream transmission, a multi-feature selective transmission strategy and a computing resource allocation strategy are determined in the edge server, including:

[0018] Extracting and compressing a feature stream in real time from a raw image signal in a retinal camera, or extracting and compressing a semantic stream in real time from a raw video signal in a retinal camera;

[0019] Deploy a real-time scheduling algorithm for multi-feature streaming in edge servers;

[0020] Using the updated information table and real-time scheduling algorithm, the selective transmission strategy and computing resource allocation strategy of multiple features are determined in the edge server.

[0021] In one implementation, a real-time scheduling algorithm for multi-feature stream transmission is deployed in an edge server, and the method also includes:

[0022] Images and videos are acquired from a single-function camera, compressed, and transmitted to multiple roadside units.

[0023] In one implementation, the updated information table and the real-time scheduling algorithm are used to determine the selective transmission strategy and computing resource allocation strategy of multiple features in the edge server, including:

[0024] Get a collection of video streams or images in a preset area;

[0025] Determine the amount of computation required for image processing based on the images in the set, or determine the amount of computation required for performing compression coding, feature extraction, and semantic segmentation based on the frame rate and resolution of the video streams in the set;

[0026] Determine the computing power and transmission energy consumption of each camera and edge server per unit time;

[0027] According to the required computing amount, computing power per unit time, and transmission energy consumption, the selective transmission strategy corresponding to each camera and the computing resource allocation strategy corresponding to the edge server are determined.

[0028] In one implementation, determining the selective transmission strategy corresponding to each camera and the computing resource allocation strategy corresponding to the edge server includes:

[0029] According to the computing power and transmission energy consumption of each camera and edge server, a selective transmission and computing resource allocation model is established;

[0030] Solve the selective transmission and computing resource allocation model according to the whale optimization algorithm to obtain the corresponding strategy set;

[0031] Introduce a penalty factor and determine the objective function based on the penalty factor;

[0032] According to the objective function and the number of iterations, the strategy set is optimized to obtain the optimized selective transmission strategy and computing resource allocation strategy of the edge server;

[0033] Decision information is fed back to each camera through multiple roadside units.

[0034] In one implementation, video streams corresponding to preset task information are uploaded to multiple cameras according to the determined strategy information, and online reasoning and analysis are performed on the uploaded video streams in the edge server, including:

[0035] Upload the corresponding video stream on each camera according to the determined strategy information and preset task information;

[0036] The edge server performs online reasoning and analysis on the uploaded video stream and feeds back information to the corresponding camera after the target object is found.

[0037] According to the feedback information, the original image or video associated with the target object is uploaded to the corresponding camera;

[0038] The associated original image or video is uploaded to the traffic control center for storage through the corresponding roadside unit.

[0039] In one implementation, real-time tracking and prediction of target objects in preset task information are performed based on online reasoning and analysis results, including:

[0040] determining the acceleration and yaw rate of the target object according to the updated information table;

[0041] Select the corresponding path prediction model according to the acceleration and yaw rate;

[0042] Predicting the location coordinates of the target object within a preset time period in the future by using the selected path prediction model;

[0043] Update the future driving trajectory of the target object according to the predicted position coordinates;

[0044] The path prediction models include: a constant speed motion model, a constant acceleration motion model, a constant yaw rate model, and a constant yaw rate and acceleration model.

[0045] In one implementation, real-time tracking and prediction of target objects in preset task information are performed based on online reasoning and analysis results, further comprising:

[0046] Predict the monitoring area that the target object will enter in the future time period based on the future driving trajectory;

[0047] If it is predicted that the target object will leave the current monitoring area in the future time period, the current monitoring camera will be switched to the camera of the next area;

[0048] Upload the associated video stream through the switched camera.

[0049] In a second aspect, the present invention provides a traffic monitoring video stream processing device, comprising: a processor and a memory, wherein the memory stores a traffic monitoring video stream processing program, and when the traffic monitoring video stream processing program is executed by the processor, it is used to implement the traffic monitoring video stream processing method as in the first aspect.

[0050] In a third aspect, the present invention provides a traffic monitoring video stream processing system, comprising:

[0051] multiple roadside units;

[0052] Multiple cameras;

[0053] Edge servers;

[0054] Traffic Control Center;

[0055] and the traffic monitoring video stream processing device as in the second aspect;

[0056] The traffic monitoring video stream processing device is used to perform the following operations:

[0057] Establishing communication connections between a plurality of roadside units in a preset area and a plurality of cameras, and updating information tables in the plurality of roadside units;

[0058] Determine a selective transmission strategy and a computing resource allocation strategy for multiple features in the edge server according to the updated information table and the real-time scheduling algorithm for multiple feature stream transmission;

[0059] Upload video streams corresponding to preset task information in multiple cameras according to the determined strategy information, and perform online reasoning and analysis on the uploaded video streams in the edge server;

[0060] Based on the online reasoning and analysis results, the target objects in the preset task information are tracked and predicted in real time, and the tracking and prediction results are output.

[0061] In a fourth aspect, the present invention provides a storage medium, which is a computer-readable storage medium and stores a traffic monitoring video stream processing program. When the traffic monitoring video stream processing program is executed by a processor, it is used to implement the traffic monitoring video stream processing method as in the first aspect.

[0062] The present invention adopts the above technical solution to achieve the following effects:

[0063] Compared with the traditional specific task-driven traffic monitoring system, the present invention proposes a traffic monitoring video stream processing method based on an end-edge-cloud system to address the problem of traffic monitoring system processing limitations caused by large bandwidth and computing resource requirements for video stream transmission and processing. Efficient processing of monitoring video streams in the context of traffic safety is achieved through the mutual collaboration of various modules in the architecture. In addition, the present invention also proposes an integrated end-edge-cloud architecture for large-scale real-time monitoring video stream processing, providing accurate low-energy response and control services for existing traffic monitoring systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0065] Figure 1 It is a flow chart of a method for processing a traffic monitoring video stream in one implementation of the present invention.

[0066] Figure 2 It is a schematic diagram of a traffic monitoring video stream processing system (framework) in one implementation of the present invention.

[0067] Figure 3 It is an interactive framework diagram for traffic monitoring video stream transmission and processing in one implementation of the present invention.

[0068] Figure 4 It is a schematic diagram of a DS reasoning system in one implementation of the present invention.

[0069] Figure 5 It is a schematic diagram of four commonly used motion models in one implementation of the present invention.

[0070] Figure 6 It is a functional principle diagram of a traffic monitoring video stream processing device in one implementation of the present invention.

[0071] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0073] Exemplary Methods

[0074] In traditional traffic monitoring systems, for monitoring of specific tasks (such as vehicle tracking), the traffic control center needs to send the task to all cameras in the jurisdiction. The cameras capture the monitoring video stream in real time and upload the compressed and encoded video stream to the roadside unit or server. The roadside unit or server performs basic processing and storage on the collected monitoring video stream for cloud server to call. This traditional traffic monitoring video stream processing method not only transmits a large amount of invalid information, resulting in a waste of bandwidth, but also increases the processing burden of the cloud server of the traffic control center.

[0075] In view of the above technical problems, this embodiment provides a method for processing traffic monitoring video streams, which realizes efficient processing of monitoring video streams in the context of traffic safety through the mutual cooperation of various modules in the architecture. In addition, the present invention also proposes an integrated end-edge-cloud architecture for large-scale real-time monitoring video stream processing, providing accurate low-energy response and control services for existing traffic monitoring systems.

[0076] like Figure 1 As shown, an embodiment of the present invention provides a method for processing a traffic monitoring video stream, and the method for processing a traffic monitoring video stream includes the following steps:

[0077] Step S100, establishing communication connections between multiple roadside units in a preset area and multiple cameras, and updating information tables in the multiple roadside units.

[0078] In this embodiment, if Figure 2 As shown, this embodiment is aimed at the end-edge-cloud scenario of traffic monitoring video stream processing, and proposes an interactive framework for traffic monitoring video stream transmission and processing (such as Figure 3 As shown in the figure), it utilizes limited communication resources and computing resources to maximize bandwidth utilization and reduce the computing power requirements for task processing.

[0079] First, this embodiment uses a heterogeneous terminal including a traditional camera and a new retinal camera to upload a multi-feature video stream. This transmission method can greatly reduce the burden on network bandwidth and reduce the processing burden of edge servers and cloud servers.

[0080] Secondly, this embodiment utilizes a real-time scheduling algorithm based on multi-feature stream transmission deployed on the edge server, which can realize the selective transmission of multi-feature streams and the allocation of computing resources of the edge server, while ensuring the minimization of energy consumption of video stream transmission and processing.

[0081] In addition, this embodiment can more accurately predict the vehicle's driving trajectory at future moments through the path prediction algorithm deployed on the edge server, so that the roadside unit can reasonably call the video stream of the corresponding camera without collecting video data from all cameras in real time.

[0082] Specifically, in this embodiment, in order to solve the problem that traffic monitoring video streams have ultra-high bandwidth and ultra-strong computing resource requirements, an edge-cloud interaction framework for traffic monitoring video stream transmission and processing is proposed. The framework includes an initialization and broadcast module, an edge scheduling and decision module, an edge reasoning module, a path prediction module, and a monitoring switching management module.

[0083] First, the traffic control center issues specific tasks to the roadside unit; second, based on the video transmission protocol, device-to-device communication is established between the camera and the roadside unit to ensure the reliable transmission of the video stream. Multiple cameras update the information table by broadcasting the current status to the roadside unit; third, the edge server realizes the selective transmission of multiple feature streams and the allocation of computing resources of the edge server according to the information table and the real-time scheduling algorithm based on multi-feature stream transmission, and at the same time feeds back decision information to the camera.

[0084] After feedback of decision information, the camera selectively processes the original video stream into a compressed video stream, feature stream or semantic stream based on the decision information, and uploads it to the edge server of the current road section for online reasoning and analysis; after the edge server finds the target vehicle, it feeds back information to the corresponding camera, and the camera uploads the key original images or videos related to the target vehicle to the roadside unit for the traffic control center to call and store; at the same time, the roadside unit uses the relevant path prediction model to predict the target vehicle's driving trajectory; then, the roadside unit sends information to the corresponding camera based on the predicted vehicle's driving trajectory, and the camera continues to monitor and upload the video stream. The end-edge-cloud system is used to provide low-bandwidth and low-computing power edge computing and control services for the existing monitoring system, ultimately realizing a low-energy integrated end-edge-cloud intelligent control and management architecture.

[0085] That is, in one implementation of this embodiment, the following steps are included before step S100:

[0086] Step S001, obtaining preset task information, and sending the preset task information to multiple roadside units in a preset area through a traffic control center.

[0087] In one implementation of this embodiment, the proposed interactive framework for traffic monitoring video stream transmission and processing includes five parts:

[0088] 1. Initialization and broadcast module: The traffic control center issues specific tasks to the roadside unit, camera-to-edge communication is established to ensure reliable transmission of video streams, and multiple cameras update the information table by broadcasting the current status to the roadside unit.

[0089] 2. Edge scheduling and decision-making module: The roadside unit uses the updated information and the real-time scheduling algorithm based on multi-feature stream transmission to realize the selective transmission of multi-feature streams and the allocation of computing resources of the edge server, and at the same time feeds back the decision information to the camera.

[0090] 3. Edge reasoning module: After receiving the information, the camera uploads the corresponding video stream. The edge server integrated with the roadside unit performs online reasoning and analysis on the received video stream. After finding the target vehicle, it feeds back information to the corresponding camera. The camera uploads the key original image or video related to the target vehicle. The roadside unit uploads the acquired image or video to the traffic control center for storage.

[0091] 4. Path prediction module: The roadside unit uses the relevant path prediction model to predict the driving trajectory of the target vehicle.

[0092] 5. Monitoring switching management module: If the current target vehicle is about to leave the monitoring range of the camera on this road section, the roadside unit sends information to the camera on the next road section based on the predicted vehicle driving trajectory, and the camera on the next road section continues to monitor and upload the video stream.

[0093] Specifically, in an implementation of this embodiment, step S100 includes the following steps:

[0094] Step S101, establishing communication connections between multiple roadside units and multiple cameras according to a video transmission protocol, and initializing the communication states of the multiple cameras;

[0095] Step S102, control each camera to broadcast the current device status to multiple roadside units, and update the corresponding information table in multiple roadside units; wherein the current device status includes: the computing resources required by each camera, the size of the video data, the percentage of used resources, and the location coordinates of the monitored object.

[0096] In an actual application scenario of this embodiment, tracking and monitoring task information is sent from the traffic control center to each camera; wherein the traffic control center is also the cloud center, which is responsible for managing road traffic safety in a certain area. The roads in the area are divided into several sections, each of which corresponds to the coverage area of ​​the current roadside unit. The roadside units on each section are connected to the cloud center via a wired link, and the cloud center centrally controls them. At the same time, the camera on each section can only upload the video stream to a roadside unit on the same section as the camera.

[0097] Furthermore, there are a series of specific tasks (for example, vehicle tracking) in the cloud center, which are sent to all roadside units in the jurisdiction, and the roadside units assist in executing the tasks; a communication link is established between multiple cameras and roadside units to ensure reliable transmission of video streams to complete communication initialization. Among them, the communication link can be wired or wireless, and the wireless link includes 4G / 5G / WIFI, etc. The camera broadcasts the current device status to the roadside units on the road section, and the broadcast information includes: the computing resources required by the camera, the size of the video data, the percentage of used resources, and the location coordinates of the monitored vehicle, etc. At the same time, the roadside unit updates the information table. Once the roadside unit obtains the status information of all cameras, the information transmission is completed at that moment.

[0098] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the traffic monitoring video stream processing method further includes the following steps:

[0099] Step S200, determining a multi-feature selective transmission strategy and a computing resource allocation strategy in the edge server according to the updated information table and the real-time scheduling algorithm for multi-feature stream transmission.

[0100] In this embodiment, based on the real-time scheduling algorithm for multi-feature stream transmission, the selective transmission strategy and computing resource allocation strategy of the multi-features are determined in the edge server;

[0101] Specifically, when determining the selective transmission strategy and computing resource allocation strategy of multiple features, for new retinal cameras in cities with feature extraction and semantic segmentation functions, compressed feature streams / semantic streams are extracted from the original image / video signal in real time for pattern recognition and visual analysis. The depth / semantic features are extracted frame by frame from the original video sequence. After the extracted features are quantized, they are losslessly compressed using arithmetic coding methods. A real-time scheduling algorithm based on multi-feature stream transmission is deployed on the edge server co-located with the roadside unit. The edge server uses the updated information table and the scheduling algorithm to determine the selective transmission of multi-feature streams and the computing resource allocation of the edge server in real time.

[0102] Specifically, in an implementation of this embodiment, step S200 includes the following steps:

[0103] Step S201, extracting and compressing a feature stream from an original image signal in a retinal camera in real time, or extracting and compressing a semantic stream from an original video signal in a retinal camera in real time;

[0104] Step S202, deploying a real-time scheduling algorithm for multi-feature stream transmission in the edge server;

[0105] Step S203, using the updated information table and the real-time scheduling algorithm, a multi-feature selective transmission strategy and a computing resource allocation strategy are determined in the edge server.

[0106] In another implementation of this embodiment, for cameras with traditional single functions in cities, each of which is usually designed for a specific purpose, the images and videos obtained from the cameras are compressed and prepared for transmission to the roadside units for online detection and storage, and provided to the central cloud as needed.

[0107] That is, before step S202, the following steps are also included:

[0108] Step S201a, acquiring images and videos from a single-function camera, compressing the acquired images and videos, and transmitting them to a plurality of roadside units.

[0109] In an actual application scenario of this embodiment, it can be assumed that the set of roadside units or road sections is N = {1, 2, ..., N}, each roadside unit can completely cover the current road section, and the set of video stream tasks or cameras on the nth road section is K = {1, 2, ..., K}. In order to avoid interference between cameras, the spectrum in the roadside unit is orthogonally allocated to each camera. Each roadside unit is co-located with an edge server and can only provide computing resources for the cameras on the road section. The frame rate of the kth video stream is s k , the resolution of each frame is p kTraditional monitoring equipment can only upload compressed video streams. It is assumed that the amount of computation required to compress a pixel video frame is f C , described by floating point operations (FLOPs). In addition to uploading compressed video streams, new surveillance devices can also choose to upload feature streams or semantic streams. The amount of computation required to perform feature extraction and semantic segmentation operations on a pixel video frame is f and f respectively. F and f S , where f F and f S Both are described by FLOPs.

[0110] Assume that the kth camera device selects a video stream to upload, let and They represent the indicators that the video stream that camera k chooses to upload is a compressed video stream and a feature stream, respectively. Then, the indicator that camera k chooses to upload a semantic stream is in, And the following inequalities are satisfied:

[0111]

[0112] If the kth camera is a traditional single-function camera, then

[0113] Furthermore, the delay of processing a video frame on the kth camera device can be expressed as:

[0114]

[0115] Among them, k Indicates the percentage of resources used by the device, f k Indicates the computing power of the kth camera device; the resolution of each frame is p k .

[0116] Furthermore, the processing energy consumption of the video stream per unit time on the kth camera device is Among them, β k represents the power consumed by the task calculation of camera k, which depends on the complexity of the calculation task; the frame rate of the kth video stream is s k .

[0117] Furthermore, the data size of a video frame of the video stream of the kth camera device after compression encoding, feature extraction, and semantic segmentation operations is Described by the number of bits, the kth camera transmits the processed video stream to the nth roadside unit, and the delay of the camera transmitting a video frame can be expressed as:

[0118]

[0119] in, and They represent the wireless and wired bandwidths provided by the nth roadside unit to the camera, and y k An indicator indicating that the kth camera is a wireless camera. If the kth camera is a wireless camera, then y k =1, if it is a wired camera, then y k =0.

[0120] Furthermore, let γ k represents the energy consumption of the kth camera to transmit 1 bit of task, then the transmission energy consumption per unit time of the camera can be expressed as:

[0121]

[0122] Furthermore, after the roadside unit receives the task, the edge server performs reasoning and target detection. represents the amount of computation required to perform reasoning operations on a pixel compression, feature, and semantic tasks, respectively, and r k,n represents the computing resources (FLOPS) allocated by the nth edge server to the kth camera, and the latency of edge reasoning for one video frame is:

[0123]

[0124] Furthermore, the energy consumption per unit time of reasoning of the edge server is where β n Represents the power consumed by the task computation at the nth edge server.

[0125] That is, in one implementation of this embodiment, step S203 includes the following steps:

[0126] Step S203a, obtaining a set of video streams or images in a preset area;

[0127] Step S203b, determining the amount of computation required for image processing according to the images in the set, or determining the amount of computation required for performing compression coding, feature extraction, and semantic segmentation according to the frame rate and resolution of the video stream in the set;

[0128] Step S203c, determining the computing power and transmission energy consumption of each camera and edge server per unit time;

[0129] Step S203d, determining the selective transmission strategy corresponding to each camera and the computing resource allocation strategy corresponding to the edge server according to the required computing amount, computing power per unit time, and transmission energy consumption.

[0130] In one implementation of this embodiment, a key step in solving the above optimal resource scheduling is how to define the utility function. Since the energy consumption of computing and transmission is an important factor in real-time scheduling, the utility function is defined as:

[0131]

[0132] The goal of this embodiment is to determine the optimal multi-feature stream selective transmission and edge server computing resource allocation strategy based on the current status of the camera and the server. This problem can be described as the following optimization model:

[0133]

[0134]

[0135]

[0136]

[0137]

[0138]

[0139] C6:r k,n >0.

[0140] Among them, C1 and C2 restrict the video transmission indicator to a binary variable, and a camera can only upload one type of video task. C3 restricts the frame rate of the video stream to not exceed the processing rate of the local and edge servers. C4 is the delay constraint of the video task. C5 and C6 limit the range of computing power allocated to the camera by the edge server.

[0141] Problem P1 is a non-convex mixed integer programming problem and is difficult to solve. The whale optimization algorithm based on mixed continuous and discrete variables can be used to solve this type of problem. The core idea of ​​the whale optimization algorithm is to imitate the hunting behavior of whales. One is to surround the prey; the other is to use a steam drum net. In each swim, the whales will randomly choose these two behaviors to hunt. These two hunting behaviors represent whether the whale swims towards the whale in the optimal position or randomly chooses a whale as its target and approaches it, which corresponds to the two methods of utilization and exploration in the algorithm idea.

[0142] In this algorithm, the whale's position vector represents the optimization variable The best whale position vector represents the optimal strategy set First, initialize the size of the whale population i={1,…,I}, each whale They all represent a set of initial solutions for different strategy sets. Set the maximum number of iterations I max , let iteration t = 1, and set the termination condition of the iteration.

[0143] Further, determining the size of the whale population Whether the initial solution represents the upper and lower bounds of the optimization variables, namely C2 and C6. If not, the upper and lower bounds are pulled back.

[0144] Furthermore, by introducing a penalty factor (which must be much greater than 0), the constraints C1, C3-C5 are uniformly transformed into a form of ≤0 and then multiplied by the penalty factor. The penalty factor is introduced into the objective function as a penalty term to obtain problem P2. The purpose of introducing the penalty term is to penalize the whale population size that does not meet the constraints.

[0145] Furthermore, the strategy set represented by each whale is brought into problem P2 to obtain the optimal objective function value. The corresponding strategy set is the optimal strategy in this iteration.

[0146] Further, update the coefficient vector and It can be obtained by the following formula:

[0147]

[0148]

[0149] in The value of decreases linearly from 2 to 0. is a random number in [0,1]. The main purpose of these two formulas is to balance development and utilization, which provides a random behavior for the position update of the population.

[0150] In the updating process, the position of the continuous variable r is first updated to generate a uniform random number p between [0,1]. If p < 0.5 and Then use the following formula to update and

[0151]

[0152]

[0153] The purpose of the above two formula updates is exploration.

[0154] If p < 0.5 and A ≥ 1, select a random whale position vector And update using the following formula and

[0155]

[0156]

[0157] The purpose of the above two formula updates is to utilize.

[0158] If p ≥ 0.5, update using the following formula and

[0159]

[0160]

[0161] The purpose of the above two formula updates is additional exploration.

[0162] Then, the position of the discrete variable x is updated to generate uniform random numbers p and p between [0,1]. BWOA .

[0163] If p < 0.5 and A < 1, use the following formulas to update σ sem ,

[0164]

[0165]

[0166]

[0167] Where C(·) represents the 1's complement operation. The purpose of updating the above three formulas is exploration.

[0168] If p < 0.5 and A ≥ 1, select a random whale position vector and update And update the step size σ using the following formula sp and

[0169]

[0170]

[0171] The purpose of the above three formula updates is to utilize.

[0172] If p ≥ 0.5, use the following formulas to update σ sem ,

[0173] renew

[0174]

[0175]

[0176] The purpose of these three formula updates is additional exploration.

[0177] Furthermore, if the maximum number of iterations is reached or the set iteration termination condition is reached, the optimal strategy set is output. That is, the algorithm finds the optimal strategy for selective transmission of multi-feature streams and computing resource allocation of edge servers; otherwise, it returns to continue determining the size of the whale population. Whether the initial solution represents the upper and lower bounds of the optimization variables.

[0178] Finally, after obtaining the optimal strategy, the roadside unit feeds back decision information to the camera and informs the camera of the type of video stream uploaded.

[0179] That is, in one implementation of this embodiment, step S203d includes the following steps:

[0180] Step S3d-1, establishing a selective transmission and computing resource allocation model based on the computing capabilities and transmission energy consumption of each camera and edge server;

[0181] Step S3d-2, solving the selective transmission and computing resource allocation model according to the whale optimization algorithm to obtain a corresponding strategy set;

[0182] Step S3d-3, introducing a penalty factor and determining an objective function according to the penalty factor;

[0183] Step S3d-4, optimizing the strategy set according to the objective function and the number of iterations to obtain an optimized selective transmission strategy and computing resource allocation strategy for the edge server;

[0184] Step S3d-5, feeding back decision information to each camera through multiple roadside units.

[0185] In this embodiment, by modeling the processing power of each camera and the computing resource allocation of the edge server, and optimizing and solving the optimal strategy through the whale optimization algorithm, the computing power of each camera and the allocated resources in the actual scenario can meet the task requirements, thereby adapting to the requirements of rapid tracking of the target.

[0186] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the traffic monitoring video stream processing method further includes the following steps:

[0187] Step S300, uploading video streams corresponding to preset task information in multiple cameras according to the determined strategy information, and performing online reasoning and analysis on the uploaded video streams in the edge server.

[0188] In this embodiment, after receiving the video task information fed back by the roadside unit, the camera uploads the video stream of the required category, and then performs online reasoning and analysis on the uploaded video stream in the edge server, so as to track the target vehicle according to the uploaded video stream.

[0189] Specifically, in an implementation of this embodiment, step S300 includes the following steps:

[0190] Step S301, uploading corresponding video streams on each camera according to the determined strategy information and preset task information;

[0191] Step S302, online reasoning and analysis of the uploaded video stream is performed through the edge server, and information is fed back to the corresponding camera after the target object is found;

[0192] Step S303, uploading the original image or video associated with the target object in the corresponding camera according to the feedback information;

[0193] Step S304: Upload the associated original image or video to the traffic control center for storage through the corresponding roadside unit.

[0194] In an actual scenario of this embodiment, the camera uploads the required category of video stream after receiving the video task information fed back by the roadside unit; the edge server integrated with the roadside unit performs online reasoning and analysis on the received video stream, and feeds back information to the corresponding camera after finding the target vehicle; the camera uploads the key original image or video related to the target vehicle, and the roadside unit uploads the acquired image or video to the traffic control center for storage.

[0195] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the traffic monitoring video stream processing method further includes the following steps:

[0196] Step S400 , tracking and predicting the target object in the preset task information in real time according to the online reasoning and analysis results, and outputting the tracking and prediction results.

[0197] In this embodiment, after the edge server detects the target vehicle, in order to achieve real-time tracking of the target vehicle, the path prediction model of the edge server is used to estimate the vehicle's driving trajectory in the future; the path prediction model used integrates the DS reasoning system (such as Figure 4 ) and four commonly used motion models (such as Figure 5The model is shown in Figure 1, which are the constant velocity motion (CV) model, the constant acceleration motion (CA) model, the constant yaw rate (CTR) model, and the constant yaw rate and acceleration (CTRA) model. This model can achieve accurate path prediction about 3-4 seconds in advance.

[0198] Specifically, in an implementation of this embodiment, step S400 includes the following steps:

[0199] Step S401, determining the acceleration and yaw rate of the target object according to the updated information table;

[0200] Step S402, selecting a corresponding path prediction model according to the acceleration and the yaw rate;

[0201] Step S403, predicting the location coordinates of the target object within a future preset time period by using the selected path prediction model;

[0202] Step S404: updating the future driving trajectory of the target object according to the predicted position coordinates.

[0203] The path prediction models include: a constant speed motion model, a constant acceleration motion model, a constant yaw rate model, and a constant yaw rate and acceleration model.

[0204] In this embodiment, the DS reasoning system first determines the most suitable motion model for the current vehicle based on the acceleration and yaw rate of the target vehicle obtained from the information table and the road curvature value extracted from the digital map, and then obtains the position coordinates of the future vehicle based on the determined motion model. The coordinate system defined in the path prediction model has an origin at the intersection of the road centerline and the left boundary of the road section, an X-axis pointing to the east, and a Y-axis pointing to the north.

[0205] The input state vector, i.e. the state space expression of the target vehicle V, is expressed as state V =[x,y,v x ,v y ,a t ,h,ω], where x and y are the horizontal and vertical coordinates of the target vehicle, respectively, and v x and v y are the speeds in the X-axis and Y-axis directions, a t is the tangential acceleration, h is the heading, and ω is the yaw rate.

[0206] For example, if V accelerates on a curve, the DS reasoning system will select the CTRA model and obtain the horizontal position coordinates of V at the next moment n+1 according to the following formula: q(n+1)=(x(n+1),y(n+1)).

[0207] x(n+1)=x(n)+[v x (n)·sin(ωτ)+vy (n)·(cos(ωτ)-1)] / ω+[a t ·(cos(h(n)+ωτ)-cos(h(n)))+a t ·ωτ·sin(h(n)+ωτ)] / ω 2 ,

[0208] y(n+1)=y(n)+[v y (n)·sin(ωτ)-v x (n)·(cos(ωτ)-1)] / ω+[a t ·(sin(h(n)+ωτ)-sin(h(n)))-a t ·ωτ·cos(h(n)+ωτ)] / ω 2 ,

[0209] v x (n+1)=v x (n)·cos(ωτ)-v y (n)·sin(ωτ)+a t ·τ·cos(h(n)+ωτ),

[0210] v y (n+1)=v y (n)·cos(ωτ)+v x (n)·sin(ωτ)+a t ·τ·sin(h(n)+ωτ),

[0211] h(n+1)=h(n)+ωτ.

[0212] The current position coordinates q(n) of the target vehicle, as well as information such as acceleration and yaw rate, are known values ​​transmitted to the roadside unit by the camera. Based on the path prediction model, the vehicle's relatively accurate position coordinates in the next 3-4 seconds can be obtained. Every time the time exceeds 3-4 seconds, the camera needs to re-upload the vehicle's position information, and the edge server will again use the path prediction model to update the vehicle's future driving trajectory in real time.

[0213] Specifically, in an implementation of this embodiment, step S400 further includes the following steps:

[0214] Step S405, predicting the monitoring area that the target object will enter in the future time period according to the future driving trajectory;

[0215] Step S406, if it is predicted that the target object will leave the current monitoring area in the future time period, the current monitoring camera is switched to the camera of the next area;

[0216] Step S407: Upload the associated video stream through the switched camera.

[0217] In this embodiment, based on the future driving trajectory of the vehicle obtained in real time, the roadside unit can predict which cameras will be under the jurisdiction of the target vehicle within 3-4 seconds. If the current target vehicle is about to leave the monitoring range of the camera of the current road section, the roadside unit sends information to the camera of the next road section according to the predicted vehicle driving trajectory, and the camera of the next road section continues to monitor and upload the corresponding video stream. Therefore, for a specific traffic safety monitoring task, once the roadside unit finds the target vehicle, it only needs to mobilize the corresponding camera to upload the video stream, and it does not need all cameras to upload the video stream in real time, which not only improves the bandwidth utilization, but also reduces the computing power demand on the edge server.

[0218] This embodiment achieves the following technical effects through the above technical solution:

[0219] 1. Different from the traditional traffic monitoring system driven by specific tasks, this embodiment proposes a traffic monitoring video stream processing method based on the end-edge-cloud system to address the challenges brought to the development of traffic monitoring by the large bandwidth and computing resource requirements for video stream transmission and processing. Efficient processing of monitoring video streams in the context of traffic safety is achieved through the mutual collaboration of various modules in the architecture. This embodiment proposes an integrated end-edge-cloud architecture for large-scale real-time monitoring video stream processing, providing accurate low-energy response and control services for existing traffic monitoring systems. Among them, the camera is responsible for video acquisition, preprocessing and uploading; the edge server is responsible for collaborative computing and management, and the central cloud is responsible for issuing tasks and storing videos.

[0220] 2. On the camera side, this embodiment considers heterogeneous terminal cameras including traditional cameras and future new retinal cameras. The three streams of video stream, feature stream, and semantic stream work together on the end-edge cloud platform. Among them, the compressed video stream is used for human vision, and the feature stream and semantic stream are used for machine vision to improve the performance of human / machine vision tasks. It is very necessary for the camera's multi-feature stream transmission to consider that the data prediction and decision-making modules analyze data only from features and semantics, rather than from the original entire video surveillance data. By sending only a small amount of feature data and semantic data to the server, the lack of bandwidth for transmitting the original video stream can be avoided. Since the amount of data is relatively small, it can be combined in real time in traffic safety for big data analysis and retrieval.

[0221] On the edge side, this embodiment deploys a real-time scheduling algorithm based on multi-feature stream transmission and a vehicle path prediction algorithm on the edge server. Among them, the real-time scheduling algorithm based on multi-feature stream transmission can realize the selective transmission of multi-feature streams and the allocation of computing resources of the edge server while minimizing the energy consumption of video stream transmission and processing; the path prediction algorithm can more accurately predict the driving trajectory of the vehicle in the future, so that the roadside unit can call the video stream of the corresponding camera with the least number, while ensuring the accuracy of target positioning. Using the above method, the matching strategy can be automatically updated when the system topology changes without manual intervention.

[0222] Exemplary Devices

[0223] Based on the above embodiments, the present invention further provides a traffic monitoring video stream processing device, whose principle block diagram can be shown as follows: Figure 6 shown.

[0224] The traffic monitoring video stream processing device includes: a processor, a memory, an interface, a display screen and a communication module connected through a system bus; wherein the processor of the traffic monitoring video stream processing device is used to provide computing and control capabilities; the memory of the traffic monitoring video stream processing device includes a storage medium and an internal memory; the storage medium stores an operating system and a computer program; the internal memory provides an environment for the operation of the operating system and the computer program in the storage medium; the interface is used to connect to external devices, such as mobile terminals and computers; the display screen is used to display corresponding traffic monitoring video stream processing information; and the communication module is used to communicate with a cloud server or a mobile terminal.

[0225] When the computer program is executed by a processor, it is used to implement a traffic monitoring video stream processing method.

[0226] It can be understood by those skilled in the art that Figure 6 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the traffic monitoring video stream processing device to which the scheme of the present invention is applied. The specific traffic monitoring video stream processing device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0227] In one embodiment, a traffic monitoring video stream processing device is provided, which includes: a processor and a memory, the memory stores a traffic monitoring video stream processing program, and the traffic monitoring video stream processing program is used to implement the above traffic monitoring video stream processing method when executed by the processor.

[0228] Exemplary Systems

[0229] Based on the above embodiments, the present invention also provides a traffic monitoring video stream processing system. Figure 2 As shown, including:

[0230] multiple roadside units;

[0231] Multiple cameras;

[0232] Edge servers;

[0233] Traffic Control Center;

[0234] And the traffic monitoring video stream processing device as described in the above embodiment;

[0235] The traffic monitoring video stream processing device is used to perform the following operations:

[0236] Establishing communication connections between a plurality of roadside units in a preset area and a plurality of cameras, and updating information tables in the plurality of roadside units;

[0237] Determine a selective transmission strategy and a computing resource allocation strategy for multiple features in the edge server according to the updated information table and the real-time scheduling algorithm for multiple feature stream transmission;

[0238] Upload video streams corresponding to preset task information in multiple cameras according to the determined strategy information, and perform online reasoning and analysis on the uploaded video streams in the edge server;

[0239] Based on the online reasoning and analysis results, the target objects in the preset task information are tracked and predicted in real time, and the tracking and prediction results are output.

[0240] In one embodiment, a storage medium is provided, wherein the storage medium stores a traffic monitoring video stream processing program, and the traffic monitoring video stream processing program is used to implement the above traffic monitoring video stream processing method when executed by a processor.

[0241] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory.

[0242] In summary, the present invention provides a method, device, system and storage medium for processing traffic monitoring video streams, the method comprising: establishing communication connections between multiple roadside units in a preset area and multiple cameras, and updating information tables in multiple roadside units; determining the selective transmission strategy and computing resource allocation strategy of multiple features in the edge server according to the updated information table and the real-time scheduling algorithm for multi-feature stream transmission; uploading video streams corresponding to preset task information in multiple cameras according to the determined strategy information, and performing online reasoning and analysis on the uploaded video streams in the edge server; performing real-time tracking and prediction of target objects in the preset task information according to the online reasoning and analysis results, and outputting the tracking and prediction results. The present invention proposes an integrated end-edge-cloud architecture for large-scale real-time monitoring video stream processing, providing accurate low-energy response and control services for existing traffic monitoring systems.

[0243] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for processing a traffic monitoring video stream, characterized in that: The traffic monitoring video stream processing method comprises: Establishing communication connections between a plurality of roadside units in a preset area and a plurality of cameras, and updating information tables in the plurality of roadside units; Determine a selective transmission strategy and a computing resource allocation strategy for multiple features in the edge server according to the updated information table and the real-time scheduling algorithm for multiple feature stream transmission; Uploading video streams corresponding to the preset task information in the plurality of cameras according to the determined strategy information, and performing online reasoning and analysis on the uploaded video streams in the edge server; According to the online reasoning and analysis results, the target object in the preset task information is tracked and predicted in real time, and the tracking and prediction results are output; The method of determining the selective transmission strategy and computing resource allocation strategy of multiple features in the edge server according to the updated information table and the real-time scheduling algorithm for the transmission of multiple feature streams includes: Extracting and compressing a feature stream in real time from a raw image signal in a retinal camera, or extracting and compressing a semantic stream in real time from a raw video signal in a retinal camera; Deploying a real-time scheduling algorithm for the multi-feature stream transmission in the edge server; Determining the selective transmission strategy of the multiple features and the computing resource allocation strategy in the edge server using the updated information table and the real-time scheduling algorithm; The method of using the updated information table and the real-time scheduling algorithm to determine the selective transmission strategy of the multiple features and the computing resource allocation strategy in the edge server includes: Acquire a set of video streams or images within the preset area; Determining the amount of computation required for image processing according to the images in the set, or determining the amount of computation required for performing compression coding, feature extraction, and semantic segmentation according to the frame rate and resolution of the video stream in the set; Determine the computing power and transmission energy consumption of each camera and the edge server per unit time; According to the required amount of calculation, the computing power per unit time, and the transmission energy consumption, the selective transmission strategy corresponding to each camera and the computing resource allocation strategy corresponding to the edge server are determined.

2. The traffic monitoring video stream processing method according to claim 1, characterized in that: The step of establishing communication connections between a plurality of roadside units in a preset area and a plurality of cameras comprises: Acquire preset task information, and send the preset task information to multiple roadside units in the preset area through a traffic control center.

3. The method for processing traffic monitoring video stream according to claim 1, characterized in that: The step of establishing communication connections between a plurality of roadside units in a preset area and a plurality of cameras and updating information tables in the plurality of roadside units includes: Establishing communication connections between the plurality of roadside units and the plurality of cameras according to a video transmission protocol, and initializing communication states of the plurality of cameras; Controlling each camera to broadcast the current device status to the plurality of roadside units, and updating the corresponding information table in the plurality of roadside units; The current device status includes: the computing resources required by each camera, the size of the video data, the percentage of used resources, and the location coordinates of the monitored object.

4. The method for processing traffic monitoring video stream according to claim 1, characterized in that: The real-time scheduling algorithm for deploying the multi-feature stream transmission in the edge server also includes: Images and videos are acquired from the single-function camera, compressed and processed, and transmitted to the plurality of roadside units.

5. The method for processing traffic monitoring video stream according to claim 1, characterized in that: The determining of the selective transmission strategy corresponding to each camera and the computing resource allocation strategy corresponding to the edge server includes: Establishing a selective transmission and computing resource allocation model based on the computing capabilities and transmission energy consumption of each camera and the edge server; Solving the selective transmission and computing resource allocation model according to the whale optimization algorithm to obtain a corresponding strategy set; Introducing a penalty factor, and determining an objective function based on the penalty factor; Optimizing the strategy set according to the objective function and the number of iterations to obtain an optimized selective transmission strategy and a computing resource allocation strategy for the edge server; The decision information is fed back to each camera through the plurality of roadside units.

6. The method for processing traffic monitoring video stream according to claim 1, characterized in that: Uploading video streams corresponding to the preset task information in the plurality of cameras according to the determined strategy information, and performing online reasoning and analysis on the uploaded video streams in the edge server, including: Uploading corresponding video streams on each camera according to the determined strategy information and the preset task information; The edge server performs online reasoning and analysis on the uploaded video stream, and feeds back information to the corresponding camera after the target object is found; Uploading the original image or video associated with the target object in the corresponding camera according to the feedback information; The associated original image or video is uploaded to the traffic control center for storage through the corresponding roadside unit.

7. The method for processing traffic monitoring video stream according to claim 1, characterized in that: The real-time tracking and prediction of the target object in the preset task information according to the online reasoning and analysis results includes: determining the acceleration and yaw rate of the target object according to the updated information table; Selecting a corresponding path prediction model according to the acceleration and the yaw rate; Predicting the location coordinates of the target object within a preset time period in the future by using the selected path prediction model; Updating the future driving trajectory of the target object according to the predicted position coordinates; The path prediction model includes: a constant speed motion model, a constant acceleration motion model, a constant yaw rate model, and a constant yaw rate and acceleration model.

8. The method for processing traffic monitoring video stream according to claim 7, characterized in that: The real-time tracking and prediction of the target object in the preset task information according to the online reasoning and analysis results also includes: Predicting the monitoring area that the target object will enter within the future preset time period according to the future driving trajectory; If it is predicted that the target object is about to leave the current monitoring area within the future preset time period, the current monitoring camera is switched to the camera of the next area; Upload the associated video stream through the switched camera.

9. A traffic monitoring video stream processing device, characterized in that: include: A processor and a memory, wherein the memory stores a traffic monitoring video stream processing program, and when the traffic monitoring video stream processing program is executed by the processor, it is used to implement the traffic monitoring video stream processing method as described in any one of claims 1-8.

10. A traffic monitoring video stream processing system, characterized in that: include: multiple roadside units; Multiple cameras; Edge servers; Traffic Control Center; And the traffic monitoring video stream processing device as claimed in claim 9; The traffic monitoring video stream processing device is used to perform the following operations: Establishing communication connections between a plurality of roadside units in a preset area and a plurality of cameras, and updating information tables in the plurality of roadside units; Determining a selective transmission strategy and a computing resource allocation strategy for multiple features in the edge server according to the updated information table and the real-time scheduling algorithm for multiple feature stream transmission; Uploading video streams corresponding to the preset task information in the plurality of cameras according to the determined strategy information, and performing online reasoning and analysis on the uploaded video streams in the edge server; The target object in the preset task information is tracked and predicted in real time according to the online reasoning and analysis results, and the tracking and prediction results are output.

11. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and the storage medium stores a traffic monitoring video stream processing program. When the traffic monitoring video stream processing program is executed by a processor, it is used to implement the traffic monitoring video stream processing method according to any one of claims 1 to 8.

Images