Construction method of intersection traffic control model and related device and application

Abstract

The application relates to a construction method of a road intersection traffic control model and related devices and applications, wherein the method comprises the following steps: a traffic police command picture is intercepted and input into a first neural network model, a field command instruction set is extracted by using a trained command instruction extraction model, field signal lamp control instruction sets are obtained by collecting signal lamp control data, two instruction sets at the same time are collected to obtain a road intersection traffic control instruction set; a multi-path road intersection panoramic video is acquired, and a plurality of road intersection real scene pictures at the same time are intercepted from the multi-path road intersection panoramic video to obtain a road intersection real scene image set at the same time; the road intersection real scene image set at the same time and the road intersection traffic control instruction set are input into a second neural network model, and a road intersection traffic control model is trained. The application uses a neural network model, which is an end-to-end learning method. By learning a large amount of command videos and signal lamp control data, the traffic police's traffic control experience is automatically learned, and then the road intersection traffic is controlled in real time according to the learning result.

Description

Technical Field

[0001] This application relates to the fields of urban traffic management and video analysis, specifically to a method for constructing an intersection traffic control model based on end-to-end visual recognition, as well as related devices and applications. Background Technology

[0002] In recent years, with the continuous increase in the number of vehicles in Chinese cities and the continuous expansion of the overall scale of urban transportation systems, traffic congestion in major and medium-sized cities has become increasingly serious. Observation and analysis have revealed that intersections are key areas of traffic congestion in urban roads and are also crucial to solving the problem.

[0003] The core of urban road traffic optimization lies in the rational allocation of right-of-way to meet the traffic demand in all directions, thereby maximizing the use of time and space resources. Past research on urban road traffic optimization has mainly focused on collecting traffic data, building optimization models, performing resource allocation calculations, and optimizing traffic signal systems. Taking the building of optimization models as an example, although a great deal of technical research has been conducted, the complexity, variability, and real-time nature of urban traffic systems, coupled with the significant simplifications made during modeling that fail to accurately and in real-time reflect the actual traffic conditions at intersections, make this approach insufficient to completely solve congestion problems at intersections. Furthermore, the aforementioned methods of collecting traffic data and performing resource allocation calculations require large-scale, high-investment implementation, making these solutions inefficient and inconvenient to implement.

[0004] Therefore, in actual traffic management, the most common method is still to resolve traffic congestion at intersections through on-site traffic police directing traffic. Although this method relies entirely on the personal experience and understanding of traffic rules of the traffic police on site and cannot be automated or intelligent, it can still be considered an efficient and indispensable solution in actual intersection command, because traffic police can make quick decisions based on the actual situation on site without going through complex data processing and modeling processes.

[0005] In conclusion, if a novel traffic control model for intersection areas can be proposed, utilizing neural network algorithms to enable the model to comprehensively learn and simulate traffic police command strategies in the on-site environment, ultimately achieving real-time traffic control at intersections, then it will undoubtedly provide a more robust and comprehensive technical path for solving urban traffic congestion problems. Summary of the Invention

[0006] To improve existing urban road traffic optimization schemes, this application provides a method for constructing an intersection traffic control model based on end-to-end visual recognition, as well as related devices and applications. The scheme utilizes a neural network model to automatically learn traffic police's experience in intersection traffic control by learning from a large amount of traffic police command videos and traffic light control data. Then, based on the learning results, it directs intersection traffic in real time.

[0007] Firstly, this application provides a method for constructing an intersection traffic control model, using the technical solution described below.

[0008] A method for constructing an intersection traffic control model includes the following steps:

[0009] Traffic police command images are extracted from traffic police command videos containing traffic police hand gestures. These images are then used as training samples and input into a first neural network model to train a command instruction extraction model. The command instruction extraction model is then used to extract on-site command gestures from the on-site command videos. The extraction results are then summarized to obtain an on-site command instruction set. The control data of the traffic lights at the intersection area are summarized to obtain an on-site traffic light control instruction set. Based on the time interval, the on-site command instruction set and the on-site traffic light control instruction set at the same time interval are summarized to obtain the intersection traffic command instruction set at that time interval.

[0010] A panoramic video of the intersection containing information from multiple video sources is acquired. The timelines of the panoramic video and the on-site traffic control video correspond to each other. Multiple real-scene images of the intersection from different video sources at the same moment are extracted from the panoramic video and summarized to obtain a set of real-scene images of the intersection at that moment. The set of real-scene images of the intersection at the same moment and the set of traffic control instructions at the intersection are used as input data training samples and output data training samples, respectively, and then input into a second neural network model to train the intersection traffic control model.

[0011] By adopting the above technical solution, an anthropomorphic modeling approach is realized. The intersection traffic command model constructed by the method takes the recorded images of cameras widely deployed in the real environment as input and the intersection traffic command instructions for the real environment as output. Through neural network algorithms, it comprehensively learns and simulates the on-site command strategies of real traffic police.

[0012] Furthermore, since the entire command model's interaction process is end-to-end, it does not require large-scale data collection during its training and application, nor does it involve complex traffic models or complex traffic optimization algorithms. Traffic command instructions can be obtained directly from the input image data. The entire process is not only simple and efficient, but also has strong real-time performance and adaptability.

[0013] Preferably, the step of extracting traffic police command images from traffic police command videos containing traffic police hand gestures, and using these images as training samples to input into a first neural network model to train a command instruction extraction model, specifically includes the following steps:

[0014] The process involves acquiring a traffic police command video containing the traffic police command gestures, extracting multiple traffic police command images from the video, with each image corresponding to a specific traffic police command gesture, labeling each image according to the type of gesture, matching all images with the corresponding gesture labels, summarizing all images to obtain the original dataset for the extraction model, and dividing the original dataset into a training set and a validation set according to a preset ratio.

[0015] The extraction model training set is used as training samples and input into the first neural network model. The first neural network model performs data augmentation processing on the traffic police command images in the extraction model training set and uses the data augmentation processing results for model training. After the model training is completed, the initial model for extracting command instructions is obtained.

[0016] The extraction model validation set is used as validation samples and input into the initial model for extracting command instructions. After validation and adjustment, the command instruction extraction model is obtained.

[0017] By adopting the above technical solution, the modeling process for the command instruction extraction model in the method is clarified. As a tool for extracting on-site command gestures in subsequent processes, the establishment of the command instruction extraction model provides basic technical support for the subsequent processing of the method.

[0018] Preferably, the gesture tag information includes: going straight, turning left, waiting to turn left, turning right, changing lanes, slowing down, pulling over, and stopping.

[0019] By adopting the above technical solution, the problem of extracting on-site traffic control gestures is simplified, providing key information about the traffic control gestures for the final intersection traffic control model. In essence, it reflects the classification idea in dealing with this problem and provides a classification example for other related research in this field.

[0020] Preferably, the step of extracting on-site command gestures from the on-site command video using the command instruction extraction model, and summarizing the extraction results to obtain an on-site command instruction set, specifically includes the following steps:

[0021] The on-site command video is acquired, and the on-site command video is divided into multiple on-site command video screenshots according to time. The on-site command gestures are extracted from each of the on-site command video screenshots using the command instruction extraction model. The extracted on-site command gestures are assigned gesture label information according to the type of traffic police command gestures they correspond to.

[0022] The duration of each on-site command gesture is determined by comparing whether the gesture tag information corresponding to the on-site command gestures is consistent across multiple consecutive moments.

[0023] The gesture label information and duration corresponding to all the on-site command gestures are summarized to obtain the on-site command instruction set.

[0024] By adopting the above technical solution, the process of acquiring the on-site command instruction set in the method is clarified, and the actual application process of the command instruction extraction model described above is also specifically explained.

[0025] Preferably, the step of summarizing the control data of the traffic lights at the intersection area to obtain the on-site traffic light control command set specifically includes the following steps:

[0026] Read the traffic light control data of the intersection signal controller in the intersection area. The time period of the control data corresponds to the time axis of the on-site command video. The control data includes the traffic light control strategy at any time and the duration of each traffic light control strategy. Summarize all the control data to obtain the on-site traffic light control instruction set.

[0027] By adopting the above technical solution, the process of acquiring the on-site traffic light control instruction set in the method is clarified. At the same time, it is stipulated here that the control data used to generate the on-site traffic light control instruction set is matched with the on-site command video in time. This also ensures the smooth progress of the subsequent intersection traffic command model training and enables the model to accurately associate on-site command gestures with on-site traffic light control instructions.

[0028] Preferably, the step of inputting the set of real-scene images of the intersection and the set of traffic control instructions at the same time as input data training samples and output data training samples into the second neural network model to train the intersection traffic control model specifically includes the following steps:

[0029] Each real-scene image of the intersection in the real-scene image set is segmented into blocks and embedded into a processing sequence. The processing sequence is input into the second neural network model to obtain the feature representation of the real-scene image of the intersection. The feature representation of the real-scene image of the intersection is then subjected to pooling processing to obtain the pooled features of the real-scene image of the intersection.

[0030] Based on the features of the intersection real-scene image, the traffic command instruction set of the intersection is processed using the second neural network model to obtain the gesture features and traffic light duration features in the intersection real-scene image respectively.

[0031] An objective function for command loss is constructed, and the result of the objective function construction is obtained. The second neural network model is trained and optimized using the gesture features, the traffic light duration features, and the result of the objective function construction to obtain the intersection traffic control model.

[0032] By adopting the above technical solution, the training and optimization process of the intersection traffic control model in the method is clarified, ensuring the operational efficiency and accuracy of the final output intersection traffic control model. Furthermore, because the input and output of the intersection traffic control model are significantly simplified, the usage threshold of the model is effectively lowered, and the applicability of the model is expanded, making the final output intersection traffic control model more universally applicable compared to various data-based control models in the prior art.

[0033] Secondly, this application provides a device for constructing a traffic control model at an intersection, which adopts the following technical solution.

[0034] A device for constructing a traffic control model at an intersection includes the following modules:

[0035] The command instruction set generation module is configured to extract traffic police command images from traffic police command videos containing traffic police command gestures, input the traffic police command images as training samples into a first neural network model to train a command instruction extraction model, use the command instruction extraction model to extract on-site command gestures from the on-site command video, summarize the extraction results to obtain an on-site command instruction set, summarize the control data of the traffic lights of the intersection signal controllers to obtain an on-site traffic light control instruction set, and summarize the on-site command instruction set and the on-site traffic light control instruction set at the same time based on the time of day to obtain the intersection traffic command instruction set at that time.

[0036] The traffic command model training module is configured to acquire a panoramic video of an intersection containing information from multiple video sources. The timelines of the panoramic video and the on-site traffic command video correspond. Multiple real-scene images of the intersection from different video sources at the same moment are extracted from the panoramic video and summarized to obtain a set of real-scene images of the intersection at that moment. The set of real-scene images of the intersection at the same moment and the set of traffic command instructions at the intersection are used as input data training samples and output data training samples, respectively, and then input into the second neural network model to train the intersection traffic command model.

[0037] By adopting the above technical solution, not only are the input and output of the intersection traffic command model greatly simplified, but also the complex traffic engineering modeling and calculation are avoided, the real-time performance of the model is guaranteed, and the model can truly simulate the on-site command of traffic police, and truly realize the intelligent command and guidance of traffic in the intersection area.

[0038] Thirdly, this application provides a smart terminal, which adopts the following technical solution:

[0039] A smart terminal includes a memory and a processor. The memory stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the method for constructing an intersection traffic control model as described above.

[0040] Fourthly, this application provides a computer-readable storage medium, which adopts the following technical solution:

[0041] A computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the method for constructing an intersection traffic control model as described above.

[0042] Fifthly, this application provides an application of an intersection traffic control model, which adopts the following technical solution.

[0043] A method for traffic control at intersections, using an intersection traffic control model constructed using the method described above, includes the following steps:

[0044] The system acquires real-time panoramic video of an intersection containing information from multiple video sources, divides the real-time panoramic video of the intersection into multiple real-time panoramic images of the intersection according to time intervals, inputs all the real-time panoramic images of the intersection into the intersection traffic control model, and obtains real-time traffic control instructions fed back by the intersection traffic control model.

[0045] In summary, this application includes at least one of the following beneficial technical effects:

[0046] 1. The solution in this application constructs a model using an anthropomorphic approach. The intersection traffic control model constructed by the method takes the recorded images from cameras widely deployed in the real environment as input and the intersection traffic control instructions tailored to the real environment as output. It comprehensively learns and simulates the on-site command strategies of real traffic police through a neural network algorithm. This intersection traffic control model can not only improve the traffic efficiency of intersection areas and reduce congestion, but also help city managers achieve more intelligent and efficient urban traffic management, thereby improving people's travel experience and enhancing the sustainability of urban development.

[0047] 2. The solution presented in this application provides end-to-end interaction with the intersection traffic control model. Therefore, its training and application do not require large-scale data collection, nor does it involve complex traffic models or complex traffic optimization algorithms. Traffic control instructions can be obtained directly from the input image data. The entire process is not only simple and efficient but also highly real-time. Furthermore, the intersection traffic control model can be extended and applied to various scenarios of urban traffic management, demonstrating strong universality and practicality.

[0048] 3. The intersection traffic command model constructed by the scheme of this application is based on a neural network model, and therefore has the characteristics of a neural network model. That is, the model can be continuously trained with data to gradually improve the efficiency and accuracy of generating real-time traffic command instructions, so that the intersection traffic command model is always in the process of optimization and iteration, and can be applied to the field of urban traffic management for a long time. Attached Figure Description

[0049] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained by referring to these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating the method for constructing an intersection traffic control model according to an embodiment of this application;

[0051] Figure 2 This is a flowchart of part S11 of the method for constructing an intersection traffic control model according to an embodiment of this application;

[0052] Figure 3 This is a schematic diagram illustrating the traffic police hand gestures in the method for constructing an intersection traffic control model according to an embodiment of this application.

[0053] Figure 4This is a flowchart of part S12 of the method for constructing an intersection traffic control model according to an embodiment of this application;

[0054] Figure 5 This is a flowchart of part S22 of the method for constructing an intersection traffic control model according to an embodiment of this application;

[0055] Figure 6 This is a schematic diagram of the architecture of the intersection traffic control model construction device according to an embodiment of this application. Detailed Implementation

[0056] This application provides a method, related apparatus, and application for constructing an intersection traffic control model based on end-to-end visual recognition. It simulates the traffic police's command strategies in a real-time environment, ultimately achieving real-time traffic control at intersections. To make the objectives, technical solutions, and advantages of this application clearer, the implementation methods of this application will be further described in detail below.

[0057] First, this application discloses a method for constructing an intersection traffic control model, such as... Figure 1 As shown, the specific steps include the following:

[0058] S1. Extract traffic police command images from traffic police command videos containing traffic police hand gestures, input the traffic police command images as training samples into the first neural network model to train a command instruction extraction model, use the command instruction extraction model to extract on-site command gestures from the on-site command video, summarize the extraction results to obtain an on-site command instruction set, summarize the control data of the traffic lights at the intersection area to obtain an on-site traffic light control instruction set, and summarize the on-site command instruction set and the on-site traffic light control instruction set at the same time according to the time division to obtain the intersection traffic command instruction set at that time.

[0059] In the above steps, the traffic police command images are extracted from the traffic police command video containing traffic police hand gestures, and these images are used as training samples to input into the first neural network model to train the command instruction extraction model. Figure 2 As shown, this part can be broken down into the following process:

[0060] S111. Obtain the traffic police command video containing the traffic police command gestures, extract multiple traffic police command images from the traffic police command video, each traffic police command image corresponds to one type of traffic police command gesture, label each traffic police command image according to the type of the corresponding traffic police command gesture, so that each traffic police command image corresponds to a gesture label information, match all the traffic police command images with the gesture label information and summarize them to obtain the original dataset of the extraction model, and divide the original dataset of the extraction model into the extraction model training set and the extraction model validation set according to a preset division ratio.

[0061] To simplify the collection of traffic hand signals, this application categorizes traffic police hand signals at intersections into eight types, with the following hand signal labels: 1. Go straight; 2. Turn left; 3. Waiting to turn left; 4. Turn right; 5. Change lanes; 6. Slow down; 7. Pull over; 8. Stop. Specific illustrations are provided. Figure 3 As shown.

[0062] In one possible implementation, the labeling of the gesture information is performed manually. This step aims to establish a preliminary training dataset. Since these traffic police command images correspond to different command actions and intentions of traffic police, this labeling process is indispensable, as it can provide key information about the command gestures for subsequent models.

[0063] S112. The extracted model training set is used as training samples and input into the first neural network model. The first neural network model performs data augmentation processing on the traffic police command images in the extracted model training set and uses the data augmentation processing results for model training. After the model training is completed, the initial model for extracting command instructions is obtained.

[0064] S113. The extraction model validation set is used as a validation sample and input into the initial model for extracting command instructions. After validation and adjustment, the command instruction extraction model is obtained.

[0065] In one possible implementation, the first neural network model is preferably the YOLOv5 model, which has a relatively mature application technology. In this case, the specific operational details of the above steps S111 to S113 are as follows:

[0066] Multiple images of traffic police commands were captured from the traffic police command video containing the aforementioned traffic police hand gestures. Each image was labeled with a tagging tool: {(1) go straight, (2) turn left, (3) wait to turn left, (4) turn right, (5) change lanes, (6) slow down, (7) pull over, (8) stop}. The tag file was in XML format, containing the coordinates and classification information of the tags. Each traffic police command image corresponds one-to-one with the labeled gesture information. All used image files and tag files are collectively referred to as the original dataset of the extraction model.

[0067] The original dataset for the extracted model is divided at a ratio of 0.8, meaning 80% of the data is used for model training (the training set) and the remaining data is used for validation (the validation set). During the dataset division, the label files are converted from XML to TXT format for compatibility with the YOLO algorithm, while retaining all relevant information within the label files.

[0068] Configure the file path of the dataset in the configuration file. The dataset contains the number of categories and the name of each category. This is how the dataset is imported into the algorithm.

[0069] After the above processing, the dataset is then processed by the YOLOv5 algorithm. Images from the training set are input into the neural network in batches, with the number of images in each batch controlled by the parameter `batch_size`, which is set to 16. The training epochs are set to 500. The Adam (Adaptive Moment Estimation) optimizer is used, and the pre-trained model is YOLOv5s, which meets the response speed requirements of the application scenario. Model training involves tuning and fitting based on this pre-trained model. The model is trained using a GPU.

[0070] After the images are given to the model, the model performs data augmentation on them. The data augmentation methods used include: Mosaic, which combines four images into one; randomized affine transformations (scaling and translation); randomized adjustments to chroma, saturation, and brightness; and randomized horizontal flipping. These data augmentation methods are randomly applied to the input images.

[0071] The input original images will be uniformly scaled to the same size, maintaining the original aspect ratio during the scaling process, and filling the blank areas after scaling with black borders.

[0072] The images processed as described above can then be used for model training. The images are first divided into several grid cells of equal size. The YOLOv5 algorithm uses predefined anchor boxes of varying widths and heights for different datasets, with each grid cell corresponding to a set of anchor boxes. The model selects the anchor box that best fits the ground truth to fit the ground truth value. The offset of the anchor box within its grid cell, along with information such as the class and confidence level, constitutes the model's actual predicted output.

[0073] During model training, the neural network extracts features from the training data and outputs predicted boxes based on the initial anchor boxes. The predicted boxes are then compared with the ground truth boxes (i.e., the baseline values), and the difference between them is calculated and backpropagated to iterate the network parameters. YOLOv5 can adaptively calculate the optimal anchor box values ​​from the training set.

[0074] The difference between the predicted bounding box and the ground truth bounding box is calculated as the bounding box loss function. YOLOv5 uses CIOU_LOSS as the bounding box loss function.

[0075]

[0076] Where IOU is the intersection of the predicted bounding box and the ground truth bounding box divided by their union. gt ρ is the center point of the predicted box and the ground truth box, ρ is the Euclidean distance between the two center points, c is the diagonal length of the smallest bounding rectangle that can contain both boxes, a is the weight parameter, and v is used to measure the consistency of the aspect ratio.

[0077] CIOU_LOSS is only a part of the overall loss function. The overall loss function for YOLOv5 is:

[0078] LOSS=λ1L cls +λ2L obj +λ3L loc ,

[0079] Among them, L cls For classification loss, BCE loss is used, calculating only the classification loss for positive samples; L obj For the loss of obj, we also use BCE loss, but calculate the loss of obj for all samples; L loc The location loss is CIOU_LOSS.

[0080] The model network parameters are adjusted through backpropagation based on the loss function. After the model has reached the set number of training epochs (500), the weights of the last and optimal epochs are saved and exported, completing the model training process. If training is interrupted, the resume parameter can be used to continue training from the breakpoint.

[0081] Once the model has been trained, it can be used for object detection.

[0082] After deploying the model, the video to be detected is first segmented frame by frame into several images. These images are then scaled and fed into a neural network for forward propagation to extract features. The model detects objects at various locations within the images and extracts features at different convolutional layers. These features include the object's location, category information, and bounding box parameters.

[0083] The model then generates a series of prediction boxes based on the anchor boxes. Each prediction box includes a category label, a confidence score, and the coordinates of the bounding box.

[0084] During the detection process, the model generates many predicted boxes. These predicted boxes are first filtered based on a confidence threshold. The confidence threshold conf_thres is set to 0.25, and predicted boxes with a confidence score less than 0.25 will be removed.

[0085] When multiple predicted bounding boxes may appear for the same detected target, the non-maximum suppression algorithm is applied to remove boxes with high overlap and retain only the most representative box.

[0086] The final model will output a set of detected objects, each represented by a label (containing category information) and a bounding box (containing location information).

[0087] After obtaining the model's detection results, the bounding boxes are converted into coordinates on the image, the actual position of the object is calculated, and the detection results are visualized on the input image to display the detected object. The image with the predicted information is then saved as a video file at the specified frame rate, thus completing one video object recognition task.

[0088] In the above steps, the command instruction extraction model is used to extract on-site command gestures from the on-site command video, and the extraction results are summarized to obtain the on-site command instruction set, such as... Figure 4 As shown, this part can be broken down into the following process:

[0089] S121. Obtain the on-site command video, divide the on-site command video into multiple on-site command video screenshots according to time, use the command instruction extraction model to extract on-site command gestures for each of the on-site command video screenshots, and assign gesture label information to the extracted on-site command gestures according to the corresponding type of traffic police command gestures.

[0090] S122. By comparing whether the gesture label information corresponding to the on-site command gestures in multiple consecutive moments is consistent, the duration of each on-site command gesture is determined.

[0091] S123. Summarize the gesture label information and duration corresponding to all the on-site command gestures to obtain the on-site command instruction set.

[0092] In one possible implementation, the specific operational details of the above steps S121 to S123 are as follows:

[0093] The command instruction extraction model extracts the on-site command gestures of traffic police officers from the on-site command video. The input data is a screenshot of the on-site command video at time t {P}. t After processing by the command instruction extraction model, the on-site command gesture {Q} at time t is obtained. t}, that is, Q t The value is assigned to the recognized gesture label information.

[0094] Determine whether the identified gesture label information has changed. If it has not changed, increment timer t by 1 and repeat the above steps. If the identified gesture label information has changed, re-evaluate Q. t The new gesture label information is assigned.

[0095] Repeat the two steps described above until the on-site command instruction set {Q} is obtained. t T q}, Q t This represents the classification of command gestures at time t, where T is the time of command. q This indicates the duration corresponding to the command gesture category.

[0096] In the above steps, the step of summarizing the control data of the traffic lights at the intersection area to obtain the on-site traffic light control command set can be further broken down into the following process:

[0097] S131. Read the traffic signal control data of the intersection signal controller in the intersection area. The time period of the control data corresponds to the time axis of the on-site command video. The control data includes the traffic light control strategy at any time and the duration of each traffic light control strategy. The traffic light control strategy includes at least straight, left turn and right turn. Summarize all the control data to obtain the on-site traffic light control instruction set.

[0098] In one possible implementation, the control data of the traffic lights at the intersection area can be read manually or by using a software interface to read the timing control data of the traffic lights. The obtained data should include the control mode and duration of the traffic lights at time t, which can be denoted as {K}. t T k}, K t T represents the traffic light control strategy at time t. k This indicates the duration corresponding to the control strategy k.

[0099] The intersection real-scene image set consists of two parts, namely the {Q} mentioned above. t T q} and {K t T k For ease of description, the set of traffic control instructions for the intersection obtained in the preceding steps can be denoted as <{Q}. t T q},{K t T k Their corresponding input is the image sequence {P} input at time t. t}

[0100] S2. Obtain a panoramic video of the intersection containing information from multiple video sources. The timelines of the panoramic video and the on-site command video correspond. Extract multiple real-scene images of the intersection from different video sources at the same time from the panoramic video and summarize them to obtain a set of real-scene images of the intersection at that time. Input the set of real-scene images of the intersection at the same time and the set of traffic command instructions at the intersection as input data training samples and output data training samples, respectively, into the second neural network model to train the intersection traffic command model.

[0101] In one possible implementation, the panoramic video of the intersection includes multiple video sources. Images are extracted from the multiple video sources at the same time point to form an input image sequence set. This represents the image captured from the i-th video stream at time t. The output data is the traffic control instruction set <{Q> obtained in step one. t T q},{K t T kThe input and output datasets described above are used as sample data to train the second neural network model.

[0102] In the above steps, the set of real-scene images of the intersection and the set of traffic control instructions at the same time are respectively used as input data training samples and output data training samples to be input into the second neural network model to train the intersection traffic control model, as follows. Figure 5 As shown, this part can be broken down into the following process:

[0103] S221. Each real-scene image of the intersection in the set of real-scene images is segmented into blocks and embedded into a processing sequence. The processing sequence is input into the second neural network model to obtain the feature representation of the real-scene image of the intersection. The feature representation of the real-scene image of the intersection is subjected to pooling processing to obtain the pooled features of the real-scene image of the intersection.

[0104] S222. Based on the features of the intersection real-scene image, the traffic command instruction set of the intersection is processed using the second neural network model to obtain the gesture features in the intersection real-scene image and the signal light duration features in the intersection real-scene image.

[0105] S223. Construct an objective function for the instruction loss, obtain the objective function construction result, and use the gesture features, the traffic light duration features, and the objective function construction result to train and optimize the second neural network model to obtain the intersection traffic control model.

[0106] In one possible implementation, the specific operational details of steps S221 to S223 above are as follows:

[0107] The second neural network model is preferably the leading Swin Transformer model in the vision field, which is used as the base training model. The model is then trained using the aforementioned input-output training sample set. Specifically, this patent uses the Swin Transformer as the backbone for feature learning. This choice is based on the fact that the Swin Transformer employs a better scaling strategy compared to the previous ViT (VisionTransformer) model. The ViT model consists of multiple stacked transformer blocks, each block including two main components: Multi-Head Self-Attention (MSA) and a Feedforward Network (FFN). In this patent, the panoramic video of the traffic intersection is first decomposed into multiple individual image frames. Then, the input panoramic image of the traffic intersection is segmented into patches, and these patches are embedded into a sequence. This sequence is then input into the transformer block for processing. The MSA network can recognize the importance between different patches of the traffic panoramic image. The MSA expression is as follows:

[0108]

[0109] Where Q = XW Q K = XW K V = XW V Let W represent the query matrix, key matrix, and value matrix of the traffic image patch, respectively. Here, W is a trainable matrix, and d is the dimension of the query matrix and key matrix. Each transformer in the Swin transformer includes an MSA layer, a normalization layer (LN), and an FFN layer. The traffic panoramic image patch passes through the above stacked network layers in sequence, and the resulting feature representation is as follows:

[0110] z′ l =MSA(LN(z) l-1 )+z l-1 ),

[0111] z l =FFN(LN(z′) l )+z′ l ),

[0112] Where, z′ l and z l These are the outputs of the current block's MSA layer and FFN layer, respectively. Compared to the standard VIT, the Swin transformer achieves a better tradeoff between speed and accuracy. It is a layered transformer constructed by replacing the standard MSA in the transformer block with a shift window. The block computation form in the Swin transformer is as follows:

[0113] z′ l =W_MSA(LN(z l-1 )+z l-1 ),

[0114] z l =FFN(LN(z′) l )+z′ l ),

[0115] z′ l+1 =SW_MSA(LN(z l )+z l ),

[0116] Z l+1 =FFN(LN(z′) l+1 )+z′ l+1 ),

[0117] Among them, W_MSA is a window-based MSA that uses periodic window configuration, while SW_MSA uses a shifted window partition configuration. The feature representation Z of the panoramic view of the traffic intersection is calculated above.

[0118] After encoding by the Swin transformer backbone network, the feature representation of the traffic intersection panoramic image is obtained. This representation then passes through a gesture command extraction module and a traffic light duration command extraction module. First, Z is pooled; this patent primarily employs mean pooling to obtain the pooled feature F, calculated as follows:

[0119] F = MeanPooling(Z),

[0120] The gesture command extraction module and the traffic light duration command extraction module are mainly composed of multilayer sensors, respectively using MLP. g and MLP s This indicates that their final linear layers are activated using Softmax and Sigmoid respectively, yielding gesture features and traffic light duration features F in the panoramic view of the traffic intersection. g F s The calculation method is as follows:

[0121] F g =Softmax(MLP) g (F)),

[0122] F s =Sigmoid(MLP) s (F)),

[0123] The intersection traffic control model mainly consists of two loss components: a multi-classification loss (loss1) for gesture commands in the panoramic view of the intersection and a regression loss (loss2) for traffic light duration commands. The calculation formulas for loss1 and loss2 are as follows:

[0124]

[0125]

[0126] Among them, f gi The gesture command to indicate prediction, y gi The actual gesture command label, where C is the number of command categories, and f si The command y represents the predicted duration of the traffic light. si This represents the actual traffic light duration instruction label. The total loss (loss_total) is represented as follows:

[0127] loss_total=α·loss1+β·loss2,

[0128] Among them, α and β are adjustment balance factors, which can be limited and adjusted according to the actual situation.

[0129] Based on the same inventive concept, this application also discloses a device for constructing a traffic control model at an intersection, such as... Figure 6 As shown, it includes the following modules:

[0130] The command instruction set generation module is configured to extract traffic police command images from traffic police command videos containing traffic police command gestures, input the traffic police command images as training samples into a first neural network model to train a command instruction extraction model, use the command instruction extraction model to extract on-site command gestures from the on-site command video, summarize the extraction results to obtain an on-site command instruction set, summarize the control data of the traffic lights of the intersection signal controllers to obtain an on-site traffic light control instruction set, and summarize the on-site command instruction set and the on-site traffic light control instruction set at the same time based on the time of day to obtain the intersection traffic command instruction set at that time.

[0131] The traffic command model training module is configured to acquire a panoramic video of an intersection containing information from multiple video sources. The timelines of the panoramic video and the on-site traffic command video correspond. Multiple real-scene images of the intersection from different video sources at the same moment are extracted from the panoramic video and summarized to obtain a set of real-scene images of the intersection at that moment. The set of real-scene images of the intersection at the same moment and the set of traffic command instructions at the intersection are used as input data training samples and output data training samples, respectively, and then input into the second neural network model to train the intersection traffic command model.

[0132] In one possible implementation, the command instruction set generation module includes the following units:

[0133] The first construction unit of the command instruction extraction model is configured to acquire the traffic police command video containing the traffic police command gestures, extract multiple traffic police command images from the traffic police command video, each traffic police command image corresponds to a traffic police command gesture, label each traffic police command image according to the type of the corresponding traffic police command gesture, so that each traffic police command image corresponds to a gesture label information, match all the traffic police command images with the gesture label information and summarize them to obtain the original dataset of the extraction model, and divide the original dataset of the extraction model into the extraction model training set and the extraction model validation set according to a preset division ratio;

[0134] The second construction unit of the command instruction extraction model is configured to input the extraction model training set as training samples into the first neural network model, and the first neural network model performs data augmentation processing on the traffic police command images in the extraction model training set and uses the data augmentation processing results for model training. After the model training is completed, the initial model for command instruction extraction is obtained.

[0135] The third construction unit of the command instruction extraction model is configured to input the extraction model validation set as validation samples into the initial command instruction extraction model, and obtain the command instruction extraction model after validation and adjustment.

[0136] In one possible implementation, the command instruction set generation module further includes the following unit:

[0137] The first acquisition unit of the on-site command instruction set is configured to acquire the on-site command video, divide the on-site command video into multiple on-site command video screenshots according to time, extract on-site command gestures for each of the on-site command video screenshots using the command instruction extraction model, and assign gesture label information to the extracted on-site command gestures according to the type of traffic police command gestures they correspond to.

[0138] The second acquisition unit of the on-site command instruction set is configured to determine the duration of each on-site command gesture by comparing whether the gesture tag information corresponding to the on-site command gestures in multiple consecutive moments is consistent.

[0139] The third acquisition unit of the on-site command instruction set is configured to summarize the gesture tag information and duration corresponding to all the on-site command gestures to obtain the on-site command instruction set.

[0140] In one possible implementation, the command instruction set generation module further includes the following unit:

[0141] The on-site traffic light control instruction set acquisition unit is configured to read the control data of the traffic lights from the intersection signal controller in the intersection area. The time period of the control data corresponds to the time axis of the on-site command video. The control data includes the traffic light control strategy at any given time and the duration of each traffic light control strategy. All the control data are summarized to obtain the on-site traffic light control instruction set.

[0142] In one possible implementation, the command model training module includes the following units:

[0143] The first construction unit of the intersection traffic control model is configured to segment each intersection real-scene image in the intersection real-scene image set into blocks and embed them into a processing sequence, input the processing sequence into the second neural network model to obtain the feature representation of the intersection real-scene image, and perform pooling processing on the feature representation of the intersection real-scene image to obtain the pooled intersection real-scene image features.

[0144] The second construction unit of the intersection traffic command model is configured to process the intersection traffic command instruction set based on the intersection real-scene image features and the second neural network model, respectively obtaining the gesture features in the intersection real-scene image and the traffic light duration features in the intersection real-scene image;

[0145] The third construction unit of the intersection traffic control model is configured to construct an objective function for instruction loss, obtain the objective function construction result, and use the gesture features, the traffic light duration features, and the objective function construction result to train and optimize the second neural network model to obtain the intersection traffic control model.

[0146] Based on the same inventive concept described above, this application also discloses a smart terminal, including a memory and a processor. The memory stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to realize the method for constructing the intersection traffic control model as described above.

[0147] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0148] Based on the same inventive concept described above, this application also discloses a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the method for constructing the intersection traffic control model as described above.

[0149] Those skilled in the art should understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media capable of storing program code.

[0150] Based on the same inventive concept described above, this application also discloses a method for traffic control at intersections. The method uses the intersection traffic control model construction method described above to construct an intersection traffic control model, and includes the following steps:

[0151] A real-time panoramic video of an intersection containing information from multiple video sources is acquired. The real-time panoramic video of the intersection is then divided into multiple real-time panoramic images of the intersection according to time. Each real-time panoramic image of the intersection contains multiple real-time real-scene images of the intersection from different video sources at the same time. All the real-time panoramic images of the intersection are input into the intersection traffic control model to obtain real-time traffic control instructions fed back by the intersection traffic control model.

[0152] It should be further noted that the real-time traffic control instructions generally include real-time on-site command gestures and the duration of the real-time on-site command gestures, real-time traffic light control strategies and the duration of the real-time traffic light control strategies, and of course, any one or more of them can be selected according to actual application needs.

[0153] In summary, the intersection traffic control model constructed in this application takes real-time traffic video of the intersection as input, which is equivalent to the scene seen by traffic police on site. It utilizes a neural network-based artificial intelligence model, trained on a large amount of traffic police command video and signal control data, to learn from the experience of traffic police in commanding traffic at intersections, achieving the highest degree of fidelity to on-site traffic police command. By inputting multiple video feeds from the intersection area, the model can calculate real-time traffic control instructions, enabling real-time intersection control, adjusting traffic lights, and regulating vehicle traffic order to address different traffic situations.

[0154] As can be seen from the above description, the intersection traffic control model will be a major innovation in the field of traffic management. It can not only improve traffic efficiency and reduce congestion, but also lower the risk of traffic accidents. By utilizing end-to-end visual recognition technology and neural network model technology, we have the opportunity to achieve smarter and more efficient urban traffic management, thereby improving people's travel experience and enhancing the sustainability of urban transportation development.

[0155] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for constructing a traffic control model at an intersection, characterized in that, Includes the following steps: Traffic police command images are extracted from traffic police command videos containing hand gestures. These images are then used as training samples and input into a first neural network model to train a command instruction extraction model. This model is then used to extract on-site command gestures from the video footage. The extracted results are summarized to obtain an on-site command instruction set. Similarly, the control data of traffic lights at intersections is summarized to obtain an on-site traffic light control instruction set. Based on time, the on-site command instruction set and the on-site traffic light control instruction set for the same time are summarized to obtain the intersection traffic command instruction set for that specific time. A panoramic video of an intersection containing information from multiple video sources is acquired. The timelines of the panoramic video and the on-site command video are corresponding. Multiple real-scene images of the intersection from different video sources at the same time are extracted from the panoramic video and summarized to obtain a set of real-scene images of the intersection at that time. The set of real-scene images of the intersection at the same time and the set of traffic command instructions at the intersection are used as input data training samples and output data training samples, respectively, and then input into a second neural network model to train an intersection traffic command model. Specifically, the step of using the set of real-world images of the intersection and the set of traffic control instructions at the same time as input data training samples and output data training samples, respectively, to train the intersection traffic control model includes the following steps. Each real-scene image of the intersection in the set of real-scene images is segmented into blocks and embedded into a processing sequence. The processing sequence is then input into the second neural network model to obtain the feature representation of the real-scene image of the intersection. The feature representation of the real-scene image of the intersection is then subjected to pooling processing to obtain the pooled features of the real-scene image of the intersection. Based on the features of the intersection's real-scene image, the traffic control instruction set of the intersection is processed using the second neural network model to obtain the gesture features and traffic light duration features in the intersection's real-scene image. An objective function for command loss is constructed, and the result of the objective function construction is obtained. The second neural network model is trained and optimized using the gesture features, the traffic light duration features, and the result of the objective function construction to obtain the intersection traffic control model.

2. The method for constructing an intersection traffic control model according to claim 1, characterized in that, The process of extracting traffic police command images from traffic police command videos containing traffic police hand gestures and using these images as training samples to input into a first neural network model to train a command instruction extraction model includes the following steps: The process involves acquiring a traffic police command video containing the traffic police command gestures, extracting multiple traffic police command images from the video, with each image corresponding to a specific traffic police command gesture, labeling each image according to the type of gesture, matching all images with the corresponding gesture labels, summarizing all images to obtain the original dataset for the extraction model, and dividing the original dataset into a training set and a validation set according to a preset ratio. The extraction model training set is used as training samples and input into the first neural network model. The first neural network model performs data augmentation processing on the traffic police command images in the extraction model training set and uses the data augmentation processing results for model training. After the model training is completed, the initial model for extracting command instructions is obtained. The extraction model validation set is used as validation samples and input into the initial model for extracting command instructions. After validation and adjustment, the command instruction extraction model is obtained.

3. The method for constructing an intersection traffic control model according to claim 2, characterized in that, The gesture label information includes: going straight, turning left, waiting to turn left, turning right, changing lanes, slowing down, pulling over, and stopping.

4. The method for constructing an intersection traffic control model according to claim 2, characterized in that, The step of extracting on-site command gestures from on-site command videos using the command instruction extraction model, and summarizing the extraction results to obtain an on-site command instruction set, specifically includes the following steps: The on-site command video is acquired, and the on-site command video is divided into multiple on-site command video screenshots according to time. The on-site command gestures are extracted from each of the on-site command video screenshots using the command instruction extraction model. The extracted on-site command gestures are assigned gesture label information according to the type of traffic police command gestures they correspond to. The duration of each on-site command gesture is determined by comparing whether the gesture tag information corresponding to the on-site command gestures is consistent across multiple consecutive moments. The gesture label information and duration corresponding to all the on-site command gestures are summarized to obtain the on-site command instruction set.

5. The method for constructing an intersection traffic control model according to claim 1, characterized in that, The process of summarizing the control data of the traffic lights at the intersection area to obtain the on-site traffic light control command set includes the following steps: Read the traffic light control data of the intersection signal controller in the intersection area. The time period of the control data corresponds to the time axis of the on-site command video. The control data includes the traffic light control strategy at any time and the duration of each traffic light control strategy. Summarize all the control data to obtain the on-site traffic light control instruction set.

6. A device for constructing a traffic control model at an intersection, characterized in that, Includes the following modules: The command instruction set generation module is configured to extract traffic police command images from traffic police command videos containing traffic police command gestures, input the traffic police command images as training samples into a first neural network model to train a command instruction extraction model, use the command instruction extraction model to extract on-site command gestures from the on-site command video, summarize the extraction results to obtain an on-site command instruction set, summarize the control data of the traffic lights of the intersection signal controllers to obtain an on-site traffic light control instruction set, and summarize the on-site command instruction set and the on-site traffic light control instruction set at the same time based on the time of day to obtain the intersection traffic command instruction set at that time. The traffic command model training module is configured to acquire a panoramic video of an intersection containing information from multiple video sources. The timelines of the panoramic video and the on-site traffic command video correspond to each other. Multiple real-scene images of the intersection from different video sources at the same time are extracted from the panoramic video and summarized to obtain a set of real-scene images of the intersection at that time. The set of real-scene images of the intersection at the same time and the set of traffic command instructions at the intersection are used as input data training samples and output data training samples, respectively, and then input into the second neural network model to train the intersection traffic command model. The command model training module includes the following units: The first construction unit of the intersection traffic control model is configured to segment each real-scene image of the intersection from the set of real-scene images into blocks and embed them into a processing sequence, input the processing sequence into the second neural network model to obtain the feature representation of the intersection real-scene image, and perform pooling processing on the feature representation of the intersection real-scene image to obtain the pooled features of the intersection real-scene image. The second construction unit of the intersection traffic control model is configured to process the intersection traffic control instruction set based on the features of the intersection real-scene image and using the second neural network model to obtain the gesture features and traffic light duration features in the intersection real-scene image, respectively. The third construction unit of the intersection traffic control model is configured to construct an objective function for instruction loss, obtain the objective function construction result, and use the gesture features, the traffic light duration features, and the objective function construction result to train and optimize the second neural network model to obtain the intersection traffic control model.

7. A smart terminal, characterized in that, The system includes a memory and a processor, wherein the memory stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the method for constructing an intersection traffic control model as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the method for constructing an intersection traffic control model as described in any one of claims 1 to 5.

9. A method for traffic control at an intersection, comprising an intersection traffic control model constructed using the method for constructing an intersection traffic control model as described in any one of claims 1 to 5, characterized in that, Includes the following steps: The system acquires real-time panoramic video of an intersection containing information from multiple video sources, divides the real-time panoramic video of the intersection into multiple real-time panoramic images of the intersection according to time intervals, inputs all the real-time panoramic images of the intersection into the intersection traffic control model, and obtains real-time traffic control instructions fed back by the intersection traffic control model.

Images