Signal lamp control model training method and device, electronic equipment and storage medium

Abstract

The invention provides a signal lamp control model training method and device, electronic equipment and a storage medium, relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning, large models, computer vision and the like, and can be applied to scenes such as intelligent traffic. According to the specific implementation scheme, the method comprises the steps of obtaining a visual image, an overhead image and traffic information in a preset simulated intersection environment; constructing a plurality of sub-task training samples based on the visual image, the traffic information and a preset entity category; performing first-stage training on a projection layer and a mapping layer in the initial model based on the multiple seed task training samples to generate an intermediate model; constructing a decision reasoning training sample based on the visual image and the aerial view image; and performing second-stage training on the projection layer and the mapping layer in the intermediate model based on a decision reasoning training sample to generate a signal lamp control model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, especially to technical fields such as deep learning, large models, and computer vision, and can be applied to scenarios such as intelligent transportation, and specifically to a signal light control model training method, device, electronic device, and storage medium. Background Art

[0002] In recent years, large language models have made significant progress in language understanding, generation, and multimodal processing, demonstrating the ability to integrate multiple information sources such as images, videos, and sensor data. These technological advances have brought new opportunities for the optimization of intelligent transportation systems, especially in the field of real-time traffic light control. Summary of the invention

[0003] The present invention provides a signal light control model training method, device, electronic device and storage medium.

[0004] According to a first aspect of the present disclosure, a method for training a traffic light control model is provided, comprising: acquiring visual images, bird's-eye view images and traffic information in a preset simulated intersection environment; constructing a variety of subtask training samples based on the visual images, traffic information and preset entity categories; performing a first-stage training on the projection layer and the mapping layer in the initial model based on the various subtask training samples to generate an intermediate model; constructing decision reasoning training samples based on the visual images and the bird's-eye view images; and performing a second-stage training on the projection layer and the mapping layer in the intermediate model based on the decision reasoning training samples to generate a traffic light control model.

[0005] According to a second aspect of the present disclosure, a traffic light control method is provided, comprising: acquiring an intersection image and an overhead view of the intersection; inputting the intersection image and the overhead view of the intersection into a traffic light control model to obtain a traffic light control decision output by the traffic light control model; wherein the traffic light control model is obtained by training the method of the first aspect; and controlling the traffic light phase based on the traffic light control decision.

[0006] According to a third aspect of the present disclosure, a signal light control model training device is provided, comprising: a data acquisition module, used to acquire visual images, bird's-eye view images and traffic information in a preset simulated intersection environment; a first sample construction module, used to construct a variety of sub-task training samples based on visual images, traffic information and preset entity categories; a first-stage training module, used to perform a first-stage training on the projection layer and the mapping layer in the initial model based on the various sub-task training samples to generate an intermediate model; a second sample construction module, used to construct a decision reasoning training sample based on the visual image and the bird's-eye view image; a second-stage training module, used to perform a second-stage training on the projection layer and the mapping layer in the intermediate model based on the decision reasoning training sample to generate a signal light control model.

[0007] According to a fourth aspect of the present disclosure, a traffic light control device is provided, comprising: an information collection module for acquiring an intersection image and an intersection overhead view; a decision output module for inputting the intersection image and the intersection overhead view into a traffic light control model to obtain a traffic light control decision output by the traffic light control model; wherein the traffic light control model is obtained by training using the method of the first aspect; and a decision execution module for controlling a traffic light phase based on the traffic light control decision.

[0008] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0009] at least one processor; and

[0010] a memory communicatively connected to the at least one processor; wherein,

[0011] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0012] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute any method according to the embodiments of the present disclosure.

[0013] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0014] By adopting the scheme disclosed in the present invention, the trained traffic light control model can directly perceive the traffic conditions in real time through image or video data and make corresponding control decisions, thereby achieving more efficient and intelligent traffic management and improving the accuracy of traffic light control.

[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0017] Figure 1 is a flowchart of a signal light control model training method according to an embodiment of the present disclosure;

[0018] Figure 2 is a structural schematic diagram of a signal light control model according to an embodiment of the present disclosure;

[0019] Figure 3 is a flow chart of a signal light control method according to an embodiment of the present disclosure;

[0020] Figure 4 is a structural schematic diagram of a signal light control model training device according to an embodiment of the present disclosure;

[0021] Figure 5 is a structural schematic diagram of a signal light control device according to an embodiment of the present disclosure;

[0022] Figure 6 is a schematic diagram of a scenario for training a signal light control model according to an embodiment of the present disclosure;

[0023] Figure 7 is a schematic diagram of a scene of signal light control according to an embodiment of the present disclosure;

[0024] Figure 8 It is a structural schematic diagram of an electronic device used to implement the signal light control model training method and / or the signal light control method of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them. They do not mean to limit the order or to limit them to only two. For example, the first feature and the second feature refer to two types / two features. The first feature can be one or more, and the second feature can also be one or more.

[0027] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0028] Before introducing the technical solutions of the embodiments of the present disclosure, the technical terms that may be used in the present disclosure are further explained:

[0029] Signal light phase: refers to a control state in which the right of way is allocated to traffic flows in different directions in traffic signal control. Specifically, the signal light phase defines the different light color combinations displayed for different directions entering the intersection within a signal cycle, and these combinations determine which directions of traffic can pass. Each signal phase periodically alternates to obtain a green light display, that is, to obtain the "right of way" to pass through the intersection. Each conversion of the "right of way" is called a signal phase stage. The main purpose of the signal light phase is to properly separate traffic flows that conflict or interfere with each other, reduce traffic conflicts and interference at intersections, and ensure traffic safety and smoothness. For example, at a standard four-way intersection, there may be the following phases: Phase 1: The east-west straight direction obtains the right of way. Phase 2: The east-west left turn direction obtains the right of way. Phase 3: The north-south straight direction obtains the right of way. Phase 4: The north-south left turn direction obtains the right of way. These phases are usually switched in a fixed order, or dynamically adjusted through intelligent algorithms to adapt to actual traffic flow needs. The phase setting information mainly includes the signal cycle (the entire operation cycle of the traffic light), the red light time, and the green light time, wherein the last part of the green light time is usually a 2-3 second yellow light prompt.

[0030] Signal control strategy: A set of methods and technologies used to manage and regulate traffic light changes in order to optimize traffic flow, reduce congestion, and improve road safety and traffic efficiency.

[0031] In the related technologies, traffic light control mainly relies on rules and traditional optimization algorithms. When faced with complex and changeable traffic conditions, it is difficult to achieve efficient, flexible, and end-to-end signal control. First, the model training process based on reinforcement learning takes a long time and requires a lot of online exploration and trial and error processes. This not only increases the consumption of computing resources, but may also cause extreme traffic congestion during the exploration phase. Secondly, although the model based on imitation learning can control traffic lights by learning the decision-making behavior of experts, it is difficult to break through the level of experts. When faced with complex or sudden traffic conditions that have never been encountered, the adaptability of the imitation learning model is poor and it is impossible to generate control strategies that exceed the performance of experts. In addition, the models based on large language models, reinforcement learning, and imitation learning still rely on numerical inputs and cannot directly process images or video data from cameras, so full end-to-end control has not yet been achieved. It is difficult to directly perceive traffic conditions in real time through visual data and make corresponding control decisions. There is still a large technical gap from practical application.

[0032] In order to at least partially solve one or more of the above problems and other potential problems, the present disclosure proposes a traffic light control model training method, which can perceive traffic conditions based on multimodal data and use the reasoning ability of a large language model to perform precise control and decision-making. The traffic light control model can directly perceive traffic conditions in real time through image or video data and make corresponding control decisions, thereby optimizing traffic light control and achieving more efficient and intelligent traffic management.

[0033] The present disclosure provides a method for training a signal light control model. Figure 1 It is a flow chart of the signal light control model training method according to an embodiment of the present disclosure. The signal light control model training method can be applied to a signal light control model training device. The signal light control model training device is located in an electronic device. The electronic device includes but is not limited to fixed devices and / or mobile devices. For example, fixed devices include but are not limited to servers, and the server can be a cloud server or an ordinary server. For example, mobile devices include but are not limited to: mobile phones, tablet computers, and vehicle-mounted terminals. In some possible implementations, the signal light control model training method can also be implemented by a processor calling computer-readable instructions stored in a memory. For example Figure 1 As shown, the traffic light control model training method includes:

[0034] S101, acquiring visual images, bird's-eye view images and traffic information in a preset simulated intersection environment;

[0035] S102, constructing a variety of sub-task training samples based on visual images, traffic information and preset entity categories;

[0036] S103, based on multiple sub-task training samples, perform the first stage training on the projection layer and mapping layer in the initial model to generate an intermediate model;

[0037] S104, constructing decision reasoning training samples based on the visual image and the bird's-eye view image;

[0038] S105. Based on the decision reasoning training samples, the projection layer and the mapping layer in the intermediate model are trained in the second stage to generate a traffic light control model.

[0039] In the disclosed embodiment, the preset simulated intersection environment refers to a virtual environment constructed to simulate the traffic conditions in the real world during the training of the signal light control model. This environment usually includes visual images, bird's-eye views and related traffic information of the intersection, which is used to simulate various situations that may be encountered in real traffic scenes. In this simulated environment, various parameters such as traffic flow, vehicle type, signal light settings, etc. can be controlled and adjusted to facilitate testing and training of the signal light control model under different conditions. Exemplarily, the preset simulated intersection environment may include: traffic flow simulation, simulating the changes in traffic flow in different time periods and weather conditions, and the impact of special events such as traffic accidents or road maintenance on traffic flow; intersection layout, simulating the geometric structure of the intersection, including the number of lanes, intersection shape, location of traffic signs and signal lights, etc.; traffic signal control, simulating the operating logic of the signal light, including the phase, duration, conversion logic, etc. of the signal light; vehicle behavior simulation, simulating the behavior of vehicles at intersections, including complying with traffic rules, avoiding pedestrians, responding to emergencies, etc.; sensors and data acquisition: simulating data collected by various sensors such as cameras and radars. The above is only an exemplary description and is not intended to limit all possible situations of the simulated intersection environment. It is just that this is not an exhaustive list.

[0040] In the embodiments of the present disclosure, visual images refer to images generated by computer simulation technology, which simulate the scenes at traffic intersections in the real world. In the process of acquiring visual images in a preset simulated intersection environment, a virtual camera can be set in an artificial scene, and images can be repeatedly acquired by adjusting the viewing angle and focal length of the camera, thereby simulating visual images at different positions and angles. The above is only an exemplary description and is not intended to limit all possible situations for acquiring visual images, but it is not exhaustive here.

[0041] In the disclosed embodiments, a bird's-eye view image refers to an image obtained from a simulated environment that simulates an intersection viewed from a high altitude in the real world, and can provide a macroscopic perspective on the overall layout of the intersection and traffic flow. In the process of obtaining a bird's-eye view image in a preset simulated intersection environment, a conditional generation model can be used to accurately render roads and lane lines from a bird's-eye view layout, as well as generate traffic scenes under different weather conditions and times, thereby generating a bird's-eye view image. At the same time, cross-view conversion and spatial attention are combined to learn the relationship between the camera and map views to ensure consistency. The above is only an exemplary description and is not intended to limit all possible situations for obtaining bird's-eye view images, but it is not exhaustive here.

[0042] In the disclosed embodiments, traffic information refers to data about traffic conditions generated by simulation software or systems, which can simulate various dynamic and static information of traffic intersections in the real world. Traffic information includes, but is not limited to: vehicle information, including the location, speed, acceleration, driving direction, vehicle type and vehicle status of the vehicle; traffic flow, the number of vehicles passing through the intersection in a specific time period, which can be real-time data or simulated periodic flow; pedestrian and non-motor vehicle information, the location, speed, moving direction and other information of pedestrians and non-motor vehicles; traffic accidents and emergencies, simulating the occurrence of traffic accidents, vehicle failures or other emergencies, and the impact of these situations on traffic flow; road conditions, including the geometry of the road, the number of lanes, the road surface conditions, etc. The process of obtaining traffic information in a preset simulated intersection environment can be generated in a simulated environment by means of sensor data, video surveillance, traffic flow models, etc. The above is only an exemplary description and is not intended to limit all possible situations for obtaining traffic information, but it is not exhaustive here.

[0043] In the disclosed embodiments, entity categories refer to a group of predefined categories of objects with common attributes, characteristics or behaviors in a simulation environment, which can serve as the basis for building a simulation environment or database and for organizing and managing data. Specifically, entity types may include "vehicles", "pedestrians", "traffic lights", etc. Exemplarily, entity categories can be defined as eight types, including vehicles, sidewalks, stop lines, white dashed lines, double yellow lines, green belts, sidewalks, and buildings. The above are only exemplary descriptions and are not intended to limit all possible situations of entity categories, but they are not exhaustive here.

[0044] In the disclosed embodiment, subtask refers to decomposing a complex traffic light control model training task into multiple smaller, easier to manage and train tasks, which can improve the efficiency and effect of model training when processing large-scale or complex data sets. After decomposing the traffic light control model training task into multiple subtasks, the model training process becomes a multi-task learning process. Subtask training samples refer to data samples for each subtask prepared for training the model under the multi-task learning framework. In the process of constructing subtask training samples, the features and attributes of each category can be first defined according to the preset entity category, and then the visual image can be annotated, including the location, size, category and other information of the entity; then the visual image and traffic information can be combined to create a multimodal training sample. Exemplarily, the subtasks can be set to four, including scene understanding, object recognition, object counting, and relational reasoning. The above is only an exemplary description and is not intended to limit all possible situations for constructing subtask training samples, but it is not exhaustive here.

[0045] In the disclosed embodiment, the projection layer and mapping layer in the initial model refer to the part of the model used for data conversion and feature extraction. Specifically, the projection layer is usually located in the middle or output part of the neural network, and its main function is to map data from one representation space to another representation space. Further, the mapping layer can refer to the prediction head, which is generally the last part of the model, and is used to make the final task prediction based on the features learned by the model. The process of performing the first stage of training can use a variety of subtask training samples to jointly train the initial model. During the training process, the initial model will learn multiple subtasks at the same time, share the features of the projection layer and the mapping layer, and the specific branches of each subtask can independently learn the task-specific features. After the first stage of training, the projection layer and mapping layer of the initial model will be adjusted to adapt to the subtask, thereby generating an intermediate model. This intermediate model can be used as the basis for further training, or for fine-tuning of specific tasks. The above is only an exemplary description and is not intended to limit all possible situations in the first stage, but it is not exhaustive here.

[0046] In the disclosed embodiments, decision reasoning training samples refer to data sets used to train models for decision reasoning, which include input features and corresponding output labels. The model learns these data to learn how to make decisions or predict outputs from the input features. Specifically, visual images and bird's-eye views can be used as inputs, and traffic light control decisions can be used as outputs to construct decision reasoning training samples, so that the model can learn the data of the training samples and ultimately make traffic light control decisions based on the visual images and bird's-eye views. The above is only an illustrative description and is not intended to limit all possible situations for constructing decision reasoning training samples, except that they are not exhaustive.

[0047] In the disclosed embodiment, the process of the second stage training can first load the intermediate model that has been trained in the first stage; then use the decision reasoning training samples to perform the second stage training on the intermediate model; illustratively, the weights of the projection layer and the mapping layer can be updated by the back propagation algorithm to minimize the defined loss function, and optimization algorithms such as batch training and stochastic gradient descent can be used in the training process to improve the training efficiency; finally, a traffic light control model that can make effective decisions in a complex traffic environment is generated. The above is only an exemplary explanation and is not intended to limit all possible situations for the second stage training, but it is not exhaustive here.

[0048] In some embodiments, various simulators are first used to construct a virtual environment that can simulate real-world traffic conditions, and then based on the virtual environment, simulated visual images, bird's-eye views, and traffic information are obtained. Further, the visual images are annotated according to preset entity categories, and then the visual images and traffic information are combined to construct multimodal training samples for preset multiple subtasks. Further, the initial model is trained in the first stage using the multimodal training samples, and the parameters of the model, especially the weights of the projection layer and the mapping layer, are iteratively updated to generate an intermediate model. Further, according to the preset traffic light control decision, the visual image and the bird's-eye view image are annotated to construct a decision reasoning training sample; finally, the intermediate model is trained in the second stage using the decision reasoning training sample, and the parameters of the model, especially the weights of the projection layer and the mapping layer, are iteratively updated to generate a traffic light control model.

[0049] The technical solution of the disclosed embodiment, by setting up a simulated intersection environment, can conduct a large number of experiments and training on the traffic light control model without actual deployment, optimize the performance of the model, and predict the performance of the model in the real world, which can significantly reduce the training cost, improve the training efficiency, and reduce the interference with actual traffic. At the same time, by defining a variety of subtasks, complex tasks can be decomposed, thereby accelerating the convergence of the model and ultimately improving the performance of the model. In addition, by constructing decision reasoning training samples, the model can make effective decisions in a complex traffic environment only through visual images and bird's-eye views, which improves the versatility of the model while making the traffic light control task explainable. Moreover, through two training tasks for the model, the model can achieve end-to-end decision-making from visual images to traffic light control strategies, achieve more efficient and intelligent traffic management, and improve the accuracy of traffic light control.

[0050] In some embodiments, the initial model includes: a visual encoder, a projection layer, a base large model and a mapping layer; the visual encoder is connected to the input end of the base large model through the projection layer, and the mapping layer is connected to the output end of the base large model.

[0051] In the disclosed embodiments, the visual encoder is a part of the model that is specifically responsible for processing visual input data, such as images or video frames. It is usually composed of a convolutional neural network that can extract features from visual data, such as edges, textures, shapes, and object information. The output of the visual encoder can be a feature vector that encodes key information of the input image.

[0052] In the disclosed embodiment, the projection layer is located between the visual encoder and the base model. Its main function is to convert or project the high-dimensional feature vector extracted by the visual encoder to a new space. The feature vector of this new space is more suitable for the input of the base model. This conversion can be linear or nonlinear, depending on the specific implementation and design of the projection layer.

[0053] In the disclosed embodiment, the base large model is a pre-trained large-scale model, which is usually trained on a large amount of data to learn common language and / or visual patterns. Exemplarily, the base large model can be a large language model. In the signal light control model, the base large model can further process and integrate the features of the visual encoder and traffic information to make complex decisions.

[0054] In the disclosed embodiments, the mapping layer, also called the prediction head, is the last part of the model and is responsible for converting the output of the base model into the final prediction or decision. In the traffic light control model, the mapping layer may be responsible for converting the output of the model into the state of the traffic light or other control signals. Exemplarily, the mapping layer may include a fully connected layer to generate class probabilities or continuous value predictions.

[0055] In this way, by using a pre-trained large pedestal model, the training time required for new tasks can be significantly reduced, while improving the performance of the model on small sample data sets. At the same time, the visual encoder is responsible for extracting features from the visual input, while the large pedestal model processes these features and generates outputs, allowing the model to automatically extract image features, eliminating the tedious manual feature design process, and being able to process and learn specific visual information. In addition, end-to-end learning directly from raw data input to final output results can simplify the model building process and allow the model to directly capture the intrinsic correlation of the data, reducing possible information loss in the intermediate links. Moreover, by combining the visual encoder and the large pedestal model, the model can process multimodal data and improve the ability to understand and process visual and language information.

[0056] In some embodiments, in a preset simulated intersection environment, visual images, bird's-eye view images and traffic information are obtained, including: in the simulated intersection environment, respectively setting up cameras and sensors for any direction; obtaining visual images and bird's-eye view images based on the cameras; obtaining traffic flow information based on the sensors; and generating traffic information based on the visual images and traffic flow information.

[0057] In the disclosed embodiments, the preset simulated intersection environment can be simulated synchronously using the Carla autonomous driving simulator in combination with the Simulation of Urban Mobility (Sumo) traffic light control simulator, or it can be simulated using other methods in the prior art. The above is only an exemplary description and is not intended to limit all possible situations of the simulation environment, but it is not exhaustive here.

[0058] In the disclosed embodiment, setting up cameras and sensors for any direction respectively means that in the virtual traffic simulation system, virtual cameras and sensors are installed in the east, south, west and north directions of the intersection respectively. Among them, the virtual camera is used to capture the visual image of the intersection, including the status of vehicles, pedestrians, traffic lights, etc., and can also provide visual information of traffic flow, vehicle behavior and traffic events for decision-making in traffic management and signal control; the virtual sensor is used to measure key traffic parameters such as traffic flow, vehicle speed, and vehicle distance. For example, the virtual sensor can simulate radar, ultrasonic sensors, etc. in the real world to detect and measure the position, speed and direction of vehicles and pedestrians. The above is only an exemplary description and is not intended to limit all possible situations of cameras and sensors, but it is not exhaustive here.

[0059] In the disclosed embodiment, according to the process of the camera acquiring the visual image and the bird's-eye view image, various parameters may be set first to control the image acquisition process, such as image width, height, frame rate, brightness, contrast, etc.; then, the image frame of the camera is read and displayed. Among them, for acquiring the bird's-eye view image, the bird's-eye view angle may be simulated by setting the position and angle of the virtual camera. The above is only an exemplary description and is not intended to limit all possible situations for acquiring the visual image and the bird's-eye view image, but it is not exhaustive here.

[0060] In the disclosed embodiment, according to the process of obtaining traffic flow information by sensors, a random traffic flow can be first generated in Sumo, and then the induction coil detector (E1 Detector) and the lane area detector (E2 Detector) can be used to obtain traffic flow information. The above is only an exemplary description and is not intended to limit all possible situations for obtaining traffic flow information, but it is not exhaustive here.

[0061] In the disclosed embodiments, the process of generating traffic information based on visual images and vehicle flow information refers to the process of using vehicle flow information to mark visual images to form credible traffic information. By way of example, computer vision techniques such as target detection and tracking algorithms can be first used to identify and track vehicles from preprocessed images, thereby identifying the location, size, and motion state of the vehicles; then, the collected vehicle flow information can be marked on the visual image; finally, the visual image and vehicle flow information are combined to generate comprehensive traffic information. In particular, traffic information includes not only the intuitive distribution and movement of vehicles in the visual image, but also quantitative vehicle flow data, such as vehicle flow density, average speed, etc. The above is only an exemplary description and is not intended to limit all possible situations for generating traffic information, but it is not exhaustive here.

[0062] In this way, by setting up cameras and sensors in a simulated intersection environment, real-world traffic conditions can be simulated more realistically, improving the accuracy and reliability of the simulation. At the same time, the generated traffic information can provide more accurate decision support for traffic management and signal light control, helping to optimize traffic flow and reduce congestion. In addition, using data in a simulated environment for model training allows a large number of experiments to be conducted without interfering with real traffic, improving the efficiency and safety of model training.

[0063] In some embodiments, a variety of subtask training samples are constructed based on visual images, traffic information and preset entity categories, including: constructing subtask texts corresponding to each subtask according to the entity categories; taking the visual image and subtask text as input and the traffic information as the true value, constructing subtask training samples corresponding to each subtask.

[0064] In the disclosed embodiment, according to the entity category, the process of constructing the subtask text corresponding to each subtask can first construct the subtask text for each subtask in turn; then the entity category can be distinguished in the subtask text. For example, for the four subtasks of scene understanding, object recognition, object counting and relational reasoning, eight entity categories are defined: vehicle, sidewalk, stop line, white dashed line, double yellow line, green belt, sidewalk and building.

[0065] In some embodiments, eight entity categories are defined for the scene understanding subtask at the same time, and the constructed scene understanding subtask text can be: "Describe the objects that appear in the images captured by the surveillance camera. Focus on all objects in the following entity categories: vehicles, pedestrian crossing facilities, stop lines, white dashed lines (i.e., lane dividers used to separate lanes of traffic flow in the same direction), double yellow lines (i.e., centerline markings used to separate traffic flow in opposite directions), medians (i.e., traffic islands with protective markings or trees used to separate traffic flow in opposite directions), pedestrian crossing facilities (i.e., zebra crossings located in front of stop lines), sidewalks (i.e., paved paths on both sides of the road for pedestrians to pass, connecting buildings), and buildings (i.e., high-rise buildings with visible windows, balconies, doors, and exterior walls). Any objects beyond the above categories shall not be discussed. Please describe the appearance, location, and orientation of each object."

[0066] In some implementations, eight entity categories are defined for the object recognition subtask, and the constructed object recognition subtask text may be: "Identify the objects surrounded by the red rectangular box in the image. Focus on the following entity categories: vehicles, pedestrian crossing facilities, stop lines, white dashed lines, double yellow lines, medians, pedestrian crossing facilities, sidewalks, and buildings."

[0067] In some implementations, eight entity categories are defined for the object counting subtask, and the constructed object counting subtask text may be: "How many objects in the image belong to the same category? Please focus on the following entity categories: vehicles, pedestrian crossing facilities (such as crosswalks), stop lines, white dashed lines, double yellow lines, medians, pedestrian crossing facilities, sidewalks, and buildings."

[0068] In some embodiments, for the relational reasoning subtask, eight entity categories are defined at the same time, and the constructed relational reasoning subtask text may be: "Describe the relationship between the two objects highlighted in the blue and red rectangular boxes. You can describe their spatial relationship, for example: in front, behind, behind, to the left, to the right, on top, etc."

[0069] The above is only an exemplary description and is not intended to limit all possible situations for constructing subtask texts. This is just not an exhaustive list.

[0070] In the disclosed embodiment, in the process of constructing subtask training samples corresponding to each subtask by taking visual images and subtask texts as inputs and traffic information as truth values, different processing is required according to different subtasks. Exemplarily, for the scene understanding subtask, the visual image and the scene understanding subtask text can be directly taken as inputs, and various dynamic and static information in the traffic information can be taken as truth values ​​to construct the scene understanding subtask training samples. Further, for the object recognition subtask, it is first necessary to use various dynamic and static information in the traffic information to establish an object bounding box in the visual image; then the visual image of the existing object bounding box and the object recognition subtask text are taken as inputs, and the object category corresponding to the object bounding box in the traffic information is taken as the truth value to construct the object recognition subtask training sample. Further, for the object counting subtask, it is first necessary to use various dynamic and static information in the traffic information to establish an object bounding box in the visual image; then the visual image of the existing object bounding box and the object counting subtask text are taken as inputs, and the object category and quantity corresponding to the object bounding box in the traffic information are taken as the truth value to construct the object counting subtask training sample. Furthermore, for the relational reasoning subtask, it is necessary to first use various dynamic and static information in the traffic information to establish two object bounding boxes of different colors in the visual image; then the visual image with two object bounding boxes and the relational reasoning subtask text are used as input, and the object relationship corresponding to the object bounding box in the traffic information is used as the true value to construct the relational reasoning subtask training sample. The above is only an exemplary explanation and is not intended to limit all possible situations for constructing subtask training samples, but it is not exhaustive here.

[0071] In this way, through the subtask training samples generated by visual images and traffic flow information, the model can be trained to effectively identify and predict traffic information in a variety of scenarios, enhance the generalization ability of the model, and generate more easily interpretable training samples, which helps to understand the decision-making process of the model and improve the interpretability of the model. At the same time, through the preset entity categories and corresponding subtask texts, training samples for specific tasks can be quickly constructed to accelerate the training process of the model and support complex traffic scene understanding tasks. In addition, by defining different subtasks and different entity categories, the flexibility and adaptability of the model can be improved, and it can be used for a variety of different traffic analysis tasks.

[0072] In some embodiments, based on multiple sub-task training samples, the projection layer and the mapping layer in the initial model are trained in the first stage to generate an intermediate model, including: based on multiple sub-task training samples, the projection layer is pre-trained to generate a first pre-selected projection layer; based on multiple sub-task training samples, the first pre-selected projection layer and the mapping layer are fine-tuned to generate an intermediate model.

[0073] In the disclosed embodiment, the projection layer is pre-trained based on a variety of subtask training samples to generate the first pre-selected projection layer. The process can first use a multimodal data fusion algorithm to combine visual images and subtask texts to construct a shared subspace, and complete data mining tasks such as clustering and classification in the subspace to achieve semantic alignment of data of different modalities; then, the synergy of the two modal data can be maximized by training in turn, which requires that each modality has sufficient data, and the objective functions of the two modalities based on the symbiotic features can predict the same true value label with a high probability; finally, the trained projection layer is used as the first pre-selected projection layer. The above is only an exemplary explanation, and it is not intended to limit all possible situations for generating the first pre-selected projection layer, but it is not exhaustive here.

[0074] In the disclosed embodiment, based on a variety of subtask training samples, the first pre-selected projection layer and mapping layer are fine-tuned to generate an intermediate model. First, the contrastive learning technique can be used to minimize the distance between similar text-image pairs and maximize the distance between unrelated pairs to achieve alignment; then, instructions and visual features can be sampled from the instruction data and constructed into an image-text instruction input model, and the image-text instruction pairs can be used to further train the pre-selected projection layer and mapping layer so that they can understand the image content and execute the corresponding text instructions; finally, the adjusted first pre-selected projection layer and mapping layer are used to generate the intermediate model. The above is only an exemplary description and is not intended to limit all possible situations for generating the intermediate model, but it is not exhaustive here.

[0075] In this way, through pre-training and fine-tuning, the model can learn how to fuse visual images and text information to extract richer multimodal features. At the same time, through instruction fine-tuning, the model can more accurately understand and respond to text instructions related to images, enhancing the accuracy and robustness of the model in practical applications. In addition, through pre-training and fine-tuning, the model can adapt to complex scenarios and different data distributions, enhancing its applicability in changing environments, and making the feature representation and decision-making process learned by the model clearer, which helps to improve the interpretability of the model.

[0076] In some embodiments, the projection layer is pre-trained based on multiple sub-task training samples to generate a first pre-selected projection layer, including: freezing all parameters of the visual encoder, the base large model and the mapping layer in the initial model; and pre-training the projection layer using multiple sub-task training samples to generate a first pre-selected projection layer.

[0077] In the disclosed embodiment, the process of freezing all parameters of the visual encoder, the base large model, and the mapping layer in the initial model can first traverse all parameters of the model to extract all parameters related to the visual encoder, the base large model, and the mapping layer; then, parameter freezing can be achieved by setting parameter attributes. For different model architectures, parameter freezing can be achieved by setting different parameter attributes. The above is only an exemplary description and is not intended to limit all possible situations for freezing parameters, but it is not exhaustive here.

[0078] In the disclosed embodiment, the projection layer is pre-trained using a variety of sub-task training samples to generate the first pre-selected projection layer. First, the multiple sub-task training samples can be input into the model. Under the premise of freezing all parameters of the visual encoder, the base large model and the mapping layer in the initial model, the projection layer is trained and its parameters are adjusted so that the projection layer can better map the input data to the new feature space. Subsequently, the performance of the model can be evaluated on each sub-task to ensure that the projection layer can provide useful feature representations for different tasks. Finally, the trained projection layer is used as the first pre-selected projection layer. The above is only an exemplary description and is not intended to limit all possible situations for generating the first pre-selected projection layer, but it is not exhaustive here.

[0079] In this way, by freezing the parameters of the visual encoder and the base large model, only the parameters of the projection layer can be changed, so that the visual features extracted by the visual encoder and the text embedding obtained by the base large model embedding layer are aligned, which significantly improves the efficiency of model training.

[0080] In some embodiments, based on a variety of sub-task training samples, instructions for fine-tuning the pre-selected projection layer and the mapping layer are performed to generate an intermediate model, including: freezing all parameters of the visual encoder and the base large model in the initial model; pre-training the first pre-selected projection layer and the mapping layer according to a variety of sub-task training samples to generate a second pre-selected projection layer and a first pre-selected mapping layer; generating an intermediate model based on the visual encoder, the second pre-selected projection layer, the base large model and the first pre-selected mapping layer.

[0081] In the disclosed embodiment, the process of freezing all parameters of the visual encoder base large model in the initial model can first traverse all parameters of the model to extract all parameters related to the visual encoder and the base large model; then, parameter freezing can be achieved by setting parameter attributes. The above is only an exemplary description and is not intended to limit all possible situations for freezing parameters, but it is not exhaustive here.

[0082] In the disclosed embodiment, the process of pre-training the first pre-selected projection layer and the mapping layer according to the training samples of various sub-tasks, and generating the second pre-selected projection layer and the first pre-selected mapping layer can first input the training samples of various sub-tasks into the model, and train the first pre-selected projection layer and the mapping layer on the premise of freezing all the parameters of the visual encoder and the base large model in the initial model, and adjust their parameters so that the first pre-selected projection layer and the mapping layer can better convert the input data into the final prediction result; then, the performance of the model can be evaluated on each sub-task to ensure that the trained first pre-selected projection layer and the mapping layer can effectively handle different sub-tasks; finally, the trained first pre-selected projection layer and the mapping layer are used as the second pre-selected projection layer and the first pre-selected mapping layer. The above is only an exemplary description, and is not intended to limit all possible situations for generating the second pre-selected projection layer and the first pre-selected mapping layer, but it is not exhaustive here.

[0083] In the disclosed embodiment, the process of generating the intermediate model according to the visual encoder, the secondary pre-selected projection layer, the base large model and the first pre-selected mapping layer can be achieved by integrating the visual encoder, the secondary pre-selected projection layer, the base large model and the first pre-selected mapping layer into a unified model architecture. The above is only an exemplary description and is not intended to limit all possible situations for generating the intermediate model, but it is not exhaustive here.

[0084] In this way, freezing the parameters of the visual encoder and the base model helps reduce the forgetting of pre-training knowledge during fine-tuning and maintains the performance of the model on specific tasks. At the same time, by using a variety of sub-task training samples for pre-training and instruction fine-tuning, the model can better understand and process the relationship between images and text, improving the model's multimodal understanding ability.

[0085] In some implementations, taking the scene recognition subtask training sample as an example, for an image X v , randomly extract a language instruction X from the scene recognition subtask text q , the true value answer is X a is the understanding and description of the image. During the training process, we keep the weights of the visual encoder and the base model unchanged, maximize the likelihood value in formula (1), freeze the weights of the visual encoder, and only train the parameters W of the projection layer after the visual encoder. In this way, the image features can be aligned with the pre-trained base model word embedding.

[0086]

[0087] Among them, p represents the probability, i represents the index of the generated element in the sequence, L represents the length of the generated sequence, and X i is the i-th element of the generated sequence of length L.

[0088] In some embodiments, all parameters of the visual encoder and the base large model in the initial model are frozen; the first pre-selected projection layer and the mapping layer are pre-trained according to a variety of sub-task training samples to generate a second pre-selected projection layer and a first pre-selected mapping layer, that is, the trainable parameters θ are {W, φ}, W represents the parameters of the projection layer, and φ represents the parameters of the mapping layer.

[0089] In some embodiments, based on the decision reasoning training samples, the projection layer and mapping layer of the intermediate model are trained in the second stage to generate a traffic light control model, including: freezing all parameters of the visual encoder and the base large model of the intermediate model; according to the decision reasoning training samples, the secondary pre-selected projection layer and the first pre-selected mapping layer are trained to generate a traffic light control projection layer and a traffic light control mapping layer; based on the visual encoder, the traffic light control projection layer, the base large model and the traffic light control mapping layer, a traffic light control model is generated.

[0090] In the disclosed embodiment, according to the decision reasoning training samples, the secondary pre-selected projection layer and the first pre-selected mapping layer are trained to generate the signal light control projection layer and the signal light control mapping layer. The decision reasoning training samples can be first input into the model, and the secondary pre-selected projection layer and the first pre-selected mapping layer are trained on the premise of freezing all the parameters of the visual encoder and the base large model in the initial model, and their parameters are adjusted so that the secondary pre-selected projection layer and the first pre-selected mapping layer can better convert the input data into the final signal light control decision; then, the performance of the model can be evaluated on each subtask to ensure that the trained secondary pre-selected projection layer and the first pre-selected mapping layer can effectively handle different subtasks; finally, the trained secondary pre-selected projection layer and the first pre-selected mapping layer are used as the signal light control projection layer and the signal light control mapping layer. The above is only an exemplary description, and is not intended to limit all possible situations for generating the signal light control projection layer and the signal light control mapping layer, but it is not exhaustive here.

[0091] In the disclosed embodiment, the process of generating the signal light control model according to the visual encoder, the signal light control projection layer, the base large model and the signal light control mapping layer can be achieved by integrating the visual encoder, the signal light control projection layer, the base large model and the signal light control mapping layer into a unified model architecture. The above is only an exemplary description and is not intended to limit all possible situations for generating the intermediate model, but it is not exhaustive here.

[0092] In this way, by freezing all parameters of the visual encoder of the intermediate model and the base large model, while keeping the basic structure of the model stable, the secondary pre-selected projection layer and the first pre-selected mapping layer are optimized. This phased training strategy effectively avoids the complexity and time cost of full model training and significantly improves training efficiency.

[0093] In some embodiments, decision reasoning training samples are constructed based on visual images and bird's-eye view images, including: splicing visual images according to preset splicing rules to generate multi-perspective images; constructing decision reasoning task texts based on the multi-perspective images; generating expert decision data based on the bird's-eye view images at the same time as the visual images; and constructing decision reasoning training samples using the visual images and the decision reasoning task texts as inputs and the expert decision data as true values.

[0094] In the disclosed embodiment, the visual images are stitched according to the preset stitching rules to generate the multi-view image. The image processing library can be used to stitch the pre-processed images according to the stitching rules. By way of example, for the visual images in the four directions of east, west, south and north, the east perspective image can be placed in the upper left corner, the west perspective image can be placed in the upper right corner, the south perspective image can be placed in the lower left corner, and the north perspective image can be placed in the lower right corner, thereby generating a visual image that includes the four directions of east, west, south and north. The above is only an exemplary description and is not intended to limit all possible situations for generating multi-view images. It is just that this is not an exhaustive list.

[0095] In the disclosed embodiment, according to the multi-perspective image, the process of constructing the decision reasoning task text can first construct the decision reasoning task text, and then adjust it according to the multi-perspective image. Exemplarily, for a multi-perspective image in which the east perspective image is placed in the upper left corner, the west perspective image is placed in the upper right corner, the south perspective image is placed in the lower left corner, and the north perspective image is placed in the lower right corner, the decision reasoning task text can be constructed as: "Given four surveillance camera images of the east (upper left), west (upper right), south (lower left), and north (lower right) of the intersection, describe each image according to the number of vehicles lined up on the approaching road. Based on the description, give suggestions on which phase direction should turn green during the upcoming interval at the intersection to reduce congestion. By analyzing and sorting the degree of traffic congestion in each direction, and then deciding which phase should turn green, give priority to alleviating congestion in the most congested direction, and give your answer. "The above is only an exemplary explanation, and is not intended to be a limitation on all possible situations for constructing the decision reasoning task text, but it is not exhaustive here.

[0096] In the disclosed embodiment, the process of generating expert decision data based on the bird's-eye view image at the same time as the visual image can query the natural language processing model in the field of artificial intelligence through the bird's-eye view image at the time corresponding to each multi-view image, generate the optimal signal light phase control decision, and provide corresponding reasoning reasons and explanations, and use the signal light phase control decision, reasoning reasons and explanations as expert decision data. The above is only an exemplary explanation and is not intended to limit all possible situations for generating expert decision data, but it is not exhaustive here.

[0097] In the disclosed embodiment, visual images and decision reasoning task texts are taken as inputs, and expert decision data are taken as true values. The process of constructing decision reasoning training samples can integrate visual images, decision reasoning task texts and expert decision data to form decision reasoning training samples.

[0098] In this way, using images from multiple perspectives and corresponding task texts to train the model can improve the model's generalization ability in different scenarios. By combining visual images and bird's-eye views, the model can gain a more comprehensive understanding of the situation. Combining decision reasoning task texts with expert decision data can make the model's decision process more transparent and enhance the model's interpretability.

[0099] Figure 2 The structure diagram of the signal light control model is shown in FIG. Figure 2 As shown, the multi-view images are input into the visual encoder, and the decision reasoning task text is input into the word segmenter. The output of the visual encoder is connected to the input of the projection layer, the output of the projection layer is connected to the first input of the base model, the output of the word segmenter is connected to the second input of the base model, the output of the base model is connected to the input of the mapping layer, and the output of the mapping layer is the reasoning process and the signal light phase decision.

[0100] Exemplarily, the visual encoder can adopt a visual encoder of contrastive language-image pretraining (CLIP). The visual encoder can be based on a visual architecture (Vision Transformer, ViT) of a self-attention mechanism, and can specifically include 24 layers of encoder-decoder (Transformer) layers, each layer has 1024-dimensional hidden units and 16 attention heads, and the received image will be divided into non-overlapping slices of 14 pixels by 14 pixels. Exemplarily, the base large language model can adopt an open source large language model with a small number of parameters and fast inference speed.

[0101] It should be understood that Figure 2 The schematic diagram shown is only exemplary and not restrictive, and it is scalable, and those skilled in the art can Figure 2 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0102] The present disclosure provides a signal light control method. Figure 3: is a flowchart of a signal light control method according to an embodiment of the present disclosure. The signal light control method can be applied to electronic devices, including but not limited to fixed devices and / or mobile devices. For example, fixed devices include but are not limited to servers, and the servers can be cloud servers or ordinary servers. For example, mobile devices include but are not limited to: mobile phones, tablet computers, vehicle-mounted devices, personal computers, etc. In some possible implementations, the signal light control method can also be implemented by a processor calling computer-readable instructions stored in a memory. Figure 3 As shown, the signal light control method includes:

[0103] S301, obtaining an intersection image and an intersection bird's-eye view;

[0104] S302, inputting the intersection image and the intersection bird's-eye view into the traffic light control model to obtain a traffic light control decision output by the traffic light control model;

[0105] S303: Control the phase of the signal light based on the signal light control decision.

[0106] In the disclosed embodiment, multiple image acquisition devices can be deployed at different locations of the traffic intersection to ensure that the viewing angle of the image acquisition device covers the entire intersection, thereby obtaining an intersection image. The bird's-eye view of the intersection can be obtained by perspective transformation and aggregation of the intersection image, or by using a drone or setting up other image acquisition devices. The above is only an exemplary description and is not intended to limit all possible situations for obtaining intersection images and intersection bird's-eye views, but it is not exhaustive here.

[0107] In the disclosed embodiment, the traffic light control model is used to generate a traffic light control strategy based on the intersection image and the intersection bird's-eye view.

[0108] In the disclosed embodiment, controlling the phase of a traffic light based on a traffic light control decision refers to directly managing the phases of a traffic light (i.e., the phases of a green light, a yellow light, and a red light) using the decision generated by the traffic light control model. According to the traffic light control decision, the control system will adjust the phase of the traffic light. Exemplarily, the control system will automatically adjust the timer of the traffic light based on the output of the model to ensure that the traffic light displays the correct color at the correct time.

[0109] The technical solution of the disclosed embodiment can more accurately capture and analyze traffic flow and intersection conditions by acquiring intersection images and bird's-eye views, combined with a signal light control model, thereby generating more optimized signal light control decisions, which can help reduce vehicle waiting time and improve the traffic capacity of intersections. Output decisions can be made through the signal light control model, which can be adjusted based on real-time traffic conditions, reduce traffic accidents caused by improper signal light settings, improve intersection safety, and achieve coordinated control between intersections.

[0110] The present disclosure provides a signal light control model training device, such as Figure 4 As shown, the device may include: a data acquisition module 401, used to acquire visual images, bird's-eye views images and traffic information in a preset simulated intersection environment; a first sample construction module 402, used to construct a variety of sub-task training samples based on visual images, traffic information and preset entity categories; a first-stage training module 403, used to perform a first-stage training on the projection layer and the mapping layer in the initial model based on the various sub-task training samples to generate an intermediate model; a second sample construction module 404, used to construct a decision reasoning training sample based on the visual image and the bird's-eye view image; a second-stage training module 405, used to perform a second-stage training on the projection layer and the mapping layer in the intermediate model based on the decision reasoning training sample to generate a traffic light control model.

[0111] In some embodiments, the initial model includes: a visual encoder, a projection layer, a base large model and a mapping layer; the visual encoder is connected to the input end of the base large model through the projection layer, and the mapping layer is connected to the output end of the base large model.

[0112] In some embodiments, the data acquisition module 401 includes: a hardware setting submodule, which is used to set cameras and sensors for any direction in a simulated intersection environment; an image acquisition submodule, which is used to acquire visual images and bird's-eye views based on the camera; an information acquisition submodule, which is used to acquire vehicle flow information based on the sensor; and an information generation submodule, which is used to generate traffic information based on visual images and vehicle flow information.

[0113] In some embodiments, the first sample construction module 402 includes: a first text construction submodule, which is used to construct a subtask text corresponding to each subtask according to the entity category; a first sample construction submodule, which is used to take the visual image and the subtask text as input and the traffic information as the true value to construct a subtask training sample corresponding to each subtask.

[0114] In some embodiments, the first stage training module 403 includes: a first stage training sub-module, which is used to pre-train the projection layer based on multiple sub-task training samples to generate a first pre-selected projection layer; an intermediate model generation sub-module, which is used to fine-tune the instructions of the first pre-selected projection layer and the mapping layer based on multiple sub-task training samples to generate an intermediate model.

[0115] In some embodiments, the first stage training submodule is used to: freeze all parameters of the visual encoder, the base large model and the mapping layer in the initial model; pre-train the projection layer using a variety of sub-task training samples to generate a first pre-selected projection layer.

[0116] In some embodiments, the intermediate model generation submodule is used to: freeze all parameters of the visual encoder and the base large model in the initial model; pre-train the first pre-selected projection layer and the mapping layer according to multiple sub-task training samples to generate the second pre-selected projection layer and the first pre-selected mapping layer; generate the intermediate model according to the visual encoder, the second pre-selected projection layer, the base large model and the first pre-selected mapping layer.

[0117] In some embodiments, the second sample construction module 404 includes: an image stitching submodule, which is used to stitch the visual image according to preset stitching rules to generate a multi-view image; a second text construction submodule, which is used to construct a decision reasoning task text based on the multi-view image; an expert decision generation submodule, which is used to generate expert decision data based on the bird's-eye view image at the same time as the visual image; and a second sample construction submodule, which is used to take the visual image and the decision reasoning task text as input and the expert decision data as the true value to construct a decision reasoning training sample.

[0118] In some embodiments, the second stage training module 405 includes: a parameter freezing submodule, used to freeze all parameters of the visual encoder and the base large model of the intermediate model; a second stage training submodule, used to train the secondary pre-selected projection layer and the first pre-selected mapping layer according to the decision reasoning training samples, and generate a traffic light control projection layer and a traffic light control mapping layer; a control model generation submodule, used to generate a traffic light control model according to the visual encoder, the traffic light control projection layer, the base large model and the traffic light control mapping layer.

[0119] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0120] The signal light control model training device of the disclosed embodiment, by setting up a simulated intersection environment, can conduct a large number of experiments and training on the signal light control model without actual deployment, optimize the performance of the model, and predict the performance of the model in the real world, which can significantly reduce the training cost, improve the training efficiency, and reduce the interference with actual traffic. At the same time, by defining a variety of subtasks, complex tasks can be decomposed, thereby improving the efficiency of model training, accelerating the convergence of the model, and ultimately improving the performance of the model. In addition, by constructing decision reasoning training samples, the model can make effective decisions in a complex traffic environment only through visual images and bird's-eye images, thereby improving the versatility of the model.

[0121] The present disclosure provides a signal light control device, such as Figure 5As shown, the device may include: an information acquisition module 501, used to obtain an intersection image and an overhead view of the intersection; a decision output module 502, used to input the intersection image and the overhead view of the intersection into a traffic light control model, and obtain a traffic light control decision output by the traffic light control model; wherein the traffic light control model is obtained by training the method described above; and a decision execution module 503, used to control the traffic light phase based on the traffic light control decision.

[0122] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0123] The traffic light control device of the disclosed embodiment can directly perceive the traffic conditions in real time through image or video data and make corresponding control decisions, thereby optimizing the traffic light control and achieving more efficient and intelligent traffic management.

[0124] The present disclosure provides a schematic diagram of a traffic light control model training scenario, such as Figure 6 shown.

[0125] As mentioned above, the signal light control model training method provided by the embodiment of the present disclosure is applied to electronic devices. The electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers.

[0126] Specifically, the electronic device can perform the following operations:

[0127] In a preset simulated intersection environment, visual images, bird's-eye views and traffic information are obtained; based on the visual images, traffic information and preset entity categories, a variety of sub-task training samples are constructed; based on the various sub-task training samples, the projection layer and mapping layer in the initial model are trained in the first stage to generate an intermediate model; based on the visual images and bird's-eye views, decision reasoning training samples are constructed; based on the decision reasoning training samples, the projection layer and mapping layer in the intermediate model are trained in the second stage to generate a traffic light control model.

[0128] It should be understood that Figure 6 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Figure 6 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0129] The present disclosure provides a schematic diagram of a signal light control scenario, such as Figure 7 shown.

[0130] As mentioned above, the signal light control method provided by the embodiment of the present disclosure is applied to electronic devices. The electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers.

[0131] Specifically, the electronic device can perform the following operations:

[0132] Obtain an intersection image and an overhead view of the intersection; input the intersection image and the overhead view of the intersection into a traffic light control model to obtain a traffic light control decision output by the traffic light control model; and control the traffic light phase based on the traffic light control decision.

[0133] It should be understood that Figure 7 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Figure 7 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0134] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0135] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0136] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0137] like Figure 8As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0138] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0139] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a signal light control model training method and / or a signal light control method. For example, in some embodiments, the signal light control model training method and / or the signal light control method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the signal light control model training method and / or the signal light control method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the signal light control model training method and / or the signal light control method in any other appropriate manner (e.g., by means of firmware).

[0140] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0142] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0144] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0145] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0146] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0147] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A traffic light control model training method, comprising: In a preset simulated intersection environment, obtain visual images, bird's-eye view images and traffic information; Constructing a variety of subtask training samples based on the visual image, the traffic information, and the preset entity categories; Based on the multiple sub-task training samples, the projection layer and the mapping layer in the initial model are trained in the first stage to generate an intermediate model; Constructing a decision reasoning training sample based on the visual image and the bird's-eye view image; Based on the decision reasoning training samples, the projection layer and the mapping layer in the intermediate model are trained in the second stage to generate a traffic light control model.

2. The method according to claim 1, wherein: The initial model includes: a visual encoder, the projection layer, a large base model and the mapping layer; the visual encoder is connected to the input end of the large base model through the projection layer, and the mapping layer is connected to the output end of the large base model.

3. The method according to claim 1, wherein: The method of obtaining visual images, bird's-eye view images and traffic information in a preset simulated intersection environment includes: In the simulated intersection environment, a camera and a sensor are respectively set for any direction; Acquire the visual image and the bird's-eye view image according to the camera; acquiring vehicle flow information according to the sensor; The traffic information is generated based on the visual image and the vehicle flow information.

4. The method according to claim 1, wherein: The method of constructing a plurality of subtask training samples based on the visual image, the traffic information and the preset entity categories includes: According to the entity category, construct a subtask text corresponding to each subtask; The visual image and the subtask text are taken as input, and the traffic information is taken as the true value, to construct the subtask training samples corresponding to each subtask.

5. The method according to claim 1, wherein: The first stage training of the projection layer and the mapping layer in the initial model based on the multiple sub-task training samples to generate an intermediate model includes: Based on the multiple subtask training samples, pre-training the projection layer to generate a first pre-selected projection layer; Based on the multiple sub-task training samples, the first pre-selected projection layer and the mapping layer are fine-tuned to generate the intermediate model.

6. The method according to claim 5, wherein: The pre-training of the projection layer based on the multiple sub-task training samples to generate a first pre-selected projection layer includes: Freeze all parameters of the visual encoder, the base large model and the mapping layer in the initial model; The projection layer is pre-trained using the multiple sub-task training samples to generate the first pre-selected projection layer.

7. The method according to claim 5, wherein: The step of fine-tuning the first pre-selected projection layer and the mapping layer based on the multiple sub-task training samples to generate the intermediate model includes: Freeze all parameters of the visual encoder and the base large model in the initial model; Pre-training the first pre-selected projection layer and the mapping layer according to the plurality of sub-task training samples to generate a second pre-selected projection layer and a first pre-selected mapping layer; The intermediate model is generated according to the visual encoder, the secondary pre-selected projection layer, the base large model and the first pre-selected mapping layer.

8. The method according to claim 7, wherein: The step of performing a second-stage training on the projection layer and the mapping layer in the intermediate model based on the decision reasoning training sample to generate a traffic light control model includes: Freeze all parameters of the visual encoder of the intermediate model and the base large model; According to the decision reasoning training samples, the secondary pre-selected projection layer and the primary pre-selected mapping layer are trained to generate a signal light control projection layer and a signal light control mapping layer; The signal light control model is generated according to the visual encoder, the signal light control projection layer, the base large model and the signal light control mapping layer.

9. The method according to claim 1, wherein: The step of constructing a decision reasoning training sample based on the visual image and the bird's-eye view image includes: splicing the visual images according to a preset splicing rule to generate a multi-view image; Constructing a decision reasoning task text according to the multi-view images; generating expert decision data based on the bird's-eye view image at the same time as the visual image; The visual image and the decision reasoning task text are taken as input, and the expert decision data is taken as the true value to construct the decision reasoning training sample.

10. A signal light control method, comprising: Obtain intersection images and intersection bird's-eye views; Inputting the intersection image and the intersection bird's-eye view into a traffic light control model to obtain a traffic light control decision output by the traffic light control model; wherein the traffic light control model is obtained by training using the method described in any one of claims 1 to 9; A signal light phase is controlled based on the signal light control decision.

11. A signal light control model training device, comprising: A data acquisition module, used to acquire visual images, bird's-eye view images and traffic information in a preset simulated intersection environment; A first sample construction module, used to construct a plurality of subtask training samples based on the visual image, the traffic information and the preset entity category; A first-stage training module, used to perform first-stage training on the projection layer and the mapping layer in the initial model based on the multiple sub-task training samples to generate an intermediate model; A second sample construction module is used to construct a decision reasoning training sample based on the visual image and the bird's-eye view image; The second stage training module is used to perform second stage training on the projection layer and the mapping layer in the intermediate model based on the decision reasoning training samples to generate a traffic light control model.

12. The device according to claim 11, wherein: The initial model includes: a visual encoder, the projection layer, a large base model and the mapping layer; the visual encoder is connected to the input end of the large base model through the projection layer, and the mapping layer is connected to the output end of the large base model.

13. The device according to claim 11, wherein: The data acquisition module comprises: A hardware setting submodule, used for setting a camera and a sensor in any direction in the simulated intersection environment; An image acquisition submodule, used for acquiring the visual image and the bird's-eye view image according to the camera; An information acquisition submodule, used to acquire vehicle flow information according to the sensor; The information generation submodule is used to generate the traffic information based on the visual image and the vehicle flow information.

14. The device according to claim 11, wherein: The first sample construction module includes: A first text construction submodule is used to construct a subtask text corresponding to each subtask according to the entity category; The first sample construction submodule is used to take the visual image and the subtask text as input and the traffic information as the true value to construct the subtask training sample corresponding to each subtask.

15. The device according to claim 11, wherein The first phase training module includes: The first-stage training submodule is used to pre-train the projection layer based on the multiple sub-task training samples to generate a first pre-selected projection layer; The intermediate model generation submodule is used to perform instruction fine-tuning on the first pre-selected projection layer and the mapping layer based on the multiple sub-task training samples to generate the intermediate model.

16. The device according to claim 15, wherein: The first stage training submodule is used to: Freeze all parameters of the visual encoder, the base large model and the mapping layer in the initial model; The projection layer is pre-trained using the multiple sub-task training samples to generate the first pre-selected projection layer.

17. The device according to claim 15, wherein: The intermediate model generation submodule is used to: Freeze all parameters of the visual encoder and the base large model in the initial model; Pre-training the first pre-selected projection layer and the mapping layer according to the plurality of sub-task training samples to generate a second pre-selected projection layer and a first pre-selected mapping layer; The intermediate model is generated according to the visual encoder, the secondary pre-selected projection layer, the base large model and the first pre-selected mapping layer.

18. The device according to claim 17, wherein: The second phase training module includes: A parameter freezing submodule, used to freeze all parameters of the visual encoder of the intermediate model and the base large model; The second-stage training submodule is used to train the secondary pre-selected projection layer and the primary pre-selected mapping layer according to the decision reasoning training samples to generate a signal light control projection layer and a signal light control mapping layer; A control model generation submodule is used to generate the signal light control model according to the visual encoder, the signal light control projection layer, the base large model and the signal light control mapping layer.

19. The device according to claim 11, wherein: The second sample construction module comprises: An image stitching submodule, used to stitch the visual images according to a preset stitching rule to generate a multi-view image; A second text construction submodule is used to construct a decision reasoning task text according to the multi-view images; An expert decision generating submodule, for generating expert decision data according to the bird's-eye view image at the same time as the visual image; The second sample construction submodule is used to take the visual image and the decision reasoning task text as input and the expert decision data as the true value to construct the decision reasoning training sample.

20. A signal light control device, comprising: An information collection module is used to obtain intersection images and bird's-eye views of intersections; A decision output module, used for inputting the intersection image and the intersection bird's-eye view into a traffic light control model to obtain a traffic light control decision output by the traffic light control model; wherein the traffic light control model is obtained by training using the method described in any one of claims 1 to 9; A decision execution module is used to control the phase of the signal light based on the signal light control decision.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to at least one processor; wherein, The memory stores instructions executable by at least one processor, and the instructions are executed by at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: Computer instructions for causing a computer to perform a method according to any one of claims 1 to 10.

23. A computer program product comprising a computer program stored on a storage medium, the computer program implementing the method according to any one of claims 1 to 10 when executed by a processor.

Images