Traffic signal tuning method, terminal and equipment based on multi-modal large model

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a traffic signal tuning method based on a multi-modal large model, a terminal and equipment. Obtaining to-be-analyzed intersection information carried in the traffic signal analysis instruction, determining each target lane allowed to pass at each signal lamp phase corresponding to the to-be-analyzed intersection information, and counting a first vehicle number of each target lane at the beginning of each signal lamp phase and a second vehicle number of each target lane at the end of each signal lamp phase, the method comprises the following steps of: analyzing a first vehicle number at the beginning of each signal lamp phase, a second vehicle number at the end of each signal lamp phase and a third vehicle number passing through each target lane during each signal lamp phase, and analyzing a first vehicle number at the beginning of each signal lamp phase, a second vehicle number at the end of each signal lamp phase and the third vehicle number passing through each signal lamp phase based on a multi-modal large model to obtain an analysis result; according to the method, the number of vehicles in each stage of each signal lamp phase is counted, the number of each vehicle is analyzed by using the multi-modal large model, whether traffic signal tuning is needed is determined, and the efficiency of traffic signal tuning is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a traffic signal optimization method, terminal and device based on a multimodal large model. Background Art

[0002] Road traffic is the main mode of daily travel, and the increasing number of vehicles has caused serious traffic congestion. Traffic signal tuning is to solve or alleviate congestion to the greatest extent possible by controlling traffic lights.

[0003] Currently, the most commonly used method is to have professional signal tuning engineers watch surveillance videos at traffic intersections to find problems, and conduct long-term tracking and analysis of surveillance videos of the same intersection or multiple intersections in different time periods and holidays to find problems and then develop corresponding solutions. This method is very labor-intensive and time-consuming, and the efficiency of traffic signal tuning is extremely low.

[0004] Therefore, how to improve the efficiency of traffic signal optimization is an urgent problem to be solved. Summary of the invention

[0005] The embodiments of the present application provide a traffic signal tuning method, terminal and device based on a multimodal large model to solve the problem of low efficiency of traffic signal tuning in the prior art.

[0006] The present application provides a traffic signal tuning method based on a multimodal large model, the method comprising:

[0007] If a traffic signal analysis instruction is received, obtaining the intersection information to be analyzed carried in the traffic signal analysis instruction;

[0008] For each signal light phase corresponding to the intersection information to be analyzed, determine each target lane that is allowed to pass through the signal light phase; count the number of first vehicles located in each target lane at the beginning of the signal light phase in the high point monitoring video corresponding to the intersection information to be analyzed, the number of second vehicles located in each target lane at the end of the signal light phase, and the number of third vehicles passing through each target lane during the signal light phase in the electric alarm monitoring video corresponding to the intersection information to be analyzed;

[0009] The first number of vehicles, the second number of vehicles and the third number of vehicles are analyzed based on a multimodal large model to determine an analysis result, wherein the analysis result is used to describe whether the intersection corresponding to the intersection information to be analyzed needs to be optimized within a time range corresponding to the phase of the traffic light.

[0010] The present application provides a traffic signal tuning terminal, the terminal comprising a display and a processor;

[0011] The processor is configured to:

[0012] If a traffic signal analysis instruction is received, obtaining the intersection information to be analyzed carried in the traffic signal analysis instruction;

[0013] For each signal light phase corresponding to the intersection information to be analyzed, determine each target lane that is allowed to pass through the signal light phase; count the number of first vehicles located in each target lane at the beginning of the signal light phase in the high point monitoring video corresponding to the intersection information to be analyzed, the number of second vehicles located in each target lane at the end of the signal light phase, and the number of third vehicles passing through each target lane during the signal light phase in the electric alarm monitoring video corresponding to the intersection information to be analyzed;

[0014] Analyze the first number of vehicles, the second number of vehicles, and the third number of vehicles based on a multimodal large model to determine an analysis result, wherein the analysis result is used to describe whether an intersection corresponding to the intersection information to be analyzed needs to be optimized within a time range corresponding to the phase of the traffic light;

[0015] The display is configured to display the analysis result of the intersection of the intersection information to be analyzed.

[0016] The present application provides a traffic signal optimization device based on a multimodal large model, the device comprising:

[0017] An acquisition module, configured to acquire the intersection information to be analyzed carried in the traffic signal analysis instruction if a traffic signal analysis instruction is received;

[0018] A statistical module is used to determine, for each signal light phase corresponding to the intersection information to be analyzed, each target lane that is allowed to pass through the signal light phase; count the number of first vehicles located in each target lane at the beginning of the signal light phase in the high point monitoring video corresponding to the intersection information to be analyzed, the number of second vehicles located in each target lane at the end of the signal light phase, and the number of third vehicles passing through each target lane during the signal light phase in the electric alarm monitoring video corresponding to the intersection information to be analyzed;

[0019] An analysis module is used to analyze the first number of vehicles, the second number of vehicles and the third number of vehicles based on a multimodal large model to determine an analysis result, wherein the analysis result is used to describe whether the intersection corresponding to the intersection information to be analyzed needs to be optimized within a time range corresponding to the phase of the traffic light.

[0020] The present application also provides an electronic device, which includes a processor, and the processor is used to implement the steps of any of the above-mentioned traffic signal optimization methods based on a multimodal large model when executing a computer program stored in a memory.

[0021] The present application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned traffic signal optimization methods based on a multimodal large model.

[0022] In the embodiment of the present application, the information of the intersection to be analyzed carried in the traffic signal analysis instruction is obtained, the target lanes allowed to pass at each signal light phase corresponding to the information of the intersection to be analyzed are determined, and the number of first vehicles in each target lane at the beginning of each signal light phase, the number of second vehicles in each target lane at the end, and the number of third vehicles passing through each target lane during each signal light phase are counted, and then the number of first vehicles at the beginning of each signal light phase, the number of second vehicles at the end, and the number of third vehicles passing through during each signal light phase are analyzed based on the multimodal large model to obtain the analysis result, and the monitoring video is analyzed to count the number of vehicles in each stage of each signal light phase, and the multimodal large model is used to analyze each number of vehicles to determine whether traffic signal optimization is needed, thereby improving the efficiency of traffic signal optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 A schematic diagram of a traffic signal optimization process based on a multimodal large model provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of lanes at a traffic intersection provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of a stop line at an intersection provided in an embodiment of the present application;

[0027] Figure 4 A lane schematic diagram provided for an embodiment of the present application;

[0028] Figure 5a A schematic diagram of a vehicle crossing the line provided in an embodiment of the present application;

[0029] Figure 5b A schematic diagram of a vehicle crossing the line provided in an embodiment of the present application;

[0030] Figure 6 A rectangular schematic diagram provided for an embodiment of the present application;

[0031] Figure 7A schematic diagram of a traffic signal tuning process provided in an embodiment of the present application;

[0032] Figure 8 A schematic diagram of a traffic signal tuning page provided in an embodiment of the present application;

[0033] Fig. 9 A schematic diagram of the structure of a traffic signal tuning terminal provided in an embodiment of the present application;

[0034] Fig.10 A schematic diagram of the structure of a traffic signal optimization device based on a multimodal large model provided in an embodiment of the present application;

[0035] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the embodiment of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiment is only a part of the embodiment of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0037] The embodiment of the present application provides a traffic signal tuning method, terminal and equipment based on a multimodal large model. In the method, if a traffic signal analysis instruction is received, the information of the intersection to be analyzed carried in the traffic signal analysis instruction is obtained; for each traffic light phase corresponding to the intersection information to be analyzed, each target lane that is allowed to pass through the traffic light phase is determined; the number of first vehicles located in each target lane at the beginning of the traffic light phase in the high-point monitoring video corresponding to the intersection information to be analyzed, the number of second vehicles located in each target lane at the end of the traffic light phase, and the number of third vehicles passing through each target lane during the traffic light phase in the electric alarm monitoring video corresponding to the intersection information to be analyzed are counted; the first number of vehicles, the second number of vehicles and the third number of vehicles are analyzed based on the multimodal large model to determine the analysis result, and the analysis result is used to describe whether the intersection corresponding to the intersection information to be analyzed needs to be tuned within the time range corresponding to the traffic light phase.

[0038] Figure 1 A schematic diagram of a traffic signal optimization process based on a multimodal large model provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the process includes the following steps:

[0039] S101: If a traffic signal analysis instruction is received, obtain the intersection information to be analyzed carried in the traffic signal analysis instruction.

[0040] The traffic signal tuning method based on a multimodal large model provided in an embodiment of the present application is applied to an electronic device, which may be a server, a PC, a smart terminal, etc.

[0041] In order to find out whether there is a problem with the traffic signal in time, in the embodiment of the present application, the user can select the intersection to be analyzed in the visual interface, and the intersection selected by the user can be used as the intersection to be analyzed, and the intersection information to be analyzed corresponding to the intersection to be analyzed can be carried in the traffic signal analysis instruction and sent to the electronic device. If the electronic device receives the traffic signal analysis instruction, it can obtain the intersection information to be analyzed carried in the traffic signal analysis instruction, and then obtain the corresponding monitoring video based on the intersection information to be analyzed for analysis.

[0042] S102: For each traffic light phase corresponding to the intersection information to be analyzed, determine each target lane that is allowed to pass through the traffic light phase; count the number of first vehicles located in each target lane at the beginning of the traffic light phase in the high-point monitoring video corresponding to the intersection information to be analyzed, the number of second vehicles located in each target lane at the end of the traffic light phase, and the number of third vehicles passing through each target lane during the traffic light phase in the electric alarm monitoring video corresponding to the intersection information to be analyzed.

[0043] Since there are multiple signal light phases at an intersection, the target lanes allowed to pass are different at different signal light phases. For example, the phases corresponding to the intersection information to be analyzed may be straight up, straight down, left turn up, left turn down, right turn up, right turn down; straight left, left turn left, left turn right, straight right, right turn right, etc.

[0044] In order to analyze whether the current traffic signal needs to be optimized, after obtaining the intersection information to be analyzed, each signal light phase saved for the intersection information to be analyzed can be obtained. For each signal light phase, each target lane that is allowed to pass by the signal light phase can be determined.

[0045] In order to count the traffic volume during each traffic light phase and determine whether the traffic light phase needs to be optimized according to the traffic volume, the high-point monitoring video and the electric alarm monitoring video of the corresponding intersection can be obtained according to the information of the intersection to be analyzed.

[0046] Since the high-point monitoring video is generally collected by an image acquisition device installed at a higher position near the intersection, the high-point monitoring video can capture relatively complete information about the intersection. In the embodiment of the present application, the number of first vehicles located in each target lane at the beginning of the signal light phase in the high-point monitoring video and the number of second vehicles located in each target lane at the end of the signal light phase can be counted. In the embodiment of the present application, the lanes above the screen can be called upper straight, upper right turn, and upper left turn, and the same applies to other different intersections. At the beginning of the phase, the number of vehicles in the current phase combination lane is counted. For example, the current phase is upper straight-up-up left turn-lower straight-down-lower left turn, which means that all four lanes are open to traffic. That is, when the green light in the upper and lower directions (straight & left turn) just turns on, extract the high-point monitoring video frame, first locate the lane areas in different directions through segmentation and lane line detection, and count the total number of lanes in the lane area in the direction of the phase through target detection. Figure 2 A schematic diagram of a traffic intersection lane provided in an embodiment of the present application, such as Figure 2 As shown in the figure, the phase information at this time is upper straight-up right turn-lower straight-down right turn. The area shown by the dotted line in the figure is located through road segmentation and lane line detection. Then the number of vehicles in these two areas is detected by the vehicle target detection model. The structured output is: there are currently X vehicles in the upper straight-up right turn lane and X vehicles in the lower straight-down right turn lane.

[0047] In addition, since the electric police monitoring video is generally collected by an image acquisition device installed above the lane, the electric police monitoring video can clearly capture the information of vehicles entering and exiting each lane. In an embodiment of the present application, the number of third vehicles passing through each target lane during the signal light phase in the electric police monitoring video can be counted.

[0048] S103: Analyze the first number of vehicles, the second number of vehicles, and the third number of vehicles based on a multimodal large model to determine an analysis result, wherein the analysis result is used to describe whether the intersection corresponding to the intersection information to be analyzed needs to be optimized within a time range corresponding to the phase of the traffic light.

[0049] After obtaining the first number of vehicles, the second number of vehicles, and the third number of vehicles, the first number of vehicles, the second number of vehicles, and the third number of vehicles can be analyzed based on the multimodal large model to determine the analysis result. The analysis result is used to describe whether the intersection corresponding to the intersection information to be analyzed needs to be optimized within the time range corresponding to the phase of the signal light. For example, the analysis result can be "very few vehicles pass through the lower straight lane, and sometimes even no vehicles pass through, and the utilization rate is low."

[0050] Specifically, a prompt word containing at least the first number of vehicles, the second number of vehicles, and the third number of vehicles can be input into the multimodal large model, and the multimodal large model analyzes the prompt word to determine the analysis result. Exemplarily, the prompt word can be: "The lane currently released is up straight-up-up right turn-down straight-down right turn. Before release, there are X vehicles in the up straight-up-up right turn lane, and X vehicles in the down straight-down right turn lane. During the release, the number of up right turn vehicles passing through is X, the number of up straight-up vehicles passing through is X, the number of down straight-down vehicles passing through is X, and the number of down right turn vehicles passing through is X. After the release, there are X vehicles in the up straight-up-up right turn lane, and there are X vehicles in the down straight-down right turn lane. Please analyze whether there are phase mismatch, low lane utilization, and lane queue overflow problems at the intersection in the video, and give an explanation."

[0051] In one possible implementation, after accumulating a certain number of statistical results for each traffic light phase, each statistical result and the corresponding video may be input into a large multimodal model for processing. That is, the large multimodal model may consider the statistical results of vehicles in multiple different time periods during analysis to improve the accuracy of signal optimization.

[0052] In the embodiment of the present application, the information of the intersection to be analyzed carried in the traffic signal analysis instruction is obtained, the target lanes allowed to pass at each signal light phase corresponding to the information of the intersection to be analyzed are determined, and the number of first vehicles in each target lane at the beginning of each signal light phase, the number of second vehicles in each target lane at the end, and the number of third vehicles passing through each target lane during each signal light phase are counted, and then the number of first vehicles at the beginning of each signal light phase, the number of second vehicles at the end, and the number of third vehicles passing through during each signal light phase are analyzed based on the multimodal large model to obtain the analysis result, and the monitoring video is analyzed to count the number of vehicles in each stage of each signal light phase, and the multimodal large model is used to analyze each number of vehicles to determine whether traffic signal optimization is needed, thereby improving the efficiency of traffic signal optimization.

[0053] In order to improve the accuracy of traffic signal analysis, based on the above embodiment, in the embodiment of the present application, the number of third vehicles passing through each target lane during the phase of the traffic light in the electric police monitoring video corresponding to the intersection information to be analyzed is counted, including:

[0054] Obtaining the stop line position information pre-saved for the intersection information to be analyzed, and determining the stop line in the intersection picture captured by the electric police monitoring video according to the stop line position information;

[0055] For each target lane, if any vehicle crosses the stop line of the exit lane corresponding to the target lane and crosses the stop line of the entrance lane corresponding to the target lane during the signal light phase, it is determined that the vehicle passes through the target lane and the third vehicle number is updated.

[0056] Due to the long-term disrepair of some roads, the stop line in the intersection picture taken by the police surveillance video may be unclear, thereby affecting the accuracy of the vehicle count. Therefore, in the embodiment of the present application, the corresponding stop line position information can be saved in advance for each intersection. Since the stop line is generally a straight line and the position of the camera is fixed, the pre-saved stop line position information can be stored in the form of a straight line equation.

[0057] When counting the third number of vehicles, the stop line position information saved for the intersection information to be analyzed can be obtained, so as to determine the stop line in the intersection picture taken by the electric police monitoring video based on the stop line position information. Figure 3 A schematic diagram of a stop line at an intersection provided in an embodiment of the present application, Figure 3 The line segments 1-7 in the figure are the stop lines of the corresponding lanes.

[0058] When a vehicle passes through an intersection, it is considered to have successfully passed the lane only when it passes through the exit lane and the entry lane. Figure 3 The line segments 1 and 8 shown in the figure are considered to have passed the target lane; when the vehicle turns left on the Figure 3 The vehicle must pass through the target lane only after passing through line segments 1 and 4 shown in the figure. Figure 3 The line segments 1 and 6 shown in are considered to have passed the target lane. Therefore, in the embodiment of the present application, when counting the third number of vehicles, for each target lane, if there is any vehicle that crosses the stop line of the exit lane corresponding to the target lane during the signal light phase, and crosses the stop line of the entry lane corresponding to the target lane, then it can be determined that the vehicle has passed the target lane, and the third number of vehicles is updated. Exemplarily, the information of the third number of vehicles finally counted can be: the number of vehicles turning right passing on the upper side X, the number of vehicles going straight on the upper side X, the number of vehicles going straight on the lower side X, and the number of vehicles turning right passing on the lower side X.

[0059] Figure 4 A lane schematic diagram provided in an embodiment of the present application, assuming that the signal light phase is upward straight, then the target lane corresponding to the signal light phase is Figure 4 The lane marked by the shaded area in the middle. The stop line of the exit lane corresponding to the target lane is line segment 1, and the stop line of the entry lane corresponding to the target lane is line segment 6. When any vehicle crosses both line segment 1 and line segment 6, it can be determined that the vehicle has passed the target lane, and the third vehicle number can be increased by 1.

[0060] In order to further improve the accuracy of traffic signal analysis, based on the above embodiments, in the embodiment of the present application, the process of determining whether any vehicle crosses the stop line of the corresponding lane includes:

[0061] Determine a corresponding vehicle detection frame in the image frame where the vehicle is located;

[0062] If any end point of the stop line is located within the vehicle detection frame, it is determined that the vehicle crosses the stop line.

[0063] In an embodiment of the present application, when determining whether a vehicle crosses the line, it can be determined whether the line segment is crossed based on the coordinates of the starting and ending points of the rectangular frame (vehicle detection frame) and the line segment (line segment fixedly marked in the video screen). In order to exclude special cases, before detecting the intersection of the line segment and the diagonal line, it is possible to detect whether the endpoints of the line segment are within the rectangle. Specifically, in order to avoid the inaccuracy of the stop line in the intersection picture in the determined electric police monitoring video, and the wrong number of vehicles being counted, or the wrong number of vehicles being counted because of irregular vehicle driving, in an embodiment of the present application, when determining whether any vehicle crosses the stop line of the corresponding lane, the corresponding vehicle detection frame in the image frame where the vehicle is located can be determined. If either endpoint of the stop line is located within the vehicle detection frame, it can be determined that the vehicle has crossed the stop line.

[0064] Figure 5a-Figure 5b A schematic diagram of a vehicle crossing line provided in an embodiment of the present application, such as Figure 5a As shown, the rectangular frame in the figure can be understood as a vehicle detection frame corresponding to a vehicle. Figure 5a The endpoint on the right side of the middle line segment is within the vehicle detection frame, so it can be considered that the vehicle has crossed the stop line. Figure 5b The endpoint on the left side of the middle line segment is located within the vehicle detection frame, and it can be considered that the vehicle has crossed the stop line.

[0065] Specifically, assume that the coordinates of the lower left corner of the rectangular box are (x1, y1), and the coordinates of the upper right corner are (x2, y2). The coordinates of the endpoints on one side of the line segment are (L1[0], L1[1]), and the coordinates of the endpoints on the other side are (L2[0], L2[1]). When the positional relationship between the line segment and the rectangular box satisfies the following formula, it can be determined that the endpoints of the line segment are within the rectangular box, that is, any endpoint of the stop line is within the vehicle detection box:

[0066] (L1[0]>=sq[0]and L1[1]>=sq[1]and L1[0]<=sq[2]and L1[1]<=sq[3])or

[0067] (L2[0]>=sq[0]and L2[1]>=sq[1]and L2[0]<=sq[2]and L2[1]<=sq[3])

[0068] Among them, L1[0] represents the horizontal coordinate of the endpoint on one side of the line segment; L1[1] represents the vertical coordinate of the endpoint on one side of the line segment; L2[0] represents the horizontal coordinate of the endpoint on the other side of the line segment; L2[1] represents the vertical coordinate of the endpoint on the other side of the line segment; sq[0] represents the horizontal coordinate of the lower left corner of the rectangular box; sq[1] represents the vertical coordinate of the lower left corner of the rectangular box; sq[2] represents the horizontal coordinate of the upper right corner of the rectangular box; sq[3] represents the vertical coordinate of the upper right corner of the rectangular box.

[0069] In order to further improve the accuracy of traffic signal analysis, based on the above embodiments, in the embodiment of the present application, the process of determining whether any vehicle crosses the stop line of the corresponding lane includes:

[0070] Determine a corresponding vehicle detection frame in the image frame where the vehicle is located;

[0071] If the diagonal line of the vehicle detection frame intersects the stop line, it is determined that the vehicle crosses the stop line.

[0072] In order to accurately determine whether a vehicle crosses the line, in the embodiment of the present application, the problem of the vehicle crossing the line can be converted into the problem of the intersection of the line segment and the two diagonals of the rectangle. In the embodiment of the present application, when determining whether any vehicle crosses the stop line of the corresponding lane, the corresponding vehicle detection frame in the image frame where the vehicle is located can be determined. If the stop line intersects with the diagonal line of the vehicle detection frame, it can be determined that the vehicle has crossed the stop line.

[0073] Specifically, when judging the intersection of line segments, it can be divided into two steps: rectangle judgment and cross product judgment.

[0074] Rectangle determination: By judging whether the two rectangles with two line segments as diagonals intersect, if they do not intersect, the two line segments must not intersect, and directly outputting non-intersection, that is, the vehicle did not complete the crossing. The conditions for whether the two rectangles intersect are: the rightmost end of any rectangle is greater than the leftmost end of the other rectangle, and the highest end of any rectangle is greater than the lowest end of the other rectangle; as long as any of these conditions is not met, the two rectangles do not intersect, that is, the two line segments do not intersect. Figure 6 A rectangular schematic diagram provided in an embodiment of the present application, such as Figure 6 As shown, Figure 6 The left and right sides (left and right in the diagram) include two rectangles. Figure 6 The rightmost end of the rectangle on the center left (left and right in the diagram) side is no larger than Figure 6 The leftmost end of the rectangle on the middle right (left and right in the figure) side, therefore, does not meet the condition that the two rectangles intersect, and it can be determined that the two line segments AB and CD do not intersect.

[0075] Cross product determination: The two line segments that have not obtained a result after the above determination enter this step for determination. The purpose of this step is to determine whether the two endpoints of a line segment are on both sides of the other line segment. If the endpoints of the two line segments are on both sides of the other line segment, the two line segments intersect.

[0076] For example, if it is determined whether points P3 and P4 are located on both sides of the diagonal line P1P2, that is, it is divided into two parts to determine (p1, p2, p3) and (p1, p2, p4). Specifically, the determination can be based on the following formula:

[0077] For the judgment of (p1, p2, p3):

[0078] x1=p2[0]-p1[0]; y1=p2[1]-p1[1]; x2=p3[0]-p1[0]; y2=p3[1]-p1[1].

[0079] Judgment of (p1, p2, p4):

[0080] x1=p2[0]-p1[0]; y1=p2[1]-p1[1]; x2=p4[0]-p1[0]; y2=p4[1]-p1[1].

[0081] Among them, p2[0] represents the x-coordinate of P2; p2[1] represents the y-coordinate of P2; p1[0] represents the x-coordinate of P1; p1[1] represents the y-coordinate of P1; p3[0] represents the x-coordinate of P3; p3[1] represents the y-coordinate of P3; p4[0] represents the x-coordinate of P4; p4[1] represents the y-coordinate of P4.

[0082] Based on the above two formulas, x1*y2-x2*y1 is calculated to obtain cross1 and cross2.

[0083] If cross1*cross2<0, it proves that points p3 and p4 are on both sides of the line segment with p1 and p2 as endpoints.

[0084] Similarly, the principle of judging whether p1 and p2 are on both sides of the diagonal line p3 p4 (p3, p4, p1) and (p3, p4, p2) is the same as above.

[0085] In order to further improve the accuracy of traffic signal analysis, on the basis of the above embodiments, in the embodiment of the present application, the target recognition large model is obtained by fine-tuning training based on the instruction data set;

[0086] The construction process of the instruction data set includes:

[0087] Get sample video;

[0088] Analyze the sample video based on the pre-trained large model to determine the target phase of the sample video;

[0089] splicing the target phase, the vehicle information during the target phase corresponding to the sample video, and the preset prompt word to obtain a sample prompt word;

[0090] Input the sample prompt words and the sample video into the pre-trained large model to obtain a sample analysis result;

[0091] The sample video, the sample prompt words and the sample analysis results are stored in the instruction data set.

[0092] In order to further improve the accuracy of traffic signal tuning, the multimodal large model described in the above embodiments can be fine-tuned and trained. In an embodiment of the present application, an instruction data set for the specified task scenario of the business can be constructed, and then a multimodal vertical domain large model can be obtained by fine-tuning LORA (Low-Rank Adaptation), which can be used for the signal tuning process described in the above embodiments. The construction process of the instruction data set is described below.

[0093] When constructing the instruction dataset, sample videos may be obtained, which may be collected in advance by relevant staff.

[0094] After obtaining the sample video, the sample video can be analyzed based on the pre-trained large model to determine the target phase of the sample video. The pre-trained large model can be any multi-modal large model with video analysis capabilities, and those skilled in the art can configure it as needed.

[0095] After the target phase is obtained, the target phase, the vehicle information during the target phase corresponding to the sample video, and the preset prompt words can be spliced ​​to obtain the sample prompt words. The vehicle information may include the number of vehicles at each stage during the target phase, such as the number of vehicles in the corresponding lane at the beginning of the target phase, the number of vehicles in the corresponding lane at the end of the target phase, and the number of vehicles passing the corresponding lane during the target phase.

[0096] After obtaining the sample prompt word, the sample prompt word and the sample video can be input into the pre-trained large model to obtain the sample analysis result. Among them, the pre-trained large model can be ChatGPT4o, of course, it can also be other large models, and technicians in this field can configure it as needed. After obtaining the sample analysis result, the sample video, sample prompt word and sample analysis result can be saved in the instruction data set.

[0097] Specifically, we can first use the pre-trained large model to analyze the sample video and determine the target phase:

[0098] Video+prompt: Please analyze which lane is in the open state based on the video.

[0099] Answer: The lanes currently open are go straight up - turn right up - go straight down - turn right down.

[0100] The sample data stored in the instruction data set can be as follows:

[0101] Definition of intersection signal problem:

[0102] 1. Phase mismatch refers to the mismatch between the phase setting of the traffic light and the actual lane layout, and there is a phenomenon of no vehicles passing through the lane or secondary queues during the phase cycle. In other words, the vehicles waiting in line at the beginning of the phase have not completely passed the intersection at the end of the phase.

[0103] 2. The lane utilization rate is low. Few vehicles use the lane during the phase cycle, and no vehicles pass through most of the time.

[0104] 3. Lane queue overflow: refers to the situation where one direction of the intersection is completely or partially blocked by one or more vehicles, resulting in the situation that vehicles in other directions cannot pass, and vehicles are stuck in the middle of the intersection.

[0105] Video+prompt: The lanes currently released are straight-up-right-down-straight-down-right-down. Before release, there are X vehicles in the straight-up-right-down lane, and X vehicles in the straight-down-right-down lane. During release, X vehicles passed through the straight-up lane, the straight-up lane, the straight-down lane, and the right-down lane. After release, there are X vehicles in the straight-up-right-down lane, and X vehicles in the straight-down-right-down lane. Please analyze whether the above three problems exist at the intersection in the video. Give an explanation.

[0106] Answer:

[0107] From the video images and statistical data, we can analyze that the up and down straight lanes and right turn lanes are currently in a released state. By comparing multiple video frames, we can draw the following conclusions:

[0108] 1. **Go straight down the lane** (the straight down lane at the bottom of the screen):

[0109] - In many video frames, there are very few vehicles passing through the lower straight lane, and sometimes there are even no vehicles passing through, indicating low utilization.

[0110] 2. **Go straight ahead** (the straight ahead lane at the top of the screen):

[0111] -In multiple video frames, there are some vehicles passing in the upper straight lane, but overall the number of vehicles is not large and the utilization rate is low, but slightly better than the lower straight lane.

[0112] Therefore, from the analysis of these video frames, the utilization rate of the **lower straight lane** is even lower. The number of vehicles passing through the lower straight lane during the green light period is very small, and no vehicles pass through most of the time. It is judged that the lower straight lane has a problem of low lane utilization.

[0113] The multimodal large model is trained with the above constructed data to enable it to detect signal problems at intersections. Given a video, the structured output data of the number of cars obtained by the small model is combined with the phase information and input into the multimodal large model to identify signal problems at intersections. The multimodal large model will output conclusions and reasoning basis.

[0114] The traffic signal optimization process is described below with reference to a specific embodiment. Figure 7 A schematic diagram of a traffic signal tuning process provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, first obtain each signal light phase and high-point monitoring video corresponding to the intersection information to be analyzed. For each signal light phase, count the number of first vehicles in each target lane at the beginning of the signal light phase based on the segmentation algorithm and the target detection algorithm. And obtain the electric police monitoring video corresponding to the intersection information to be analyzed, and count the number of third vehicles that successfully pass through the target lanes in each direction during the signal light phase based on the cross-line detection. In the high-point monitoring video, count the number of second vehicles in each target lane at the end of the signal light phase based on the segmentation algorithm and the target detection algorithm. Analyze the first number of vehicles, the second number of vehicles, and the third number of vehicles based on the Vision Language Model (VLM) to determine the analysis result.

[0115] In an embodiment of the present application, a small model is used in combination with the phase information of traffic lights to analyze surveillance videos from different angles of the intersection to obtain structured information (the number of waiting vehicles, the number of vehicles turning left / right / going straight through the intersection, and the number of vehicles stuck at the intersection). The structured information is combined with the large model to analyze the utilization of the intersection (whether it is empty, low utilization, overflow, phase mismatch, etc.), which greatly saves manpower and time costs.

[0116] In the embodiment of the present application, the traffic signal optimization method based on the multimodal large model is mainly used to analyze the traffic volume and traffic signal phase of various intersections, find out the existing problems, and help further optimize the signal to make the traffic flow smoother. Through the combination of large and small models, the accuracy of the small model in cardinality is used to assist the large model in completing the identification of intersection signal scheme problems, which greatly improves the accuracy and efficiency of the analysis.

[0117] The traffic signal optimization process is described below with reference to a specific embodiment. Figure 8 A schematic diagram of a traffic signal optimization page provided in an embodiment of the present application, such as Figure 8 As shown, the user can select the intersection to be analyzed on the page. The electronic device automatically divides the intersection surveillance video into phase segments according to the phase of the signal light. The divided video will use a small model to obtain quantitative structured information according to the above scheme. For example, "Before release, there are X vehicles in the upper straight-up and upper right turn lanes, and X vehicles in the lower straight-down and lower right turn lanes. During the release period, the number of vehicles passing through the upper right turn, the number of vehicles passing through the upper straight, the number of vehicles passing through the lower straight, and the number of vehicles passing through the lower right turn. After the release, there are X vehicles in the upper straight-up and upper right turn lanes, and X vehicles in the lower straight-down and lower right turn lanes."

[0118] Phase segment videos and structured information are input into a large multimodal model to analyze and discover signal problems at intersections. Figure 8 The analyzed intersection video is displayed in the video frame on the left side (left and right in the diagram). Figure 8 The right side (left and right in the diagram) of the multimodal large model shows the analysis output. The video clips with problems are saved to the specified directory together with the analysis results for easy viewing. If there are newly defined intersection problems, you can also enter them in the text box on the right to redefine new signal problems, and let the large model combine the video and data to determine whether the problem exists and give an explanation.

[0119] Based on the same inventive concept, the embodiment of the present application also provides a traffic signal tuning terminal. Fig. 9 A schematic diagram of the structure of a traffic signal tuning terminal provided in an embodiment of the present application, the traffic signal tuning terminal 90 includes a display 901 and a processor 902; the processor 902 is configured to: if a traffic signal analysis instruction is received, obtain the intersection information to be analyzed carried in the traffic signal analysis instruction; for each signal light phase corresponding to the intersection information to be analyzed, determine each target lane that is allowed to pass through the signal light phase; count the number of first vehicles located in each target lane at the beginning of the signal light phase in the high point monitoring video corresponding to the intersection information to be analyzed, the number of second vehicles located in each target lane at the end of the signal light phase, and the number of third vehicles passing through each target lane during the signal light phase in the electric alarm monitoring video corresponding to the intersection information to be analyzed; analyze the first number of vehicles, the second number of vehicles and the third number of vehicles based on a multimodal large model to determine an analysis result, the analysis result is used to describe whether there is a need for tuning processing at the intersection corresponding to the intersection information to be analyzed within the time range corresponding to the signal light phase;

[0120] The display 901 is configured to display the analysis result of the intersection of the intersection information to be analyzed.

[0121] Since the processing process of the traffic signal optimization terminal 90 is consistent with the process of the traffic signal optimization method based on the multimodal large model described in the above embodiments, the embodiments of the present application will not be described in detail.

[0122] Based on the same inventive concept, the embodiment of the present application also provides a traffic signal optimization device based on a multi-modal large model. Fig.10 A schematic diagram of the structure of a traffic signal optimization device based on a multimodal large model provided in an embodiment of the present application is shown in FIG. Fig.10 As shown, the device comprises:

[0123] The acquisition module 1001 is used to acquire the intersection information to be analyzed carried in the traffic signal analysis instruction if a traffic signal analysis instruction is received;

[0124] The statistical module 1002 is used to determine, for each traffic light phase corresponding to the intersection information to be analyzed, each target lane that is allowed to pass through the traffic light phase; count the number of first vehicles located in each target lane at the beginning of the traffic light phase in the high point monitoring video corresponding to the intersection information to be analyzed, the number of second vehicles located in each target lane at the end of the traffic light phase, and the number of third vehicles passing through each target lane during the traffic light phase in the electric police monitoring video corresponding to the intersection information to be analyzed;

[0125] The analysis module 1003 is used to analyze the first number of vehicles, the second number of vehicles and the third number of vehicles based on a multimodal large model to determine an analysis result, and the analysis result is used to describe whether the intersection corresponding to the intersection information to be analyzed needs to be optimized within the time range corresponding to the phase of the traffic light.

[0126] In one possible implementation, the statistical module 1002 is specifically used to obtain the stop line position information pre-saved for the intersection information to be analyzed, and determine the stop line in the intersection image captured by the electric police monitoring video based on the stop line position information; for each target lane, if any vehicle crosses the stop line of the exit lane corresponding to the target lane during the traffic light phase, and crosses the stop line of the entry lane corresponding to the target lane, it is determined that the vehicle has passed through the target lane, and the third vehicle number is updated.

[0127] In a possible implementation, the statistical module 1002 is specifically used to determine a corresponding vehicle detection frame in the image frame where the vehicle is located; if any end point of the stop line is located in the vehicle detection frame, it is determined that the vehicle crosses the stop line.

[0128] In a possible implementation, the statistical module 1002 is specifically used to determine a vehicle detection frame corresponding to the image frame where the vehicle is located; if a diagonal line of the vehicle detection frame intersects with the stop line, it is determined that the vehicle crosses the stop line.

[0129] In a possible implementation, the target recognition large model is obtained by fine-tuning training based on the instruction data set;

[0130] The acquisition module 1001 is also used to acquire sample videos;

[0131] Construction module 1004 is used to analyze the sample video based on the pre-trained large model to determine the target phase of the sample video; splice the target phase, the vehicle information during the target phase corresponding to the sample video, and the preset prompt words to obtain sample prompt words; input the sample prompt words and the sample video into the pre-trained large model to obtain sample analysis results; save the sample video, the sample prompt words and the sample analysis results in the instruction data set.

[0132] Based on the same inventive concept, an embodiment of the present application further provides an electronic device, Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig.11 As shown, it includes: a processor 1101, a communication interface 1102, a memory 1103 and a communication bus 1104, wherein the processor 1101, the communication interface 1102, and the memory 1103 communicate with each other through the communication bus 1104;

[0133] The memory 1103 stores a computer program. When the program is executed by the processor 1101, the processor 1101 executes the steps of the traffic signal optimization method based on the multimodal large model described in the above embodiments.

[0134] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 1102 is used for communication between the above electronic device and other devices. The memory may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0135] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0136] On the basis of the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program executable by a processor. When the program runs on the processor, the processor implements the steps of the traffic signal optimization method based on a multimodal large model described in the above embodiments.

[0137] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0138] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0139] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0140] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0141] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A traffic signal optimization method based on a multimodal large model, characterized in that: The method comprises: If a traffic signal analysis instruction is received, obtaining the intersection information to be analyzed carried in the traffic signal analysis instruction; For each signal light phase corresponding to the intersection information to be analyzed, determine each target lane that is allowed to pass through the signal light phase; count the number of first vehicles located in each target lane at the beginning of the signal light phase in the high point monitoring video corresponding to the intersection information to be analyzed, the number of second vehicles located in each target lane at the end of the signal light phase, and the number of third vehicles passing through each target lane during the signal light phase in the electric police monitoring video corresponding to the intersection information to be analyzed; The first number of vehicles, the second number of vehicles and the third number of vehicles are analyzed based on a multimodal large model to determine an analysis result, wherein the analysis result is used to describe whether the intersection corresponding to the intersection information to be analyzed needs to be optimized within a time range corresponding to the phase of the traffic light.

2. The method according to claim 1, characterized in that Counting the number of third vehicles passing through each target lane during the signal light phase in the electric police monitoring video corresponding to the intersection information to be analyzed, including: Obtaining the stop line position information pre-saved for the intersection information to be analyzed, and determining the stop line in the intersection picture captured by the electric police monitoring video according to the stop line position information; For each target lane, if any vehicle crosses the stop line of the exit lane corresponding to the target lane and crosses the stop line of the entrance lane corresponding to the target lane during the signal light phase, it is determined that the vehicle passes through the target lane and the third vehicle number is updated.

3. The method according to claim 2, characterized in that The process of determining whether any vehicle crosses the stop line of the corresponding lane includes: Determine a corresponding vehicle detection frame in the image frame where the vehicle is located; If any end point of the stop line is located within the vehicle detection frame, it is determined that the vehicle crosses the stop line.

4. The method according to claim 2, characterized in that: The process of determining whether any vehicle crosses the stop line of the corresponding lane includes: Determine a corresponding vehicle detection frame in the image frame where the vehicle is located; If the diagonal line of the vehicle detection frame intersects the stop line, it is determined that the vehicle crosses the stop line.

5. The method according to claim 1, characterized in that The multimodal large model is obtained by fine-tuning training based on the instruction data set; The construction process of the instruction data set includes: Get sample video; Analyze the sample video based on the pre-trained large model to determine the target phase of the sample video; splicing the target phase, the vehicle information during the target phase corresponding to the sample video, and the preset prompt word to obtain a sample prompt word; Input the sample prompt words and the sample video into the pre-trained large model to obtain a sample analysis result; The sample video, the sample prompt words and the sample analysis results are stored in the instruction data set.

6. A traffic signal tuning terminal, characterized in that: The terminal includes a display and a processor; The processor is configured to: If a traffic signal analysis instruction is received, obtaining the intersection information to be analyzed carried in the traffic signal analysis instruction; For each signal light phase corresponding to the intersection information to be analyzed, determine each target lane that is allowed to pass through the signal light phase; count the number of first vehicles located in each target lane at the beginning of the signal light phase in the high point monitoring video corresponding to the intersection information to be analyzed, the number of second vehicles located in each target lane at the end of the signal light phase, and the number of third vehicles passing through each target lane during the signal light phase in the electric police monitoring video corresponding to the intersection information to be analyzed; Analyze the first number of vehicles, the second number of vehicles, and the third number of vehicles based on a multimodal large model to determine an analysis result, wherein the analysis result is used to describe whether an intersection corresponding to the intersection information to be analyzed needs to be optimized within a time range corresponding to the phase of the traffic light; The display is configured to display the analysis result of the intersection of the intersection information to be analyzed.

7. A traffic signal optimization device based on a multimodal large model, characterized in that: The device comprises: An acquisition module, configured to acquire the intersection information to be analyzed carried in the traffic signal analysis instruction if a traffic signal analysis instruction is received; A statistical module is used to determine, for each signal light phase corresponding to the intersection information to be analyzed, each target lane that is allowed to pass through the signal light phase; count the number of first vehicles located in each target lane at the beginning of the signal light phase in the high point monitoring video corresponding to the intersection information to be analyzed, the number of second vehicles located in each target lane at the end of the signal light phase, and the number of third vehicles passing through each target lane during the signal light phase in the electric alarm monitoring video corresponding to the intersection information to be analyzed; An analysis module is used to analyze the first number of vehicles, the second number of vehicles and the third number of vehicles based on a multimodal large model to determine an analysis result, wherein the analysis result is used to describe whether the intersection corresponding to the intersection information to be analyzed needs to be optimized within a time range corresponding to the phase of the traffic light.

8. The device according to claim 7, characterized in that The statistical module is specifically used to obtain the stop line position information pre-saved for the intersection information to be analyzed, and determine the stop line in the intersection picture captured by the electric police monitoring video based on the stop line position information; for each target lane, if any vehicle crosses the stop line of the exit lane corresponding to the target lane during the signal light phase, and crosses the stop line of the entrance lane corresponding to the target lane, it is determined that the vehicle has passed through the target lane, and the third vehicle number is updated.

9. An electronic device, characterized in that: The electronic device comprises a processor, and the processor is used to implement the steps of the traffic signal optimization method based on a multimodal large model as claimed in any one of claims 1 to 5 when executing a computer program stored in a memory.

10. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the steps of the traffic signal optimization method based on a multimodal large model as claimed in any one of claims 1 to 5.

Images