Processing method and system for identifying state of signal lamp based on multistage model linkage

Abstract

The embodiment of the invention relates to a processing method and system for linkage identification of a signal lamp state based on a multistage model. The method comprises the following steps: a roadside device generates signal lamp sub-graph sequences of two sizes for each signal lamp based on a camera video and records the signal lamp sub-graph sequences as a first lamp graph sequence and a second lamp graph sequence; the signal lamp state of each first lamp graph sequence is analyzed, the data consistency of the analysis result is judged, and the signal lamp database is updated when the data consistency is satisfied. When the data consistency is not met, the signal lamp state recognition model is used for recognizing the second lamp graph sequence and judging the reliability of the recognition result, and when the reliability is met, the signal lamp database is updated; and when the reliability is not satisfied, identification is carried out through a multi-mode large model of the remote cloud platform, and a signal lamp database is updated based on an identification result. According to the invention, the resource occupancy rate of the signal lamp state identification task on the roadside equipment can be reduced, and the identification accuracy can be improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a processing method and system for recognizing traffic light status based on multi-level model linkage. Background Technology

[0002] In intelligent traffic management scenarios, edge computing devices (roadside devices) deployed on the side of road sections (referred to as road sections) or intersections (referred to as intersections) can be used to calculate the perception data of traffic lights. In simple terms, this calculation process is to identify the status of traffic lights (such as red light, green light, yellow light, etc.) by video data captured by roadside cameras whose perception range (i.e., shooting range) covers one or more traffic lights.

[0003] Conventional roadside equipment mostly relies on offline calculations based on visual recognition models when processing traffic light perception data. That is, a visual classification model is deployed locally on the equipment, and this local model is responsible for the task of recognizing the traffic light status of all video data.

[0004] Through practice, we have found that this single-model offline computing mechanism of conventional roadside equipment has some problems: 1) Computational resource consumption problem: Even a lightweight visual classification model will still consume a certain amount of memory resources due to its model parameters and model operation framework. If multiple video data are processed simultaneously, the overall memory consumption rate will increase exponentially; 2) Computational accuracy problem: If only a lightweight visual classification model is deployed at the front end, it is very likely to output recognition results with low confidence in poor visual environments (such as reduced nighttime illumination, severe rainstorms, or foggy and snowy weather).

[0005] To address the above problems, we propose an improved solution: 1) Design two types of offline recognition mechanisms and one type of online recognition mechanism: based on HSV features. Figure 3The proposed solution includes three mechanisms: a signal light status determination mechanism based on color pixel ratio (referred to as offline mechanism one), an intelligent determination mechanism based on the signal light status recognition model (referred to as offline mechanism two), and an intelligent recognition mechanism based on a multimodal large model (referred to as online mechanism). The first two offline mechanisms are deployed locally on the roadside equipment, while the last online mechanism is deployed on a remote cloud platform. 2) A multi-level linkage scheme is designed based on these three recognition mechanisms: when processing the traffic light perception data calculation task, recognition is first performed based on offline mechanism one, and the data consistency of the recognition results is judged. If the data consistency does not meet the standard, recognition is then performed by offline mechanism two, and the reliability of the recognition results is judged. If the reliability does not meet the standard, recognition is then performed based on the online mechanism. The offline mechanism one provided in the above improved scheme has a much lower computational load and memory usage than the visual classification model, and the multimodal large model provided has much higher recognition accuracy and efficiency than the conventional visual classification model. This improved scheme not only reduces the overall computational load and overall memory usage of the signal light status recognition task on the roadside equipment but also improves recognition accuracy. How to implement the above improved scheme is the technical problem that this invention needs to solve. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a processing method and system for recognizing traffic light status based on a multi-level model linkage. The roadside equipment of this invention first identifies traffic light status based on HSV characteristics. Figure 3 The signal light status determination mechanism based on color pixel ratio analyzes the signal light status of each traffic light, generates a first state sequence based on the analysis results, performs a data consistency check on it, and updates the signal light database based on each first state sequence that passes the data consistency check. If any first state sequence fails the data consistency check, the device-side signal light status recognition model identifies the signal light status of each traffic light that fails the data consistency check, generates a corresponding second state sequence based on the identification results, performs a reliability check on it, and updates the signal light database based on each second state sequence that passes the reliability check. If there are still second state sequences that fail the reliability check, the system interacts with a remote cloud platform, and a multimodal large model deployed on the platform identifies the signal light status of each traffic light that fails the reliability check, generates a corresponding third state sequence based on the identification results, and updates the signal light database based on the third state sequence. This invention not only reduces the overall computational load and overall memory usage of the signal light status recognition task on roadside equipment but also improves recognition accuracy.

[0007] To achieve the above objectives, a first aspect of the present invention provides a processing method for recognizing traffic light states based on a multi-level model linkage, the method comprising:

[0008] A roadside device receives a first video from a first camera; the roadside device is an edge computing device deployed in a road segment or intersection scenario; the roadside device is connected to the first camera; at least one traffic light exists within the visual perception range of the first camera; the traffic light group includes red, green, and yellow lights; the camera intrinsic and extrinsic parameters of the first camera are known, and the camera installation position of the first camera is fixed and known; the installation positions of the traffic lights corresponding to the first camera are fixed and known.

[0009] The roadside equipment extracts frame images from the first video to obtain a first image sequence; and performs sequential segmentation on the first image sequence to obtain a first sequence segment set; and extracts corresponding video segments from each sequence segment in the first video to obtain a first video segment set; and based on the installation positions of the first camera and its corresponding traffic lights, as well as the preset first and second light image sizes, extracts two sizes of traffic light sub-images for each traffic light on each image of the first image sequence to obtain corresponding first and second light images; and generates corresponding first and second light image sequence sets for each sequence segment of the first sequence segment set.

[0010] The roadside equipment takes each first light pattern sequence of each first light pattern sequence set as the current sequence; and according to HSV characteristics... Figure 3 The signal light status determination mechanism based on the proportion of color pixels analyzes the signal light status of each light image in the current sequence and generates a corresponding first state sequence based on the analysis results; and determines the data consistency of the first state sequence to obtain a first determination result; the first determination result includes data consistency and data inconsistency.

[0011] When the total number of the first judgment results with consistent data is not 0, the roadside device updates the traffic light database based on the first state sequence with consistent data; the traffic light database is deployed on the local or remote cloud platform of the roadside device.

[0012] When the total number of the first judgment results with inconsistent data is not 0, the roadside equipment takes the second light diagram sequence corresponding to each of the first light diagram sequences with inconsistent data as the current sequence; and uses the signal light status recognition model on the equipment side to identify the signal light status of each light diagram in the current sequence and generates the corresponding second status sequence based on the recognition result; and judges the data reliability of the second status sequence to obtain a second judgment result; the second judgment result includes reliable and unreliable.

[0013] When the total number of reliable second judgment results is not 0, the roadside equipment updates the traffic light database based on each reliable second state sequence;

[0014] When the total number of unreliable second judgment results is not zero, the roadside equipment takes each unreliable second light image sequence as the current light image sequence; and takes the traffic light and the first sequence segment corresponding to the current light image sequence as the corresponding current traffic light and current sequence segment; and uses all the image timestamps of the current sequence segment to form a corresponding timestamp sequence; and uses the camera installation location, the camera's intrinsic and extrinsic parameters, the traffic light installation location of the current traffic light, the first video segment corresponding to the previous sequence segment, the timestamp sequence, and the second light image size to form a corresponding remote analysis data packet and sends it to the remote cloud platform.

[0015] Each time the remote cloud platform receives a remote analysis data packet, it encapsulates the current data packet into a large model instruction to obtain a corresponding traffic light status recognition instruction; it then inputs the traffic light status recognition instruction into the multimodal large model on the platform side to perform traffic light status recognition to obtain a corresponding third state sequence; and finally sends the third state sequence back to the roadside equipment.

[0016] The roadside equipment updates the traffic light database based on the third state sequence.

[0017] Preferably, the installation positions of the camera and each of the signal lights are three-dimensional coordinates in the same three-dimensional coordinate system;

[0018] The first image sequence includes a plurality of first images; each first image corresponds to an image timestamp;

[0019] The first sequence segment set includes multiple first sequence segments; the first sequence segment includes multiple first images; the time span of the first sequence segment is 2L1, where L1 is a preset first duration;

[0020] The segmentation rule for the first sequence segment is as follows: the video segment of the first video is sequentially divided into multiple first segment segments with a segment length of 2L1; the first images in the first image sequence that are in each of the first segment segments are extracted to form the corresponding first sequence segments; and image deduplication is performed on the two first sequence segments corresponding to each two adjacent first segment segments.

[0021] The first video segment set includes multiple first video segments; each first video segment corresponds one-to-one with a first sequence segment; the time span of each first video segment is less than or equal to 2L2, where L2 is a preset second duration and L2 > L1;

[0022] The extraction rule for the first video segment is as follows: take the center time of the first segment time period corresponding to each first sequence segment as the current time, and extract the video content in each L2 before and after the current time in the first video to form the corresponding first video segment;

[0023] The dimensions of the first light diagram include a first height and a first width; the dimensions of the second light diagram include a second height and a second width; the first height < the second height, and the first width < the second width.

[0024] The first light image sequence set corresponds one-to-one with the first sequence segment; the first light image sequence set consists of one or more first light image sequences, and the first light image sequence corresponds one-to-one with the traffic lights; the first light image sequence includes multiple first light images; the image size of the first light image matches the size of the first light image.

[0025] The second light image sequence set corresponds one-to-one with the first sequence segment; the second light image sequence set consists of one or more second light image sequences, and each second light image sequence corresponds one-to-one with the traffic lights; the second light image sequence includes multiple second light images; the image size of the second light image matches the size of the second light image; the first and second light image sequence sets correspond one-to-one, and the first and second light image sequences in two corresponding sequence sets correspond one-to-one, and the first and second light images in two corresponding light image sequences... Figure 1 One-to-one correspondence;

[0026] The first state sequence corresponds one-to-one with the first light diagram sequence; the first state sequence includes multiple first signal light states; the first signal light states include red, green, yellow, all black, all bright, and unknown; the first signal light states in the first state sequence correspond to the first lights in the currently corresponding first light diagram sequence. Figure 1 One-to-one correspondence;

[0027] Each second state sequence corresponds to a second light diagram sequence; the second state sequence includes multiple second signal light states; the second signal light states include red, green, yellow, all black, all bright, and unknown; the second signal light states in the second state sequence correspond to the second lights in the currently corresponding second light diagram sequence. Figure 1 One-to-one correspondence;

[0028] Each of the third state sequences corresponds to one of the second light diagram sequences; the third state sequence includes multiple third signal light states; the third signal light states include red light, green light, yellow light, all black, all bright, and unknown; the third signal light states of the third state sequence correspond to the second lights in the currently corresponding second light diagram sequence. Figure 1 One-to-one correspondence;

[0029] The traffic light database includes multiple traffic light status sequences; each traffic light status sequence corresponds one-to-one with a traffic light; each traffic light status sequence is formed by sequentially sorting multiple signal status data; each signal status data consists of a set of signal timestamps and signal status; the signal status includes red light, green light, yellow light, all black, all bright, and unknown.

[0030] Preferably, the traffic light status recognition model is implemented based on a visual classification model; the traffic light status recognition model is used to predict the traffic light status of the input traffic light image and output the corresponding predicted status and status confidence; the predicted status includes red light, green light, yellow light, all black, all bright, and unknown; the traffic light status recognition model has been trained before being deployed to the roadside equipment;

[0031] The traffic light status recognition model consists of a feature extraction module and a classification task head connected together; the feature extraction network is implemented based on a CNN network or a residual network; the classification task head consists of a feature mapping network and a classification output layer connected sequentially; the feature mapping network consists of one or more feature transformation layers connected sequentially, and each feature transformation layer consists of a set of corresponding linear layers and ReLU activation function layers connected sequentially.

[0032] The feature extraction module is used to perform image feature extraction processing on the traffic light image to obtain the corresponding image feature vector and send it to the classification task head;

[0033] The classification task head is used to perform state classification prediction based on the image feature vector to obtain the corresponding predicted state and the state confidence score, and output them. Specifically, the feature mapping network maps the image feature vector to the traffic light state feature space to obtain the corresponding state feature vector and sends it to the classification output layer; the classification output layer inputs the state feature vector into the softmax function to calculate the state probability to obtain the corresponding state prediction vector, and takes the state type and probability value corresponding to the prediction probability with the highest probability value in the state prediction vector as the corresponding predicted state and the state confidence score, and outputs them; the state prediction vector consists of the prediction probability of red light state, green light state, yellow light state, all-black state, all-on state, and unknown state, and the sum of the probability values ​​of the six prediction probabilities of the state prediction vector is 1;

[0034] The traffic light status recognition command includes model command text, command video, and video timestamp sequence; each frame of the command video has a traffic light label box; the rectangular size of the traffic light label box is consistent with the size of the second traffic light image;

[0035] The instruction text encapsulation template corresponding to the traffic light status recognition instruction is denoted as the first instruction template;

[0036] The first instruction template is a preset natural language description text; the first instruction template is used to prompt the multimodal big model to identify the state of the traffic lights within the bounding boxes of the traffic lights in each frame of the instruction video image, and to extract the state of the traffic lights within the bounding boxes corresponding to each timestamp of the video timestamp sequence based on the identification results, and to sort all the extracted states in chronological order to form the corresponding third state sequence and output it, and to define the state range of the traffic lights within the bounding boxes as including red light, green light, yellow light, all black, all bright, and unknown;

[0037] The multimodal large model is implemented based on the Qwen2.5-VL-7B model; the multimodal large model is used to identify the traffic light status according to the traffic light status identification instruction and output the corresponding third state sequence; the multimodal large model has completed model training before being deployed to the remote cloud platform.

[0038] Preferably, the step of extracting the corresponding first and second light images from the two sizes of traffic light sub-images of each traffic light on each image of the first image sequence, based on the installation positions of the first camera and its corresponding traffic lights, and the preset sizes of the first and second light images, specifically includes:

[0039] Based on the installation positions of the camera and each of the traffic lights, the relative pose of the first camera and its corresponding traffic lights is confirmed.

[0040] Each of the first images in the first image sequence is taken as the current image; the pixel points corresponding to the installation positions of each traffic light in the current image are estimated based on the camera intrinsic and extrinsic parameters and the relative poses, and the estimation results are taken as the corresponding traffic light center points; a rectangular sub-image with the center points of each traffic light as the rectangle center and the size of the first light image as the rectangle size is extracted from the current image and taken as the corresponding first light image; and a rectangular sub-image with the center points of each traffic light as the rectangle center and the size of the second light image as the rectangle size is extracted from the current image and taken as the corresponding second light image.

[0041] Preferably, generating corresponding first and second light diagram sequence sets for each sequence segment of the first sequence segment set specifically includes:

[0042] Each of the first sequence segments in the first sequence segment set is taken as the current sequence segment; and all the first light images corresponding to the current sequence segment form a corresponding first segment light image set, and all the second light images corresponding to the current sequence segment form a corresponding second segment light image set; and multiple first light images corresponding to the same traffic light in the first segment light image set are extracted and sorted in chronological order to form a corresponding first light image sequence, and multiple second light images corresponding to the same traffic light in the second segment light image set are extracted and sorted in chronological order to form a corresponding second light image sequence; and all the first light image sequences corresponding to the current sequence segment form a corresponding first light image sequence set, and all the second light image sequences corresponding to the current sequence segment form a corresponding second light image sequence set.

[0043] Preferably, the HSV feature Figure 3 The signal light state determination mechanism based on color pixel ratio analyzes the signal light state of each light image in the current sequence and generates the corresponding first state sequence based on the analysis results. Specifically, it includes:

[0044] Each of the first light images in the current sequence is taken as the current light image; wherein, the current light image includes a plurality of first pixels; the pixel features of the first pixel include RGB pixel vectors; the RGB pixel vectors are composed of RGB three primary color pixel values;

[0045] The HSV feature vector is converted for each RGB pixel vector in the current light image to obtain the corresponding HSV feature map; wherein, the HSV feature map includes multiple second pixels; the first and second pixels correspond one-to-one; the pixel feature of the second pixel includes the HSV feature vector; the HSV feature vector is composed of color feature H, lightness / darkness feature S, and brightness / darkness feature V;

[0046] Based on preset red, green, and yellow feature ranges, the HSV feature map is transformed using three types of mask images to obtain corresponding first, second, and third mask images. Specifically, the second pixels in the HSV feature map whose color feature H satisfies the red feature range are designated as first foreground pixels, and the second pixels that do not satisfy the red feature range are designated as first background pixels. All the first foreground pixels and the first background pixels form the corresponding first mask image. The second pixels in the HSV feature map whose color feature H satisfies the green feature range are designated as second foreground pixels. The first, second, and third masks are defined as follows: First, foreground pixels and second background pixels that do not satisfy the green feature range are designated as second background pixels; second, foreground pixels that satisfy the yellow feature range are designated as third foreground pixels, and second, background pixels that do not satisfy the yellow feature range are designated as third background pixels; third, foreground pixels and third background pixels are designated as third mask. The image sizes of the first, second, and third masks are all consistent with the image size of the HSV feature map.

[0047] The first, second, and third mask images are subjected to image denoising processing, specifically: the first, second, or third mask image is used as the current mask image, and the current mask image is first denoised using morphological opening operation processing, and then the denoised current mask image is filled with holes using morphological closing operation processing;

[0048] The first, second, and third percentages are calculated for the foreground pixels corresponding to the first, second, and third masks, respectively. Specifically, the total number of pixels in the HSV feature map is counted to obtain the first total number; the total number of foreground pixels in the first, second, and third masks after noise reduction is counted to obtain the first, second, and third total foreground pixels; and the first, second, and third percentages are calculated based on the first total number and the first, second, and third total foreground pixels: first percentage = (total first foreground pixels / total first total number) × 100%, second percentage = (total second foreground pixels / total first total number) × 100%, third percentage = (total third foreground pixels / total first total number) × 100%.

[0049] Based on the preset traffic light status determination rules, the traffic light status is determined according to the proportions of the first, second, and third traffic lights corresponding to the current traffic light diagram to obtain the corresponding first traffic light status.

[0050] The rule for determining the status of the traffic lights is as follows:

[0051]

[0052] SA represents the first traffic light state; a1, a2, and a3 represent the first, second, and third percentages, respectively; the six state types s1, s2, s3, s4, s5, and s6 represent all black, red, green, yellow, all on, and unknown, respectively.

[0053] The first state sequence is formed by sorting all the first signal light states corresponding to the current sequence in chronological order.

[0054] Preferably, the step of determining the data consistency of the first state sequence to obtain a first determination result specifically includes:

[0055] The total number of the first traffic light states in the first state sequence is counted to obtain the corresponding second total number; the total number of traffic light states of the six state types in the first state sequence is counted to obtain the corresponding total number of the six state types; the maximum value among the total number of the six state types is taken as the third total number; and the corresponding first state percentage is calculated based on the second and third total numbers = (third total number / second total number) × 100%; and whether the first state percentage is greater than or equal to a preset state percentage threshold is identified; if yes, the corresponding first judgment result is set as data consistency; if no, the corresponding first judgment result is set as data inconsistency.

[0056] Preferably, updating the traffic light database based on the first state sequence with consistent data specifically includes:

[0057] The first state sequence corresponding to each of the first judgment results set to be consistent with the data is taken as the current source sequence; the traffic light corresponding to the current source sequence is taken as the current traffic light; the traffic light state sequence corresponding to the current traffic light in the traffic light database is taken as the current target sequence; each of the first traffic light states in the current source sequence and its corresponding image timestamp is taken as a group of corresponding signal states and signal timestamps to form a corresponding signal state data; and all the signal state data corresponding to the current source sequence are taken as a corresponding current addition sequence; and the current addition sequence is added to the current target sequence.

[0058] Preferably, the step of using the device-side traffic light status recognition model to identify the traffic light status of each light in the current sequence and generating a corresponding second status sequence based on the recognition results specifically includes:

[0059] Each of the second light images in the current sequence is used as the current traffic light image and input into the traffic light state recognition model for processing to obtain the corresponding predicted state and the state confidence; the predicted state is used as the corresponding second traffic light state; and all the second traffic light states corresponding to the current sequence are sorted in chronological order to form the corresponding second state sequence.

[0060] Preferably, the step of determining the data reliability of the second state sequence to obtain a second determination result specifically includes:

[0061] The total number of the second traffic light states in the second state sequence is counted to obtain the corresponding total number N1; the total number of traffic light states in the six state types of the second state sequence is counted to obtain the corresponding total number of the six state types; the maximum value among the total number of the six state types is taken as the corresponding maximum class total number N2; the corresponding maximum class confidence set is formed by the N2 state confidence scores corresponding to the maximum class total number N2; and the corresponding second state percentage is calculated as (N2 / N1)×100% based on the total number N1 and the maximum class total number N2.

[0062] The system identifies whether the proportion of the second state is greater than or equal to a preset state proportion threshold. If the second state is less than the state proportion threshold, the corresponding second judgment result is set to unreliable. If the second state is greater than or equal to the state proportion threshold, the system identifies whether the confidence scores of all states in the maximum class confidence set are greater than a preset confidence threshold. If so, the corresponding second judgment result is set to reliable; otherwise, the corresponding second judgment result is set to unreliable.

[0063] Preferably, updating the traffic light database based on each reliable second state sequence specifically includes:

[0064] The second state sequence corresponding to each of the second judgment results set as reliable is taken as the current source sequence; the traffic light corresponding to the current source sequence is taken as the current traffic light; the traffic light state sequence corresponding to the current traffic light in the traffic light database is taken as the current target sequence; each second traffic light state of the current source sequence and its corresponding image timestamp are taken as a group of corresponding signal states and signal timestamps to form a corresponding signal state data; and all the signal state data corresponding to the current source sequence are taken as a corresponding current addition sequence; and the current addition sequence is added to the current target sequence.

[0065] Preferably, the step of encapsulating the current data packet into a large model instruction to obtain the corresponding traffic light status recognition instruction specifically includes:

[0066] Extract the corresponding camera installation location, traffic light installation location, camera intrinsic and extrinsic parameters, first video segment, timestamp sequence, and second light pattern size from the current remote analysis data packet;

[0067] The first video segment is then subjected to frame image extraction to obtain a corresponding second image sequence; wherein the second image sequence includes multiple second images;

[0068] Based on the installation positions of the camera and the traffic light, the relative pose of the first camera and the current traffic light is confirmed to obtain the corresponding current relative pose.

[0069] Each second image in the second image sequence is taken as the current image; the pixel points corresponding to the traffic light installation position on the current image are estimated based on the camera intrinsic and extrinsic parameters and the current relative pose, and the estimation result is taken as the corresponding current traffic light center point; the rectangular area in the current image with the current traffic light center point as the rectangle center and the size of the second light image as the rectangle size is recorded as the current traffic light area; the corresponding traffic light annotation box is loaded on the current image for the current traffic light area; the current image with the completed annotation box is taken as the corresponding third image; and all the obtained third images form a third image sequence;

[0070] The corresponding instruction video is obtained by video reconstruction based on the third image sequence; the timestamp sequence is used as the corresponding video timestamp sequence; the corresponding model instruction text is generated based on the first instruction template; and the corresponding traffic light status recognition instruction is composed of the obtained model instruction text, the instruction video and the video timestamp sequence.

[0071] Preferably, updating the traffic light database based on the third state sequence specifically includes:

[0072] Each received third state sequence is taken as the current source sequence; the traffic light corresponding to the current source sequence is taken as the current traffic light; the traffic light state sequence corresponding to the current traffic light in the traffic light database is taken as the current target sequence; each third traffic light state in the current source sequence and its corresponding image timestamp are taken as a set of corresponding signal states and signal timestamps to form a corresponding signal state data; all the signal state data corresponding to the current source sequence are taken as a corresponding current addition sequence; and the current addition sequence is added to the current target sequence.

[0073] A second aspect of this invention provides a system for implementing the processing method for recognizing traffic light states based on a multi-level model linkage as described in the first aspect. The system includes: a roadside device and a remote cloud platform; the roadside device includes a data receiving module, a data preprocessing module, a first state recognition module, a first state update module, a second state recognition module, a second state update module, a first remote interaction module, and a third state update module; the remote cloud platform includes a third state recognition module.

[0074] The data receiving module of the roadside device is used to receive the first video from the first camera; the roadside device is an edge computing device deployed in a road segment or intersection scene; the roadside device is connected to the first camera; there is at least one traffic light within the visual perception range of the first camera; the traffic light group includes red, green, and yellow lights; the camera intrinsic and extrinsic parameters of the first camera are known, and the camera installation position of the first camera is fixed and known; the installation positions of the traffic lights corresponding to the first camera are fixed and known.

[0075] The data preprocessing module of the roadside equipment is used to extract frame images from the first video to obtain a first image sequence; and to perform sequential segmentation of the first image sequence to obtain a first sequence segment set; and to extract corresponding video segments from each sequence segment of the first video to obtain a first video segment set; and based on the installation position of the first camera and its corresponding traffic lights and the preset first and second light image sizes, to extract the two sizes of traffic light sub-images of each traffic light on each image of the first image sequence to obtain the corresponding first and second light images; and to generate corresponding first and second light image sequence sets for each sequence segment of the first sequence segment set;

[0076] The first state recognition module of the roadside equipment is used to take each first light pattern sequence of each first light pattern sequence set as the current sequence; and according to HSV features Figure 3 The signal light status determination mechanism based on the proportion of color pixels analyzes the signal light status of each light image in the current sequence and generates a corresponding first state sequence based on the analysis results; and determines the data consistency of the first state sequence to obtain a first determination result; the first determination result includes data consistency and data inconsistency.

[0077] The first state update module of the roadside device is used to update the traffic light database based on the first state sequence with consistent data when the total number of the first judgment results with consistent data is not 0; the traffic light database is deployed locally on the roadside device or on the remote cloud platform.

[0078] The second state recognition module of the roadside equipment is used to take the second light diagram sequence corresponding to each inconsistent first light diagram sequence as the current sequence when the total number of inconsistent first judgment results is not 0; and to use the signal light state recognition model on the equipment side to recognize the signal light state of each light diagram in the current sequence and generate the corresponding second state sequence based on the recognition result; and to judge the data reliability of the second state sequence to obtain a second judgment result; the second judgment result includes reliable and unreliable.

[0079] The second state update module of the roadside equipment is used to update the traffic light database based on each reliable second state sequence when the total number of reliable second judgment results is not 0.

[0080] The first remote interaction module of the roadside equipment is used to, when the total number of unreliable second judgment results is not 0, take each unreliable second light image sequence as the current light image sequence; and take the traffic light and the first sequence segment corresponding to the current light image sequence as the corresponding current traffic light and current sequence segment; and use the timestamps of all images of the current sequence segment to form a corresponding timestamp sequence; and use the camera installation location, the camera intrinsic and extrinsic parameters, the installation location of the current traffic light, the first video segment corresponding to the previous sequence segment, the timestamp sequence, and the size of the second light image to form a corresponding remote analysis data packet and send it to the remote cloud platform.

[0081] The third state recognition module of the remote cloud platform is used to encapsulate the corresponding traffic light state recognition instruction based on the large model instruction of the current data packet when it receives a remote analysis data packet; and input the traffic light state recognition instruction into the multimodal large model on the platform side to perform traffic light state recognition to obtain the corresponding third state sequence; and send the third state sequence back to the roadside equipment.

[0082] The third state update module of the roadside equipment updates the traffic light database based on the third state sequence.

[0083] This invention provides a method and system for identifying traffic light status based on a multi-level model linkage. As can be seen from the above description, the roadside equipment in this invention first identifies traffic light status based on HSV characteristics. Figure 3The signal light status determination mechanism based on color pixel ratio analyzes the signal light status of each traffic light, generates a first state sequence based on the analysis results, performs a data consistency check on it, and updates the signal light database based on each first state sequence that passes the data consistency check. If any first state sequence fails the data consistency check, the device-side signal light status recognition model identifies the signal light status of each traffic light that fails the data consistency check, generates a corresponding second state sequence based on the identification results, performs a reliability check on it, and updates the signal light database based on each second state sequence that passes the reliability check. If there are still second state sequences that fail the reliability check, the system interacts with a remote cloud platform, and a multimodal large model deployed on the platform identifies the signal light status of each traffic light that fails the reliability check, generates a corresponding third state sequence based on the identification results, and updates the signal light database based on the third state sequence. This invention not only reduces the overall computational load and overall memory usage of the signal light status recognition task on roadside equipment, but also improves the recognition accuracy. Attached Figure Description

[0084] Figure 1 This is a schematic diagram of a processing method for recognizing traffic light status based on a multi-level model linkage, provided in Embodiment 1 of the present invention.

[0085] Figure 2 This is a schematic diagram of the first sequence segment and the first video segment provided in Embodiment 1 of the present invention;

[0086] Figure 3 This is a schematic diagram of the first light pattern sequence set and the second light pattern sequence set provided in Embodiment 1 of the present invention;

[0087] Figure 4 This is a schematic diagram of the traffic light status recognition model provided in Embodiment 1 of the present invention;

[0088] Figure 5 This is a schematic diagram of a processing system for recognizing traffic light status based on a multi-level model linkage, as provided in Embodiment 2 of the present invention. Detailed Implementation

[0089] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0090] Embodiment 1 of the present invention provides a processing method for recognizing traffic light status based on a multi-level model linkage, such as... Figure 1The schematic diagram of a processing method for recognizing traffic light status based on a multi-level model linkage provided in Embodiment 1 of the present invention is shown, which includes the following main steps:

[0091] Step 1: The roadside equipment receives the first video from the first camera.

[0092] Here, the roadside device in this embodiment of the invention is an edge computing device deployed in a road segment or intersection scenario; the roadside device is connected to a first camera. At least one traffic light exists within the visual perception range of the first camera in this embodiment of the invention; the traffic light group in this embodiment of the invention includes red, green, and yellow lights. It should be noted that the camera's intrinsic and extrinsic parameters are known, and the camera's installation position is fixed and known; the installation positions of the traffic lights corresponding to the first camera are fixed and known; the camera's installation position and the installation positions of each traffic light are all three-dimensional coordinates in the same three-dimensional coordinate system, such as the world coordinate system or the NED coordinate system.

[0093] Step 2: The roadside equipment extracts frame images from the first video to obtain a first image sequence; and performs sequential segmentation on the first image sequence to obtain a first sequence segment set; and extracts corresponding video segments from each sequence segment in the first video to obtain a first video segment set; and based on the installation positions of the first camera and its corresponding traffic lights and the preset first and second light image sizes, extracts the two sizes of traffic light sub-images for each traffic light on each image in the first image sequence to obtain the corresponding first and second light images; and generates corresponding first and second light image sequence sets for each sequence segment of the first sequence segment set.

[0094] Specifically, this includes: Step 21, extracting frame images from the first video to obtain a first image sequence;

[0095] Here, the first image sequence in this embodiment of the invention includes a plurality of first images; each first image corresponds to an image timestamp;

[0096] Step 22, and perform sequential segmentation on the first image sequence to obtain the first sequence segment set;

[0097] Here, the first sequence segment set of this embodiment of the invention includes a plurality of first sequence segments; the first sequence segment includes a plurality of first images; the time span of the first sequence segment is 2L1, where L1 is a preset first duration;

[0098] The segmentation rule for the first sequence segment in this embodiment of the invention is as follows: with a segment length of 2L1, the video segment of the first video is sequentially divided into multiple first segment segments; the first images in the first image sequence that are in each first segment segment are extracted to form the corresponding first sequence segments; and image deduplication is performed on the two first sequence segments corresponding to each two adjacent first segment segments. Figure 2 This is a schematic diagram of the first sequence segment and the first video segment provided in Embodiment 1 of the present invention, based on Figure 2 This allows for an intuitive understanding of the sequential segmentation mechanism of the first sequence segment;

[0099] Step 23: Extract the corresponding video segments from each sequence segment in the first video to obtain the first video segment set;

[0100] Here, the first video segment set in this embodiment of the invention includes multiple first video segments; each first video segment corresponds one-to-one with a first sequence segment; the time span of each first video segment is less than or equal to 2L2, where L2 is a preset second duration and L2 > L1;

[0101] The extraction rule for the first video segment in this embodiment of the invention is as follows: taking the center time of the first segment time period corresponding to each first sequence segment as the current time, and extracting the video content within each L2 before and after the current time in the first video to form the corresponding first video segment; based on Figure 2 It allows for an intuitive understanding of the one-to-one correspondence between the first video segment and the first sequence segment, as well as the extraction logic of the first video segment;

[0102] Step 24: Based on the installation position of the first camera and its corresponding traffic lights, as well as the preset first and second light image sizes, extract the two sizes of traffic light sub-images of each traffic light from each image in the first image sequence to obtain the corresponding first and second light images.

[0103] The dimensions of the first light diagram include a first height and a first width; the dimensions of the second light diagram include a second height and a second width; the first height < the second height, and the first width < the second width.

[0104] Specifically, this includes: Step 241, confirming the relative pose of the first camera and its corresponding traffic lights based on the camera installation location and the installation location of each traffic light;

[0105] Step 242: Take each first image in the first image sequence as the current image; estimate the pixel points corresponding to the installation positions of each traffic light in the current image based on the camera intrinsic and extrinsic parameters and each relative pose, and take the estimation results as the corresponding traffic light center points; extract the rectangular sub-images in the current image with the center points of each traffic light as the rectangle center and the size of the first light image as the rectangle size, and take them as the corresponding first light image; extract the rectangular sub-images in the current image with the center points of each traffic light as the rectangle center and the size of the second light image as the rectangle size, and take them as the corresponding second light image;

[0106] Step 25, and generate corresponding first and second light diagram sequence sets for each sequence segment of the first sequence segment set;

[0107] Specifically, this includes: taking each first sequence segment of the first sequence segment set as the current sequence segment; forming a corresponding first segment light image set by all first light images corresponding to the current sequence segment, and forming a corresponding second segment light image set by all second light images corresponding to the current sequence segment; extracting multiple first light images corresponding to the same traffic light from the first segment light image set and sorting them in chronological order to form a corresponding first light image sequence; extracting multiple second light images corresponding to the same traffic light from the second segment light image set and sorting them in chronological order to form a corresponding second light image sequence; forming a corresponding first light image sequence set by all first light image sequences corresponding to the current sequence segment, and forming a corresponding second light image sequence set by all second light image sequences corresponding to the current sequence segment.

[0108] Here, in this embodiment of the invention, the first light image sequence set corresponds one-to-one with the first sequence segment; the first light image sequence set consists of one or more first light image sequences, and the first light image sequence corresponds one-to-one with the traffic lights; the first light image sequence includes multiple first light images; the image size of the first light image matches the size of the first light image. Figure 3 This is a schematic diagram of the first light pattern sequence set and the second light pattern sequence set provided in Embodiment 1 of the present invention, based on Figure 3 The first light diagram sequence set can be intuitively understood.

[0109] The second light image sequence set of this invention corresponds one-to-one with the first sequence segment; the second light image sequence set consists of one or more second light image sequences, and each second light image sequence corresponds one-to-one with a traffic light; the second light image sequence includes multiple second light images; the image size of the second light image matches the size of the second light image; the first and second light image sequence sets correspond one-to-one, and the first and second light image sequences in two corresponding sequence sets correspond one-to-one, and the first and second light images in two corresponding light image sequences... Figure 1 One-to-one correspondence. Based on Figure 3 The second light diagram sequence set can be intuitively understood.

[0110] Step 3: The roadside equipment takes each first light image sequence from each first light image sequence set as the current sequence; and according to HSV characteristics... Figure 3 The signal light status determination mechanism based on the proportion of color pixels analyzes the signal light status of each light image in the current sequence and generates a corresponding first state sequence based on the analysis results; and determines the data consistency of the first state sequence to obtain a first determination result;

[0111] Specifically, this includes: Step 31, taking each first light image sequence of each first light image sequence set as the current sequence;

[0112] Step 32, and according to HSV characteristics Figure 3 The signal light status determination mechanism based on the proportion of color pixels analyzes the signal light status of each light image in the current sequence and generates the corresponding first state sequence based on the analysis results;

[0113] The first state sequence corresponds one-to-one with the first light diagram sequence; the first state sequence includes multiple first light states; the first light states include red, green, yellow, all black, all bright, and unknown; the first light state in the first state sequence corresponds to the first light in the currently corresponding first light diagram sequence. Figure 1 One-to-one correspondence;

[0114] Specifically, this includes: step 321, taking each of the first light diagrams in the current sequence as the current light diagram;

[0115] The current light image includes multiple first pixels; the pixel features of each first pixel include an RGB pixel vector; the RGB pixel vector is composed of the pixel values ​​of the three primary colors RGB.

[0116] Step 322: Perform HSV feature vector conversion on each RGB pixel vector of the current light image to obtain the corresponding HSV feature map;

[0117] The HSV feature map includes multiple second pixels; the first and second pixels correspond one-to-one; the pixel features of the second pixel include the HSV feature vector; the HSV feature vector is composed of color feature H, lightness / darkness feature S, and brightness / darkness feature V.

[0118] Here, calculating the H, S, and V feature values ​​corresponding to the HSV feature based on the RGB three primary color pixel values ​​is a publicly available conversion calculation scheme, which will not be elaborated further here;

[0119] Step 323: Based on the preset red feature range, green feature range and yellow feature range, the HSV feature map is transformed into three types of mask maps to obtain the corresponding first, second and third mask maps;

[0120] Specifically, this includes: designating second pixels in the HSV feature map whose color feature H satisfies the red feature range as first foreground pixels and second pixels whose color feature H does not satisfy the red feature range as first background pixels, and forming a corresponding first mask image by combining all first foreground pixels and first background pixels; designating second pixels in the HSV feature map whose color feature H satisfies the green feature range as second foreground pixels and second pixels whose color feature H does not satisfy the green feature range as second background pixels, and forming a corresponding second mask image by combining all second foreground pixels and second background pixels; and designating second pixels in the HSV feature map whose color feature H satisfies the yellow feature range as third foreground pixels and second pixels whose color feature H does not satisfy the yellow feature range as third background pixels, and forming a corresponding third mask image by combining all third foreground pixels and third background pixels.

[0121] Here, the red feature range, green feature range, and yellow feature range in this embodiment of the invention are three pre-set color feature value ranges; the image sizes of the first, second, and third mask images are all consistent with the image size of the HSV feature image.

[0122] Step 324: Perform image noise reduction processing on the first, second, and third mask images;

[0123] Specifically, this includes: using the first, second, or third mask image as the current mask image, first denoising the current mask image using morphological opening operation processing, and then filling the holes in the denoised current mask image using morphological closing operation processing;

[0124] Here, the morphological opening operation for noise reduction and the closing operation for hole removal are also publicly available morphological computation schemes, which will not be elaborated further here;

[0125] Step 325: Calculate the proportion of foreground points corresponding to the first, second, and third masks to obtain the corresponding proportions of the first, second, and third masks.

[0126] Specifically, this includes: counting the total number of pixels in the HSV feature map to obtain the corresponding first total; counting the total number of the first, second, and third foreground pixels in the first, second, and third mask maps after noise reduction to obtain the corresponding first, second, and third foreground pixel totals; and calculating the corresponding first, second, and third percentages based on the first total and the first, second, and third foreground pixel totals, where the first percentage = (total first foreground pixel total / first total) × 100%, the second percentage = (total second foreground pixel total / first total) × 100%, and the third percentage = (total third foreground pixel total / first total) × 100%.

[0127] Step 326: Based on the preset traffic light status determination rules, determine the corresponding first traffic light status according to the proportion of the first, second and third traffic lights in the current traffic light diagram.

[0128] Here, the traffic light status determination rule in this embodiment of the invention is as follows:

[0129]

[0130] SA represents the first signal light state; a1, a2, and a3 represent the first, second, and third percentages, respectively; the six state types s1, s2, s3, s4, s5, and s6 represent all black, red, green, yellow, all on, and unknown, respectively.

[0131] Step 327: Sort all the first signal light states corresponding to the current sequence in chronological order to form the corresponding first state sequence;

[0132] Step 33, and determine the data consistency of the first state sequence to obtain the first judgment result;

[0133] The first judgment result includes data consistency and data inconsistency;

[0134] Specifically, this includes: calculating the total number of the first traffic light states in the first state sequence to obtain the corresponding second total; calculating the total number of the traffic light states of the six state types in the first state sequence to obtain the corresponding total number of the six state types; taking the maximum value among the total number of the six state types as the third total; calculating the corresponding first state percentage based on the second and third totals = (third total / second total) × 100%; and identifying whether the first state percentage is greater than or equal to a preset state percentage threshold; if so, setting the corresponding first judgment result as data consistency; if not, setting the corresponding first judgment result as data inconsistency.

[0135] Here, the state percentage threshold in this embodiment of the invention is a pre-set threshold parameter.

[0136] Step 4: When the total number of first judgment results with consistent data is not 0, the roadside equipment updates the traffic light database based on the first state sequence with consistent data.

[0137] The traffic light database is deployed on the roadside equipment locally or on a remote cloud platform. The traffic light database includes multiple traffic light status sequences. Each traffic light status sequence corresponds one-to-one with a traffic light. Each traffic light status sequence is composed of multiple signal status data ordered sequentially. Each signal status data consists of a set of signal timestamps and signal status. The signal status includes red, green, yellow, all black, all bright, and unknown.

[0138] Specifically, this includes: taking the first state sequence corresponding to each first judgment result that is set to be consistent with the data as the current source sequence; taking the traffic light corresponding to the current source sequence as the current traffic light; taking the traffic light state sequence corresponding to the current traffic light in the traffic light database as the current target sequence; taking each first traffic light state of the current source sequence and its corresponding image timestamp as a set of corresponding signal states and signal timestamps to form a corresponding signal state data; taking all the signal state data corresponding to the current source sequence to form a corresponding current addition sequence; and adding the current addition sequence to the current target sequence.

[0139] Step 5: When the total number of first judgment results of data inconsistency is not 0, the roadside equipment takes the second light diagram sequence corresponding to each first light diagram sequence with inconsistent data as the current sequence; and uses the signal light status recognition model on the equipment side to identify the signal light status of each light diagram in the current sequence and generates the corresponding second status sequence based on the recognition result; and judges the data reliability of the second status sequence to obtain the second judgment result.

[0140] Specifically, this includes: Step 51, taking the second light diagram sequence corresponding to the first light diagram sequence with inconsistent data as the current sequence;

[0141] Step 52: Use the signal light status recognition model on the device side to identify the signal light status of each light in the current sequence and generate the corresponding second status sequence based on the recognition results;

[0142] Each second state sequence corresponds to a second light diagram sequence; the second state sequence includes multiple second light states; the second light states include red, green, yellow, all black, all bright, and unknown; the second light states in the second state sequence correspond to the second lights in the currently corresponding second light diagram sequence. Figure 1 One-to-one correspondence;

[0143] Specifically, this includes: inputting each of the second light images in the current sequence into the traffic light state recognition model as the current traffic light image for processing to obtain the corresponding predicted state and state confidence; using the predicted state as the corresponding second traffic light state; and arranging all the second traffic light states corresponding to the current sequence in chronological order to form the corresponding second state sequence.

[0144] It should be noted that the traffic light status recognition model in this embodiment of the invention is based on a visual classification model; the traffic light status recognition model is used to predict the traffic light status of the input traffic light image and output the corresponding predicted status and status confidence; wherein, the predicted status includes red light, green light, yellow light, all black, all bright, and unknown; the traffic light status recognition model has been trained before being deployed to roadside equipment;

[0145] like Figure 4 As shown in the schematic diagram of the traffic light status recognition model provided in Embodiment 1 of the present invention, the traffic light status recognition model is composed of a feature extraction module and a classification task head. The feature extraction network is implemented based on a CNN network or a residual network. The classification task head is composed of a feature mapping network and a classification output layer connected in sequence. The feature mapping network is composed of one or more feature transformation layers connected in sequence. Each feature transformation layer is composed of a set of corresponding linear layers and ReLU activation function layers connected in sequence.

[0146] The functionalities of the model components for the traffic light status recognition model are as follows:

[0147] 1) The feature extraction module is used to perform image feature extraction processing on the traffic light image to obtain the corresponding image feature vector and send it to the classification task head;

[0148] 2) The classification task head is used to perform state classification prediction based on image feature vectors, obtain the corresponding predicted state and state confidence, and output them. Specifically:

[0149] The feature mapping network maps the image feature vector to the signal light state feature space to obtain the corresponding state feature vector, which is then sent to the classification output layer. The classification output layer takes the state feature vector and puts it into the softmax function to calculate the state probability to obtain the corresponding state prediction vector. The state type and probability value corresponding to the prediction probability with the highest probability value in the state prediction vector are taken as the corresponding predicted state and state confidence and output.

[0150] The state prediction vector consists of the prediction probabilities of red light, green light, yellow light, all-black, all-light, and unknown states. The sum of the probability values ​​of the six prediction probabilities in the state prediction vector is 1.

[0151] Step 53, and determine the reliability of the data in the second state sequence to obtain the second judgment result;

[0152] The second judgment result includes reliable and unreliable;

[0153] Specifically, this includes: Step 531, counting the total number of the second signal light states in the second state sequence to obtain the corresponding total number N1; and counting the total number of the six state types of the signal light states in the second state sequence to obtain the corresponding total number of the six states; taking the maximum value among the total number of the six states as the corresponding maximum class total number N2; and forming the corresponding maximum class confidence set by the N2 state confidence scores corresponding to the maximum class total number N2; and calculating the corresponding second state percentage = (N2 / N1) × 100% based on the total number N1 and the maximum class total number N2.

[0154] Step 532, and identify whether the proportion of the second state is greater than or equal to the state proportion threshold; if the second state is less than the state proportion threshold, then set the corresponding second judgment result as unreliable; if the second state is greater than or equal to the state proportion threshold, then identify whether the confidence of all states in the maximum class confidence set is greater than the preset confidence threshold. If so, set the corresponding second judgment result as reliable; otherwise, set the corresponding second judgment result as unreliable.

[0155] Here, the confidence threshold in this embodiment of the invention is a pre-set threshold parameter.

[0156] Step 6: When the total number of reliable second judgment results is not 0, the roadside equipment updates the traffic light database based on each reliable second state sequence;

[0157] Specifically, this includes: taking the second state sequence corresponding to each reliable second judgment result as the current source sequence; taking the traffic light corresponding to the current source sequence as the current traffic light; taking the traffic light state sequence corresponding to the current traffic light in the traffic light database as the current target sequence; taking each second traffic light state in the current source sequence and its corresponding image timestamp as a set of corresponding signal states and signal timestamps to form a corresponding signal state data; taking all the signal state data corresponding to the current source sequence to form a corresponding current addition sequence; and adding the current addition sequence to the current target sequence.

[0158] Step 7: When the total number of unreliable second judgment results is not 0, the roadside equipment takes each unreliable second light image sequence as the current light image sequence; and takes the traffic light and the first sequence segment corresponding to the current light image sequence as the corresponding current traffic light and current sequence segment; and takes the timestamps of all images in the current sequence segment as the corresponding timestamp sequence; and takes the camera installation location, camera intrinsic and extrinsic parameters, the traffic light installation location of the current traffic light, the first video segment corresponding to the previous sequence segment, the timestamp sequence, and the second light image size as the corresponding remote analysis data packet and sends it to the remote cloud platform.

[0159] Here, the remote analysis data packet of this invention consists of a set of camera installation locations, camera intrinsic and extrinsic parameters, signal light installation locations, a first video segment, a timestamp sequence, and a second light image size.

[0160] Step 8: Each time the remote cloud platform receives a remote analysis data packet, it encapsulates the large model instruction based on the current data packet to obtain the corresponding traffic light status recognition instruction; and inputs the traffic light status recognition instruction into the multimodal large model on the platform side to perform traffic light status recognition to obtain the corresponding third state sequence; and sends the third state sequence back to the roadside equipment.

[0161] Specifically, this includes: Step 81, for each remote analysis data packet received, encapsulating the large model instruction based on the current data packet to obtain the corresponding signal light status recognition instruction;

[0162] Specifically, this includes: step 811, extracting the corresponding camera installation location, signal light installation location, camera intrinsic and extrinsic parameters, first video segment, timestamp sequence, and second light image size from the current remote analysis data packet;

[0163] Step 812: Extract frame images from the first video segment to obtain the corresponding second image sequence;

[0164] The second image sequence includes multiple second images;

[0165] Step 813: Based on the installation positions of the camera and the traffic light, confirm the relative pose of the first camera and the current traffic light to obtain the corresponding current relative pose;

[0166] Step 814: Take each second image in the second image sequence as the current image; estimate the pixel points corresponding to the traffic light installation position on the current image based on the camera intrinsic and extrinsic parameters and the current relative pose, and take the estimation result as the corresponding current traffic light center point; denote the rectangular area in the current image with the current traffic light center point as the rectangle center and the size of the second light image as the rectangle size as the current traffic light area; load the corresponding traffic light annotation box on the current image for the current traffic light area; take the current image with the annotation box as the corresponding third image; and form the third image sequence from all the obtained third images.

[0167] Step 815: Perform video reconstruction based on the third image sequence to obtain the corresponding instruction video; use the timestamp sequence as the corresponding video timestamp sequence; generate the corresponding model instruction text based on the first instruction template; and compose the corresponding traffic light status recognition instruction by combining the obtained model instruction text, instruction video, and video timestamp sequence.

[0168] It should be noted that the traffic light status recognition instruction in this embodiment of the invention includes model instruction text, instruction video, and video timestamp sequence; each frame of the instruction video has a traffic light label box; the rectangular size of the traffic light label box is consistent with the size of the second light image;

[0169] It should also be noted that the first instruction template in this embodiment of the invention is an instruction text encapsulation template corresponding to the traffic light status recognition instruction; the first instruction template is a preset natural language description text; the first instruction template is used to prompt the multimodal large model to identify the status of the traffic lights within the box of the traffic light annotation box on each frame of the instruction video image, and extract the status of the traffic lights within the box corresponding to each timestamp of the video timestamp sequence based on the identification result, and sort all the extracted states in chronological order to form the corresponding third state sequence and output it, and it is agreed that the status range of the traffic lights within the box includes red light, green light, yellow light, all black, all bright, and unknown;

[0170] It should also be noted that the multimodal large model in this embodiment of the invention is based on a type of visual text multimodal large model, such as the Qwen2.5-VL-7B model; this multimodal large model can be used to identify the state of traffic lights according to traffic light state recognition instructions and output the corresponding third state sequence; this multimodal large model has completed model training before being deployed to a remote cloud platform;

[0171] Step 82: Input the traffic light status recognition command into the multimodal large model on the platform side to perform traffic light status recognition and obtain the corresponding third state sequence;

[0172] Here, each third state sequence in this embodiment of the invention corresponds to a second light diagram sequence; the third state sequence includes multiple third signal light states; the third signal light states include red light, green light, yellow light, all black, all bright, and unknown; the third signal light state of the third state sequence corresponds to the second light in the currently corresponding second light diagram sequence. Figure 1 One-to-one correspondence;

[0173] Step 83, and send the third state sequence back to the roadside equipment.

[0174] Step 9: The roadside equipment updates the traffic light database based on the third state sequence;

[0175] Specifically, this includes: using each received third state sequence as the current source sequence; using the traffic light corresponding to the current source sequence as the current traffic light; using the traffic light state sequence corresponding to the current traffic light in the traffic light database as the current target sequence; using each third traffic light state in the current source sequence and its corresponding image timestamp as a set of corresponding signal states and signal timestamps to form a corresponding signal state data; using all the signal state data corresponding to the current source sequence to form a corresponding current addition sequence; and adding the current addition sequence to the current target sequence.

[0176] Figure 5 This is a schematic diagram of a processing system for recognizing traffic light status based on a multi-level model linkage, as provided in Embodiment 2 of the present invention. Figure 5As shown, the processing system for recognizing traffic light status based on multi-level model linkage provided in Embodiment 2 of the present invention includes: a roadside device 201 and a remote cloud platform 202. The roadside device 201 includes a data receiving module 2011, a data preprocessing module 2012, a first status recognition module 2013, a first status update module 2014, a second status recognition module 2015, a second status update module 2016, a first remote interaction module 2017, and a third status update module 2018. The remote cloud platform 202 includes a third status recognition module 2021.

[0177] The data receiving module 2011 of the roadside device 201 is used to receive the first video from the first camera 200; the roadside device 201 is an edge computing device deployed in a road segment or intersection scene; the roadside device 201 is connected to the first camera 200; there is at least one traffic light within the visual perception range of the first camera 200; the traffic light group includes red light, green light, and yellow light; the camera intrinsic and extrinsic parameters of the first camera 200 are known, and the camera installation position of the first camera 200 is fixed and known; the installation positions of the traffic lights corresponding to the first camera 200 are fixed and known.

[0178] The data preprocessing module 2012 of the roadside equipment 201 is used to extract frame images from the first video to obtain a first image sequence; and to perform sequential segmentation of the first image sequence to obtain a first sequence segment set; and to extract corresponding video segments from each sequence segment in the first video to obtain a first video segment set; and based on the installation position of the first camera and its corresponding traffic lights and the preset first and second light image sizes, to extract the two sizes of traffic light sub-images of each traffic light on each image of the first image sequence to obtain the corresponding first and second light images; and to generate corresponding first and second light image sequence sets for each sequence segment of the first sequence segment set.

[0179] The first state recognition module 2013 of the roadside equipment 201 is used to take each first light image sequence of each first light image sequence set as the current sequence; and according to HSV features Figure 3 The signal light status determination mechanism based on the proportion of color pixels analyzes the signal light status of each light in the current sequence and generates a corresponding first state sequence based on the analysis results; and determines the data consistency of the first state sequence to obtain a first judgment result; the first judgment result includes data consistency and data inconsistency.

[0180] The first state update module 2014 of the roadside device 201 is used to update the traffic light database based on the first state sequence of each data consistency when the total number of the first judgment results of data consistency is not 0; the traffic light database is deployed on the local or remote cloud platform 202 of the roadside device 201.

[0181] The second state recognition module 2015 of the roadside equipment 201 is used to take the second light diagram sequence corresponding to each first light diagram sequence with inconsistent data as the current sequence when the total number of the first judgment results of data inconsistency is not 0; and to use the signal light state recognition model on the equipment side to identify the signal light state of each light diagram in the current sequence and generate the corresponding second state sequence based on the recognition result; and to judge the data reliability of the second state sequence to obtain the second judgment result; the second judgment result includes reliable and unreliable.

[0182] The second state update module 2016 of the roadside device 201 is used to update the traffic light database based on each reliable second state sequence when the total number of reliable second judgment results is not 0.

[0183] The first remote interaction module 2017 of the roadside device 201 is used to take each unreliable second light image sequence as the current light image sequence when the total number of unreliable second judgment results is not 0; and take the traffic light and the first sequence segment corresponding to the current light image sequence as the corresponding current traffic light and current sequence segment; and take the timestamps of all images of the current sequence segment as the corresponding timestamp sequence; and take the camera installation position, camera intrinsic and extrinsic parameters, the traffic light installation position of the current traffic light, the first video segment corresponding to the previous sequence segment, the timestamp sequence and the second light image size as the corresponding remote analysis data packet and send it to the remote cloud platform 202.

[0184] The third state recognition module 2021 of the remote cloud platform 202 is used to encapsulate the corresponding traffic light state recognition instruction based on the large model instruction of the current data packet when it receives a remote analysis data packet; and input the traffic light state recognition instruction into the multimodal large model on the platform side to perform traffic light state recognition to obtain the corresponding third state sequence; and send the third state sequence back to the roadside device 201.

[0185] The third state update module 2018 of the roadside equipment 201 updates the traffic light database based on the third state sequence.

[0186] The present invention provides a processing system for recognizing traffic light status based on a multi-level model linkage, which can execute the method steps in the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0187] It should be noted that the division of the various system modules in the above system is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software through processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, the data receiving module can be a separate processing element, or it can be integrated into a chip in the above system. Alternatively, it can be stored as program code in a memory within the above system, and called and executed by a processing element of the system. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, the method steps of the above method embodiments or the modules of the above system embodiments can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0188] For example, these modules may be one or more integrated circuits configured to implement the above method embodiments, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). As another example, when a module of the above system embodiments is implemented in the form of processing element scheduler code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling program code. Furthermore, these modules may be integrated together and implemented as a system-on-a-chip (SOC).

[0189] In the above-described method or system embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the foregoing method embodiments are generated. The computer described above can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, Bluetooth, microwave, etc.) means. The aforementioned computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0190] This invention provides a method and system for identifying traffic light status based on a multi-level model linkage. As can be seen from the above, the roadside equipment in this invention first identifies traffic light status based on HSV characteristics. Figure 3 The signal light status determination mechanism based on color pixel ratio analyzes the signal light status of each traffic light, generates a first state sequence based on the analysis results, performs a data consistency check on it, and updates the signal light database based on each first state sequence that passes the data consistency check. If any first state sequence fails the data consistency check, the device-side signal light status recognition model identifies the signal light status of each traffic light that fails the data consistency check, generates a corresponding second state sequence based on the identification results, performs a reliability check on it, and updates the signal light database based on each second state sequence that passes the reliability check. If there are still second state sequences that fail the reliability check, the system interacts with a remote cloud platform, and a multimodal large model deployed on the platform identifies the signal light status of each traffic light that fails the reliability check, generates a corresponding third state sequence based on the identification results, and updates the signal light database based on the third state sequence. This invention not only reduces the overall computational load and overall memory usage of the signal light status recognition task on roadside equipment, but also improves the recognition accuracy.

[0191] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0192] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A processing method for recognizing a signal light state based on multi-level model linkage, characterized in that, The method includes: A roadside device receives a first video from a first camera; the roadside device is an edge computing device deployed in a road segment or intersection scenario; the roadside device is connected to the first camera; at least one traffic light exists within the visual perception range of the first camera; the traffic light group includes red, green, and yellow lights; the camera intrinsic and extrinsic parameters of the first camera are known, and the camera installation position of the first camera is fixed and known; the installation positions of the traffic lights corresponding to the first camera are fixed and known. The roadside equipment extracts frame images from the first video to obtain a first image sequence; and performs sequential segmentation on the first image sequence to obtain a first sequence segment set; and extracts corresponding video segments from each sequence segment in the first video to obtain a first video segment set; and based on the installation positions of the first camera and its corresponding traffic lights, as well as the preset first and second light image sizes, extracts two sizes of traffic light sub-images for each traffic light on each image of the first image sequence to obtain corresponding first and second light images; and generates corresponding first and second light image sequence sets for each sequence segment of the first sequence segment set. The roadside equipment takes each first light image sequence of each first light image sequence set as the current sequence; and analyzes the signal light status of each light image in the current sequence according to the signal light status determination mechanism based on the proportion of three-color pixels in the HSV feature map, and generates a corresponding first state sequence based on the analysis results; and judges the data consistency of the first state sequence to obtain a first judgment result; the first judgment result includes data consistency and data inconsistency. When the total number of the first judgment results with consistent data is not 0, the roadside device updates the traffic light database based on the first state sequence with consistent data; the traffic light database is deployed on the local or remote cloud platform of the roadside device. When the total number of the first judgment results with inconsistent data is not 0, the roadside equipment takes the second light diagram sequence corresponding to each of the first light diagram sequences with inconsistent data as the current sequence; and uses the signal light status recognition model on the equipment side to identify the signal light status of each light diagram in the current sequence and generates the corresponding second status sequence based on the recognition result; and judges the data reliability of the second status sequence to obtain a second judgment result; the second judgment result includes reliable and unreliable. When the total number of reliable second judgment results is not 0, the roadside equipment updates the traffic light database based on each reliable second state sequence; When the total number of unreliable second judgment results is not zero, the roadside equipment takes each unreliable second light image sequence as the current light image sequence; and takes the traffic light and the first sequence segment corresponding to the current light image sequence as the corresponding current traffic light and current sequence segment; and uses all the image timestamps of the current sequence segment to form a corresponding timestamp sequence; and uses the camera installation location, the camera's intrinsic and extrinsic parameters, the traffic light installation location of the current traffic light, the first video segment corresponding to the previous sequence segment, the timestamp sequence, and the second light image size to form a corresponding remote analysis data packet and sends it to the remote cloud platform. Each time the remote cloud platform receives a remote analysis data packet, it encapsulates the current data packet into a large model instruction to obtain a corresponding traffic light status recognition instruction; it then inputs the traffic light status recognition instruction into the multimodal large model on the platform side to perform traffic light status recognition to obtain a corresponding third state sequence; and finally sends the third state sequence back to the roadside equipment. The roadside equipment updates the traffic light database based on the third state sequence.

2. The processing method for recognizing traffic light status based on multi-level model linkage according to claim 1, characterized in that, The installation positions of the cameras and the installation positions of each of the traffic lights are all three-dimensional coordinates in the same three-dimensional coordinate system. The first image sequence includes a plurality of first images; each first image corresponds to an image timestamp; The first sequence segment set includes multiple first sequence segments; the first sequence segment includes multiple first images; the time span of the first sequence segment is 2L1, where L1 is a preset first duration; The segmentation rule for the first sequence segment is as follows: the video segment of the first video is sequentially divided into multiple first segment segments with a segment length of 2L1; and the first images in the first image sequence that are in each of the first segment segments are extracted to form the corresponding first sequence segments. And perform image deduplication on the two first sequence segments corresponding to every two adjacent first segment time periods; The first video segment set includes multiple first video segments; each first video segment corresponds one-to-one with a first sequence segment; the time span of each first video segment is less than or equal to 2L2, where L2 is a preset second duration and L2 > L1; The extraction rule for the first video segment is as follows: take the center time of the first segment time period corresponding to each first sequence segment as the current time, and extract the video content in each L2 before and after the current time in the first video to form the corresponding first video segment; The dimensions of the first light diagram include a first height and a first width; the dimensions of the second light diagram include a second height and a second width; the first height < the second height, and the first width < the second width. The first light image sequence set corresponds one-to-one with the first sequence segment; the first light image sequence set consists of one or more first light image sequences, and the first light image sequence corresponds one-to-one with the traffic lights; the first light image sequence includes multiple first light images; the image size of the first light image matches the size of the first light image. The second light image sequence set corresponds one-to-one with the first sequence segment; the second light image sequence set consists of one or more second light image sequences, and the second light image sequence corresponds one-to-one with the traffic lights; the second light image sequence includes multiple second light images; the image size of the second light image matches the size of the second light image; the first and second light image sequence sets correspond one-to-one, the first and second light image sequences in two corresponding sequence sets correspond one-to-one, and the first and second light images in two corresponding light image sequences correspond one-to-one. The first state sequence corresponds one-to-one with the first light pattern sequence; the first state sequence includes multiple first signal light states; the first signal light states include red light, green light, yellow light, all black, all bright, and unknown; the first signal light states in the first state sequence correspond one-to-one with the first light patterns in the currently corresponding first light pattern sequence; Each second state sequence corresponds to a second light diagram sequence; the second state sequence includes multiple second signal light states; the second signal light states include red light, green light, yellow light, all black, all bright, and unknown; the second signal light states of the second state sequence correspond one-to-one with the second light diagrams in the currently corresponding second light diagram sequence; Each of the third state sequences corresponds to a second light diagram sequence; the third state sequence includes multiple third signal light states; the third signal light states include red light, green light, yellow light, all black, all bright, and unknown; the third signal light states of the third state sequence correspond one-to-one with the second light diagrams in the currently corresponding second light diagram sequence; The traffic light database includes multiple traffic light status sequences; each traffic light status sequence corresponds one-to-one with a traffic light; each traffic light status sequence is formed by sequentially sorting multiple signal status data; each signal status data consists of a set of signal timestamps and signal status; the signal status includes red light, green light, yellow light, all black, all bright, and unknown.

3. The processing method for recognizing traffic light status based on multi-level model linkage according to claim 2, characterized in that, The traffic light status recognition model is based on a visual classification model. The model is used to predict traffic lights from the input traffic light image and output the corresponding predicted status and status confidence. The predicted status includes red, green, yellow, completely dark, fully lit, and unknown. The traffic light status recognition model has been trained before being deployed to the roadside equipment. The traffic light status recognition model consists of a feature extraction module and a classification task head connected together; the feature extraction network is implemented based on a CNN network or a residual network; the classification task head consists of a feature mapping network and a classification output layer connected sequentially; the feature mapping network consists of one or more feature transformation layers connected sequentially, and each feature transformation layer consists of a set of corresponding linear layers and ReLU activation function layers connected sequentially. The feature extraction module is used to perform image feature extraction processing on the traffic light image to obtain the corresponding image feature vector and send it to the classification task head; The classification task head is used to perform state classification prediction based on the image feature vector to obtain the corresponding predicted state and the state confidence score, and output them. Specifically, the feature mapping network maps the image feature vector to the traffic light state feature space to obtain the corresponding state feature vector and sends it to the classification output layer; the classification output layer inputs the state feature vector into the softmax function to calculate the state probability to obtain the corresponding state prediction vector, and takes the state type and probability value corresponding to the prediction probability with the highest probability value in the state prediction vector as the corresponding predicted state and the state confidence score, and outputs them; the state prediction vector consists of the prediction probability of red light state, green light state, yellow light state, all-black state, all-on state, and unknown state, and the sum of the probability values ​​of the six prediction probabilities of the state prediction vector is 1; The traffic light status recognition command includes model command text, command video, and video timestamp sequence; each frame of the command video has a traffic light annotation box. The rectangular dimensions of the signal light label frame are consistent with the dimensions of the second light diagram; The instruction text encapsulation template corresponding to the traffic light status recognition instruction is denoted as the first instruction template; The first instruction template is a pre-defined natural language description text; The first instruction template is used to prompt the multimodal large model to identify the state of the traffic lights within the bounding boxes of the traffic lights in each frame of the instruction video image, and to extract the state of the traffic lights within the bounding boxes corresponding to each timestamp of the video timestamp sequence based on the identification results. The extracted states are then sorted in chronological order to form the corresponding third state sequence and output. The state range of the traffic lights within the bounding boxes is defined as red light, green light, yellow light, all black, all bright, and unknown. The multimodal large model is implemented based on the Qwen2.5-VL-7B model; the multimodal large model is used to identify the traffic light status according to the traffic light status identification instruction and output the corresponding third state sequence; the multimodal large model has completed model training before being deployed to the remote cloud platform.

4. The processing method for recognizing the signal light state based on the multi-stage model linkage according to claim 2, characterized in that, Based on the installation positions of the first camera and its corresponding traffic lights, and the preset sizes of the first and second light images, the first and second light images are extracted from the two sizes of traffic light sub-images of each traffic light in each image of the first image sequence to obtain the corresponding first and second light images. Specifically, this includes: Based on the installation positions of the camera and each of the traffic lights, the relative pose of the first camera and its corresponding traffic lights is confirmed. Each of the first images in the first image sequence is taken as the current image; the pixel points corresponding to the installation positions of each traffic light in the current image are estimated based on the camera intrinsic and extrinsic parameters and the relative poses, and the estimation results are taken as the corresponding traffic light center points; a rectangular sub-image with the center points of each traffic light as the rectangle center and the size of the first light image as the rectangle size is extracted from the current image and taken as the corresponding first light image; and a rectangular sub-image with the center points of each traffic light as the rectangle center and the size of the second light image as the rectangle size is extracted from the current image and taken as the corresponding second light image.

5. The processing method for recognizing traffic light status based on multi-level model linkage according to claim 2, characterized in that, The signal light state determination mechanism based on the proportion of three-color pixels in the HSV feature map analyzes the signal light state of each light in the current sequence and generates a corresponding first state sequence based on the analysis results. Specifically, it includes: Step 51: Take each of the first light diagrams in the current sequence as the current light diagram; The current light pattern includes multiple first pixels; the pixel features of the first pixels include RGB pixel vectors; the RGB pixel vectors are composed of RGB three primary color pixel values; Step 52: Perform HSV feature vector conversion on each of the RGB pixel vectors of the current light image to obtain the corresponding HSV feature map; The HSV feature map includes multiple second pixels; the first and second pixels correspond one-to-one; the pixel features of the second pixel include the HSV feature vector; the HSV feature vector is composed of color feature H, lightness / darkness feature S, and brightness / darkness feature V. Step 53: Based on the preset red feature range, green feature range and yellow feature range, perform three types of mask image conversion on the HSV feature map to obtain the corresponding first, second and third mask images; Specifically: The second pixel in the HSV feature map whose color feature H satisfies the red feature range is designated as a first foreground pixel, and the second pixel that does not satisfy the red feature range is designated as a first background pixel. A first mask is formed by all the first foreground pixels and the first background pixels. Similarly, the second pixel in the HSV feature map whose color feature H satisfies the green feature range is designated as a second foreground pixel, and the second pixel that does not satisfy the green feature range is designated as a second background pixel. A second mask is formed by all the second foreground pixels and the second background pixels. Finally, the second pixel in the HSV feature map whose color feature H satisfies the yellow feature range is designated as a third foreground pixel, and the second pixel that does not satisfy the yellow feature range is designated as a third background pixel. A third mask is formed by all the third foreground pixels and the third background pixels. The image sizes of the first, second, and third mask images are all consistent with the image size of the HSV feature image; Step 54: Perform image noise reduction processing on the first, second, and third mask images; Specifically, the first, second, or third mask image is used as the current mask image, and the current mask image is first denoised using the morphological opening operation processing method, and then the denoised current mask image is filled with holes using the morphological closing operation processing method. Step 55: Calculate the proportion of foreground points corresponding to the first, second, and third masks to obtain the corresponding proportions. Specifically, the process involves: counting the total number of pixels in the HSV feature map to obtain a first total number; counting the total number of first, second, and third foreground pixels in the first, second, and third mask maps after noise reduction to obtain the corresponding first, second, and third foreground pixel totals; and calculating the corresponding first, second, and third percentages based on the first total number and the first, second, and third foreground pixel totals: first percentage = (total first foreground pixel total / first total number) × 100%, second percentage = (total second foreground pixel total / first total number) × 100%, and third percentage = (total third foreground pixel total / first total number) × 100%. Step 56: Based on the preset traffic light status determination rules, determine the corresponding first traffic light status according to the proportions of the first, second, and third traffic light indicators corresponding to the current traffic light diagram. The rule for determining the status of the traffic lights is as follows: SA represents the first traffic light state; a1, a2, and a3 represent the first, second, and third percentages, respectively; the six state types s1, s2, s3, s4, s5, and s6 represent all black, red, green, yellow, all on, and unknown, respectively. Step 57: The first state sequence is formed by sorting all the first signal light states corresponding to the current sequence in chronological order.

6. The processing method for recognizing traffic light status based on multi-level model linkage according to claim 2, characterized in that, The step of determining the data consistency of the first state sequence to obtain a first determination result specifically includes: The total number of the first traffic light states in the first state sequence is counted to obtain the corresponding second total number; the total number of traffic light states of the six state types in the first state sequence is counted to obtain the corresponding total number of the six state types; the maximum value among the total number of the six state types is taken as the third total number; and the corresponding first state percentage is calculated based on the second and third total numbers = (third total number / second total number) × 100%; and whether the first state percentage is greater than or equal to a preset state percentage threshold is identified; if yes, the corresponding first judgment result is set as data consistency; if no, the corresponding first judgment result is set as data inconsistency.

7. The processing method for recognizing traffic light status based on multi-level model linkage according to claim 3, characterized in that, The method of using the signal light status recognition model on the device side to identify the status of each signal light in the current sequence and generating a corresponding second status sequence based on the recognition results specifically includes: Each of the second light images in the current sequence is used as the current traffic light image and input into the traffic light state recognition model for processing to obtain the corresponding predicted state and the state confidence; the predicted state is used as the corresponding second traffic light state; and all the second traffic light states corresponding to the current sequence are sorted in chronological order to form the corresponding second state sequence.

8. The processing method for recognizing traffic light status based on multi-level model linkage according to claim 2, characterized in that, The step of determining the reliability of the data in the second state sequence to obtain a second determination result specifically includes: The total number of the second traffic light states in the second state sequence is counted to obtain the corresponding total number N1; the total number of traffic light states in the six state types of the second state sequence is counted to obtain the corresponding total number of the six state types; the maximum value among the total number of the six state types is taken as the corresponding maximum class total number N2; the corresponding maximum class confidence set is formed by the N2 state confidence scores corresponding to the maximum class total number N2; and the corresponding second state percentage is calculated as (N2 / N1)×100% based on the total number N1 and the maximum class total number N2. The system identifies whether the proportion of the second state is greater than or equal to a preset state proportion threshold. If the second state is less than the state proportion threshold, the corresponding second judgment result is set to unreliable. If the second state is greater than or equal to the state proportion threshold, the system identifies whether the confidence scores of all states in the maximum class confidence set are greater than a preset confidence threshold. If so, the corresponding second judgment result is set to reliable; otherwise, the corresponding second judgment result is set to unreliable.

9. The processing method for recognizing traffic light status based on multi-level model linkage according to claim 3, characterized in that, The process of encapsulating large model instructions based on the current data packet to obtain the corresponding traffic light status recognition instructions specifically includes: Step 91: Extract the corresponding camera installation location, traffic light installation location, camera intrinsic and extrinsic parameters, first video segment, timestamp sequence, and second light image size from the current remote analysis data packet; Step 92: Extract frame images from the first video segment to obtain the corresponding second image sequence; The second image sequence includes multiple second images; Step 93: Based on the installation position of the camera and the installation position of the traffic light, confirm the relative pose of the first camera and the current traffic light to obtain the corresponding current relative pose; Step 94: Take each second image in the second image sequence as the current image; estimate the pixel points on the current image corresponding to the installation position of the traffic light based on the camera intrinsic and extrinsic parameters and the current relative pose, and take the estimation result as the corresponding current traffic light center point; denote the rectangular area in the current image with the current traffic light center point as the rectangle center and the size of the second light image as the rectangle size as the current traffic light area; load the corresponding traffic light annotation box on the current image for the current traffic light area; take the current image with the completed annotation box as the corresponding third image; and form a third image sequence from all the obtained third images. Step 95: Perform video reconstruction based on the third image sequence to obtain the corresponding instruction video; use the timestamp sequence as the corresponding video timestamp sequence; generate the corresponding model instruction text based on the first instruction template; and combine the obtained model instruction text, the instruction video, and the video timestamp sequence to form the corresponding traffic light status recognition instruction.

10. A system for executing the processing method for recognizing traffic light states based on a multi-level model linkage as described in any one of claims 1-9, characterized in that, The system includes: roadside equipment and a remote cloud platform; the roadside equipment includes a data receiving module, a data preprocessing module, a first state recognition module, a first state update module, a second state recognition module, a second state update module, a first remote interaction module, and a third state update module; the remote cloud platform includes a third state recognition module. The data receiving module of the roadside device is used to receive the first video from the first camera; the roadside device is an edge computing device deployed in a road segment or intersection scene; the roadside device is connected to the first camera; there is at least one traffic light within the visual perception range of the first camera; the traffic light group includes red, green, and yellow lights; the camera intrinsic and extrinsic parameters of the first camera are known, and the camera installation position of the first camera is fixed and known; the installation positions of the traffic lights corresponding to the first camera are fixed and known. The data preprocessing module of the roadside equipment is used to extract frame images from the first video to obtain a first image sequence; and to perform sequential segmentation of the first image sequence to obtain a first sequence segment set; and to extract corresponding video segments from each sequence segment of the first video to obtain a first video segment set; and based on the installation position of the first camera and its corresponding traffic lights and the preset first and second light image sizes, to extract the two sizes of traffic light sub-images of each traffic light on each image of the first image sequence to obtain the corresponding first and second light images; and to generate corresponding first and second light image sequence sets for each sequence segment of the first sequence segment set; The first state recognition module of the roadside equipment is used to take each first light image sequence of each first light image sequence set as the current sequence; and analyze the signal light status of each light image in the current sequence according to the signal light status determination mechanism of the three-color pixel ratio of the HSV feature map and generate the corresponding first state sequence based on the analysis results; and judge the data consistency of the first state sequence to obtain a first judgment result; the first judgment result includes data consistency and data inconsistency. The first state update module of the roadside device is used to update the traffic light database based on the first state sequence with consistent data when the total number of the first judgment results with consistent data is not 0; the traffic light database is deployed locally on the roadside device or on the remote cloud platform. The second state recognition module of the roadside equipment is used to take the second light diagram sequence corresponding to each inconsistent first light diagram sequence as the current sequence when the total number of inconsistent first judgment results is not 0; and to use the signal light state recognition model on the equipment side to recognize the signal light state of each light diagram in the current sequence and generate the corresponding second state sequence based on the recognition result; and to judge the data reliability of the second state sequence to obtain a second judgment result; the second judgment result includes reliable and unreliable. The second state update module of the roadside equipment is used to update the traffic light database based on each reliable second state sequence when the total number of reliable second judgment results is not 0. The first remote interaction module of the roadside equipment is used to, when the total number of unreliable second judgment results is not 0, take each unreliable second light image sequence as the current light image sequence; and take the traffic light and the first sequence segment corresponding to the current light image sequence as the corresponding current traffic light and current sequence segment; and use the timestamps of all images of the current sequence segment to form a corresponding timestamp sequence; and use the camera installation location, the camera intrinsic and extrinsic parameters, the installation location of the current traffic light, the first video segment corresponding to the previous sequence segment, the timestamp sequence, and the size of the second light image to form a corresponding remote analysis data packet and send it to the remote cloud platform. The third state recognition module of the remote cloud platform is used to encapsulate the corresponding traffic light state recognition instruction based on the large model instruction of the current data packet when it receives a remote analysis data packet; and input the traffic light state recognition instruction into the multimodal large model on the platform side to perform traffic light state recognition to obtain the corresponding third state sequence; and send the third state sequence back to the roadside equipment. The third state update module of the roadside equipment updates the traffic light database based on the third state sequence.

Images