Traffic light time domain state detection method, model training method and related devices

Abstract

The present invention discloses a method for detecting the temporal state of a signal light, a model training method, and related devices. The method comprises: acquiring an image sequence, the image sequence comprising at least two consecutive frames containing a signal light; inputting the images in the image sequence into a signal light temporal state detection model in chronological order to obtain the signal light's temporal state. This method can identify temporal changes in signal light information and improve detection accuracy.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a signal light time domain state detection method, a model training method and related devices. Background Art

[0002] In intelligent traffic electronic police scenarios, accurate signal light positioning and status detection technology are crucial for regulating vehicle behavior. However, in order to simultaneously identify vehicle type, body color, license plate number, faces inside the vehicle, and other information, existing high-definition digital cameras require long exposure times for video streams and snapshots. The pixel values ​​in the signal light area are oversaturated, and the color of the red light is yellowish or white, close to the color of the yellow light, making it difficult to distinguish between the red and yellow light states.

[0003] Single-frame image detection technology has defects. For example, red and yellow lights are difficult to distinguish in color. Sometimes, electronic signal detectors are needed to output the status of signal lights, which increases the workload of technicians for debugging and requires additional signal detector equipment. This has low accuracy and increases product costs. Summary of the Invention

[0004] The main technical problem solved by the present invention is to provide a method, device and computer-readable storage medium for detecting the time domain status of a traffic light, which can identify the change information of the traffic light in the time domain and improve the detection accuracy.

[0005] In order to solve the above technical problems, a technical solution adopted by the present invention is: to provide a method for detecting the time domain state of a traffic light, which includes: obtaining an image sequence, the image sequence includes at least two consecutive frames of images, and the images contain a traffic light; inputting the images in the image sequence into a traffic light time domain state detection model according to the time order of the images to obtain the time domain state of the traffic light.

[0006] Among them, the traffic light time domain state detection model includes a feature extraction network and a detection network. The images in the image sequence are input into the traffic light time domain state detection model according to the time order of the images. Obtaining the time domain state of the traffic light includes: inputting the images in the image sequence into the feature extraction network according to the time order of the images to obtain multiple time domain feature information; inputting the multiple time domain feature information into the detection network to obtain the time domain state of the traffic light.

[0007] Among them, the feature extraction network includes a backbone network and an enhancement network. The images in the image sequence are input into the feature extraction network according to the time sequence of the images. Obtaining multiple time domain feature information includes: inputting the images in the image sequence into the backbone network according to the time sequence of the images to obtain multiple backbone feature information; inputting multiple backbone feature information into the enhancement network, performing information fusion on the multiple backbone feature information, and obtaining the time domain feature information of the traffic light.

[0008] Among them, multiple backbone feature information include at least first backbone feature information, second backbone feature information and third backbone feature information with decreasing dimensions in sequence. The multiple backbone feature information are input into the enhancement network, and the multiple backbone feature information are fused to obtain the time domain information of the traffic light, including: upsampling the third backbone feature information and fusing it with the second backbone feature information to obtain the second fused backbone feature information, upsampling the second fused backbone feature information and fusing it with the first backbone feature information to output the first time domain feature information; downsampling the first time domain feature information and fusing it with the second fused backbone feature information to output the second time domain feature information; and outputting the third backbone feature information to obtain the third time domain feature information.

[0009] Among them, the backbone network includes at least three residual network layers, and the images in the image sequence are input into the backbone network according to the time order of the images. Obtaining multiple backbone feature information includes: using the first residual network layer to perform downsampling feature extraction to obtain first backbone feature information; inputting the first backbone feature information into the second residual network layer for downsampling feature extraction to obtain second backbone feature information; inputting the second backbone feature information into the third residual network layer for downsampling feature extraction to obtain third backbone feature information.

[0010] The signal detection method further includes: detecting the time domain information of the signal light to obtain the state of the signal light, where the state of the signal light includes any one of on, off, unchanged, and changed.

[0011] To solve the above technical problems, another technical solution adopted by the present invention is: to provide a signal light time domain state detection model training method, which includes: obtaining a sample image sequence, the image sequence includes at least a continuous first sample image and a second sample image, the first sample image and the second sample image contain a signal light; obtaining a first annotation box of the signal light in the first sample image, and a second annotation box of the signal light in the second sample image; calculating the intersection-and-union ratio of the first annotation box and the second annotation box, the intersection-and-union ratio is the overlap rate of the annotation boxes; obtaining the signal light state corresponding to the intersection-and-union ratio; inputting the first sample image and the second sample image into the signal light time domain state detection model, and using the signal light state as the true value to train the signal light time domain state detection model, so as to realize the above signal light time domain state detection method.

[0012] Among them, obtaining the traffic light state corresponding to the intersection-and-union ratio includes: if the intersection-and-union ratio is less than or equal to the threshold and only the first annotation box is detected, the traffic light state is off; if the intersection-and-union ratio is less than or equal to the threshold and only the second annotation box is detected, the traffic light state is on; if the intersection-and-union ratio is greater than the threshold and the first annotation box and the second annotation box are different, the traffic light state is a state change; if the intersection-and-union ratio is greater than the threshold and the first annotation box and the second annotation box are the same, the traffic light state is an unchanged state.

[0013] In order to solve the above technical problems, another technical solution adopted by the present invention is: to provide a signal light detection device, which includes a processor, and the processor is used to execute to implement the above-mentioned signal light time domain state detection method or signal light time domain state detection model training method.

[0014] In order to solve the above technical problems, another technical solution adopted by the present invention is: providing a computer-readable storage medium, which is used to store instructions / program data, and the instructions / program data can be executed to implement the above-mentioned traffic light time domain state detection method or traffic light time domain state detection model training method.

[0015] The beneficial effects of the present invention are: different from the existing technology, the present invention uses a deep convolutional neural network to detect the time domain transformation information of traffic lights, which can identify the state changes of three types of traffic lights in the time domain: red, green and yellow lights, and can identify the change information of traffic lights in the time domain, thereby improving the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 1 is a flow chart of a method for detecting the time domain state of a signal light in an embodiment of the present application;

[0017] Figure 2 1 is a flow chart of another method for detecting the time domain state of a signal light in an embodiment of the present application;

[0018] Figure 3 1 is a flow chart of a method for training a signal light time domain state detection model in an embodiment of the present application;

[0019] Figure 4 This is a partial structural diagram of the signal light time domain state detection model of this application;

[0020] Figure 5 This is a schematic diagram of the labels of the time domain relationship between two consecutive frames of images in this application;

[0021] Figure 6 This is a schematic structural diagram of a signal light detection device in an embodiment of the present application;

[0022] Figure 7 This is a schematic structural diagram of a signal light detection device in an embodiment of the present application;

[0023] Figure 8 It is a schematic diagram of the structure of the computer-readable storage medium in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and effects of the present invention more clear and explicit, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0025] Considering the application scenarios of intelligent traffic video processing, video detection technology processes richer information than single-frame detection technology, including color, spatial information and time domain. Adding time domain information detection can provide more information for the detection system and help accurately determine the status of traffic lights. This application provides a method for detecting the time domain status of traffic lights. By adopting a deep convolutional neural network to perform time domain up-conversion information detection of traffic lights, it can identify the time domain status changes of three types of traffic lights: red, green and yellow, and can identify the change information of traffic lights in the time domain, thereby improving the detection accuracy.

[0026] See also Figure 1 , Figure 1 It is a flow chart of a method for detecting the time domain state of a signal light in an embodiment of the present application. It should be noted that if there are substantially the same results, this embodiment is not based on Figure 1 The process sequence shown is limited. Figure 1 As shown, this embodiment includes:

[0027] S110: Acquire an image sequence.

[0028] First, an image sequence to be detected is obtained. The image sequence includes at least two consecutive frames of images, each of which contains a signal light. The two consecutive frames of images are used to reflect the signal change information of the signal light in the time domain.

[0029] S130: Inputting the images in the image sequence into a traffic light time domain state detection model according to the time sequence of the images to obtain the time domain information of the traffic light.

[0030] A traffic light time domain state detection model is pre-trained, and two frames of images in the image sequence are merged in the channel direction according to the time sequence of the images and input into the traffic light time domain state detection model. Specifically, the channel of the image that comes first in the time sequence is also in front. The two consecutive frames of images are input into the trained traffic light time domain state detection model in time sequence, the traffic light information in the image is detected, and the changes in the traffic light information in the two frames of images are identified as the time domain information of the traffic light.

[0031] In this embodiment, by using a deep convolutional neural network to detect the time-domain up-conversion information of traffic lights, it is possible to identify the state changes of the three traffic lights, red, green, and yellow, in the time domain, and to identify the change information of traffic lights in the time domain, thereby improving the detection accuracy.

[0032] The traffic light time domain state detection model includes a feature extraction network and a detection network. The images in the image sequence are input into the feature extraction network according to the time order of the images to obtain multiple time domain feature information; the multiple time domain feature information is input into the detection network to obtain the time domain state of the traffic light.

[0033] The feature extraction network includes a backbone network and an enhancement network. The backbone network is used to extract the multi-dimensional feature information of the traffic light, and the enhancement network is used to fuse the multi-dimensional feature information to improve the feature expression ability of the backbone network. Figure 2 , Figure 2 It is a flow chart of another method for detecting the time domain state of a signal light in the embodiment of the present application. It should be noted that if there are substantially the same results, this embodiment is not based on Figure 2 The process sequence shown is limited. Figure 2 As shown, this embodiment includes:

[0034] S210: Acquire an image sequence.

[0035] First, an image sequence to be detected is obtained. The image sequence includes at least two consecutive frames of images, each of which contains a signal light. The two consecutive frames of images are used to reflect the signal change information of the signal light in the time domain.

[0036] S230: Input the images in the image sequence into the backbone network according to the time sequence of the images to obtain a plurality of backbone feature information.

[0037] Two consecutive image frames are input into the backbone network of a trained traffic light temporal state detection model in chronological order to detect traffic light information in the images. The backbone network includes one or more convolutional network layers and multiple residual network layers. The convolutional network layers are used for preliminary feature extraction. Multiple residual network layers are then used to extract multiple backbone feature information of successively decreasing dimensions as multidimensional features extracted by the backbone network. Specifically, the backbone network includes at least three residual network layers, with the first, second, and third residual network layers respectively extracting first, second, and third backbone feature information of successively decreasing dimensions.

[0038] S250: Input the multiple backbone feature information into the enhancement network, perform information fusion on the multiple backbone feature information, and obtain the time domain information of the traffic light. The time domain information of the traffic light includes the multiple time domain feature information.

[0039] Multiple backbone feature information is input into the enhancement network, and the enhancement network fuses multiple backbone feature information of different dimensions to obtain multiple time domain feature information of corresponding scales as the time domain information of the traffic light.

[0040] S270: Input multiple time domain feature information into the detection network to obtain a detection result.

[0041] The time domain information of the traffic light output by the enhanced network cannot be used to directly obtain the detection result. The time domain information of the traffic light can be detected to obtain the position and state of the traffic light. The position of the traffic light is a detection box containing the traffic light. The detection box is generally a rectangular box. This method can accurately locate the traffic light and obtain the minimum circumscribed rectangular box of the traffic light. The rectangular box information can also provide additional uses. The state of the traffic light includes any one of lighting up, extinguishing, unchanged state, and state changing. Therefore, multiple time domain feature information is input into the detection network to obtain the corresponding detection results. The detection results include the probability value of each state and the confidence of the detection result. Among them, the confidence of the detection result is the confidence of the network model. This method can accurately detect the state changes and state estimation of three traffic lights: red, green, and yellow, and mark the position information of the minimum circumscribed rectangle at the same time. There is no need to use a threshold method to limit it, which greatly improves the scene adaptability.

[0042] S290: Select the detection result with the highest confidence information as the state of the traffic light.

[0043] Analyze the multiple detection results output and select the detection result with the highest reliability information as the change state of the traffic light.

[0044] In this embodiment, the present application uses a deep convolutional neural network to detect the time-domain up-conversion information of traffic lights, and can identify the time-domain state changes of three types of traffic lights: red, green, and yellow. It can also identify the change information of traffic lights in the time domain, thereby improving the detection accuracy. The traffic light time-domain state detection method of the present application uses a CSP structure in the backbone network to accelerate the network forward reasoning while improving the feature extraction capability. The enhanced PAN structure in the network can complete the top-down and bottom-up feature fusion links, and the output network predicts the features at three scales, thereby improving the model's multi-scale detection capability. In actual applications, compared with traditional deep learning detection methods, only the input end of the deep neural network model changes from the original single frame to double frames, and the number of channels doubles. Since the additional calculation caused by the doubling of the model input channel has little impact on the entire model forward prediction, the proposed method can complete the extraction of time-domain information in video images at a very low computational cost.

[0045] In a specific implementation, before model detection is performed, a preliminary model training of the signal light time domain state detection model is required. Figure 3 , Figure 3 It is a flow chart of the method for training a signal light time domain state detection model in the embodiment of the present application. It should be noted that if there are substantially the same results, this embodiment does not use Figure 3The process sequence shown is limited.

[0046] like Figure 3 As shown, this embodiment includes:

[0047] S310: Collect video image data.

[0048] Imaging equipment is used to collect continuous video image data in the time domain. The collected video image data lies in the richness of samples. Repeated training samples not only waste computing resources, but also easily lead to overfitting of the model. Therefore, the richness of training samples has a crucial impact on the generalization ability of the model and the actual application effect. The video image data collection criteria should ensure that the training samples are the data predicted by the model in actual applications, and ensure that the training samples cover all sample types encountered in the actual application of the model. Therefore, the present application proposes a method for collecting video image data: first, the data characteristics of the video image samples are analyzed to obtain the main factors that cause the sample type to change; second, under each changing factor, an equal amount of video image data is collected. The video image data sample types collected by this method are richer, covering the main characteristics of the samples, which helps to improve the generalization ability of the model and the effect in actual use.

[0049] In the embodiments of the present application, due to different camera exposure parameters at different time periods, the colors of traffic lights in the captured video may vary. Different weather conditions may affect the image quality of traffic lights in the captured video, such as the presence of halos in rainy and cloudy weather. Therefore, surveillance cameras are used to capture surveillance video data containing traffic lights under various changing factors. Traffic light data with different outlines, such as arrow-shaped and pie-shaped traffic lights, are captured. Traffic light data with different colors and types, such as red, green, and yellow traffic lights, is captured. Traffic light data is captured at different time periods, such as early morning (4:00-6:00), morning (8:00-9:00), noon (12:00-1:00), afternoon (15:00-16:00), evening (17:00-18:00), and night (20:00-21:00), as well as traffic light data under different weather conditions, such as sunny, cloudy, and rainy days. Specifically, video image data is collected from the above-mentioned different feature dimensions, such as the red arrow light in the early morning on a sunny day and the yellow round light in the early morning on a sunny day, etc., and sufficient and equal amounts of video data are collected under each changing factor to maintain the balance between samples and obtain video image data with rich types.

[0050] S330: Label the video image data to obtain training sample data.

[0051] Traverse each frame of the video image data and use the labelimg annotation tool to mark the position and category of each traffic light in the image. Categories include red light, green light, and yellow light. Specifically, use a rectangular box to mark the traffic light and use (x0, y0, x1, y1) to represent the position of the traffic light, where (x0, y0) represents the coordinates of the upper left corner of the rectangular box of the traffic light and (x1, y1) represents the coordinates of the lower right corner of the rectangular box of the traffic light. Use red light, green light, and yellow light to represent the categories of red, green, and yellow lights.

[0052] The labeled data is organized into a large-scale training set, with each video image sequence as a sample. This training set contains many training samples. A training sample is an image sequence consisting of multiple images, arranged in time domain order. Each image has a corresponding annotation file.

[0053] S350: Construct a traffic light time-domain state detection model.

[0054] A deep learning network model is constructed as a traffic light time-domain state detection model. The traffic light time-domain state detection model consists of a backbone network, an enhancement network, and a detection network. The backbone network includes one or more convolutional network layers and multiple residual network layers. The convolutional network layers are used for preliminary feature extraction, and then multiple residual network layers are used to extract multiple backbone feature information with successively lower dimensions as the multi-dimensional features extracted by the backbone network. Figure 4 , Figure 4 This is a partial structural diagram of the signal light time domain state detection model of this application. Figure 4 It includes a backbone network and an enhanced network. The backbone network is constructed with 2 convolutional network layers and 3 CSP residual network layers in 5 stages, which downsamples the input resolution by 32 times and expands the channels to 512 to extract high-dimensional features of the input. Specifically, after the input image passes through 2 convolutional network layers, it is downsampled by 8 times using the first residual network layer and the channels are expanded to 128. The second residual network layer is then used to downsample by 2 times and the channels are expanded to 256. The third residual network layer is then used to downsample by 2 times and the channels are expanded to 512. The proposed model applies detection networks at 3 scales, corresponding to features downsampled by 8 times, 16 times, and 32 times, respectively, to enhance the model's multi-scale detection capabilities to adapt to different traffic light scenarios.

[0055] The enhanced network adopts a PAN structure, with both top-down and bottom-up paths, deeply fusing abstract high-level features with concrete low-level features to improve the feature expression capabilities of the backbone network. Specifically, the output features of the third residual network are sequentially upsampled, fused with the output results of the second residual network layer, fused with the output results of the first residual network layer, and then downsampled and fused with the output results of the second residual network layer, resulting in three output features of corresponding dimensions.

[0056] The detection network maps high-dimensional abstract features to the output features of the model. Post-processing operations parse the model output into prediction results according to certain rules. Each detection result requires 4 position information, 4 classification information, and 1 confidence data representation. Therefore, the high-dimensional features are mapped to 9 channel dimensions to represent all prediction results.

[0057] S370: Training a traffic light time-domain state detection model.

[0058] Randomly extract a video sequence sample from the training sample set, randomly extract two consecutive frames of images from the sequence, and construct a label for training and extracting time domain relationships based on the annotation information of the two frames of images. Figure 5 , Figure 5 This is a schematic diagram of the labels of the time domain relationship between two consecutive frames of images in this application. According to the information association of the annotation boxes on the images at time t-1 and t, the training labels of the time domain information are obtained. Specifically, the intersection-and-union ratio of each annotation box on the image at time t-1 and each annotation box on the image at time t is first calculated. The intersection-and-union ratio refers to the overlap rate between the two annotation boxes. When the value of the intersection-and-union ratio is greater than a certain threshold, it is considered that the annotation boxes are successfully associated, that is, the same annotation box is on the two frames before and after.

[0059] If the intersection-over-union ratio is less than or equal to the threshold and only the first annotation box is detected, that is, there are some annotation boxes on the image at time t-1 that are not associated with them on the image at time t, then it is considered that the lights are off at the locations of these annotation boxes from time t-1 to time t, case 0. If the intersection-over-union ratio is less than or equal to the threshold and only the second annotation box is detected, that is, there are some annotation boxes on the image at time t that are not associated with them on the image at time t-1, then it is considered that the lights are on at the locations of these annotation boxes from time t-1 to time t, case 2. If there are some annotation boxes that are successfully associated with time t-1 and time t, if the first and second annotation boxes are different, and the light state has changed before and after, then it is considered that the light state has changed at the locations of these annotation boxes, case 1. If the first and second annotation boxes are the same, that is, the light state has not changed before and after, then it is considered that the light state has not changed at the locations of these annotation boxes, case 4. Each case corresponds to a labeling result, and the labels of the time domain information of the training model are constructed using the above method. Figure 5The first labeled box shows the light off condition 0, the second labeled box shows the light changes from red to green condition 1, and the third labeled box shows the light on condition 2. The time domain label is consistent with the single frame label format, including a position box and a label information.

[0060] This application merges two frames of continuous images in the channel direction as the input of the model, that is, inputs images of six channels, takes the labels of the above-mentioned time domain information as the true value, and uses the Darknet deep learning framework for model training. During the training process, the batch size is set to 64, the subdivisions is 4, the SGD optimization algorithm is used to reduce the training error, the momentum is set to 0.9, the learning rate adopts a step-by-step decay strategy, the initial size is set to 0.001, the maximum number of iterations is 500200, and when the number of iterations reaches 400,000 and 450,000, the learning rate is reduced by 10 times. Data enhancement such as random cropping, stretching, jitter exposure parameters, Mosaic and multi-scale training are used. The training method of the deep neural network model is basically the same as that of single-frame target detection. Compared with the cyclic recursive network that extracts time domain information, it is simpler to implement and more efficient in calculation. Therefore, the ability to extract time domain information in video images can be completed at a very low computational cost.

[0061] The images to be detected are input into the trained traffic light time domain state detection model for detection. Specifically, the images in the image sequence are input into the backbone network in chronological order. After passing through the convolutional network layer, the first residual network layer is used to perform downsampling feature extraction to obtain the first backbone feature information; the first backbone feature information is input into the second residual network layer for downsampling feature extraction to obtain the second backbone feature information; the second backbone feature information is input into the third residual network layer for downsampling feature extraction to obtain the third backbone feature information. Multiple backbone feature information are input into the enhancement network, the third backbone feature information is upsampled and fused with the second backbone feature information to obtain the second fused backbone feature information, the second fused backbone feature information is upsampled and fused with the first backbone feature information to output the first time domain feature information; the first time domain feature information is downsampled and fused with the second fused backbone feature information to output the second time domain feature information; and the third backbone feature information is output to obtain the third time domain feature information.

[0062] Multiple time domain feature information is input into the detection network to obtain the detection results.

[0063] The detection results are then post-processed. Specifically, the detection results output by the above model cannot directly obtain the time domain information of the traffic light. Therefore, it is necessary to activate the prediction box, confidence, and classification information in the output features. The prediction box information is the prediction box position, including the center point coordinates and width and height. The confidence information refers to the current predicted probability value. The classification information refers to the probability value of each category predicted by the current prediction box. The activation formula is as follows:

[0064] b x =σ(t x )+c x ,

[0065] b y =σ(t y )+c y ,

[0066]

[0067]

[0068] P=σ(t0)*max(σ(t cls )).

[0069] Among them, t x , t y , t w , t h , t o , t cls To output feature information, c x 、c y is the reference coordinate of the detection box, p w 、p h is the preset box width and height, σ() is the sigmoid function.

[0070] After completing the activation of the output feature data, the prediction frames with higher confidence are screened out according to the threshold. Each frame corresponds to a situation, such as the light on, light off, state change or state unchanged in the time domain label. The prediction frame with the highest built-in confidence in the neighborhood is selected as the final output. Specifically, all prediction frames are divided into different sets according to different categories; each category set is traversed and the prediction frames in the set are sorted in descending order of confidence; each category set is traversed, starting with the first prediction frame as the reference frame, and its intersection and union ratio with the following ones is calculated. Prediction frames greater than a certain threshold will be suppressed, that is, removed from the set; the reference frame in each category set is traversed to the last one, and the final prediction result is output.

[0071] In another embodiment, the state of the traffic light at the previous moment is obtained. Using the aforementioned signal light time-domain state detection model, logical inference can be used to estimate the current state of the traffic light, thereby obtaining the current state information of the traffic light. Compared to the single-frame state detection method, this proposed method utilizes time-domain information.

[0072] In this implementation, a deep convolutional neural network is used to detect time-domain upconversion information of traffic lights. Through learning, training, and testing a large number of monitoring scene samples, model parameters with a high recognition rate are obtained. This model can accurately detect the time-domain state changes of three traffic lights: red, green, and yellow. This eliminates the need to consider red and green color information, set threshold parameters, or control camera exposure, thus reducing the workload of manual parameter debugging. By extracting the time-domain information of traffic lights, the original state detection system is integrated with information from the time dimension. Through a simple and effective fusion method, more accurate estimation results can be obtained, to a certain extent avoiding the problems of missed detection and false detection associated with single-frame detection. Furthermore, the system can be used with standard police surveillance cameras without requiring any additional hardware. It does not change the overall brightness of the image, nor does it affect image quality, thus reducing hardware and manual debugging costs.

[0073] See also Figure 6 , Figure 6 FIG. 6 is a schematic diagram of the structure of a signal light detection device in an embodiment of the present application. In this embodiment, the signal light detection device includes an acquisition module 61 and a detection module 62 .

[0074] The acquisition module 61 is used to acquire an image sequence, which includes at least two consecutive frames containing traffic lights. The detection module 62 is used to input the images in the image sequence into a traffic light temporal state detection model in chronological order to obtain the traffic light's temporal state. This traffic light detection device utilizes a deep convolutional neural network to detect traffic light temporal up-conversion information. It can identify temporal state changes of three types of traffic lights: red, green, and yellow. It can also identify temporal changes in traffic lights, thereby improving detection accuracy.

[0075] See also Figure 7 , Figure 7 FIG. 7 is a schematic diagram of the structure of a signal light detection device in an embodiment of the present application. In this embodiment, the signal light detection device 71 includes a processor 72 .

[0076] The processor 72 may also be referred to as a CPU (Central Processing Unit). The processor 72 may be an integrated circuit chip having signal processing capabilities. The processor 72 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. A general-purpose processor may be a microprocessor, or the processor 72 may be any conventional processor.

[0077] The signal light detection device 71 may further include a memory (not shown in the figure) for storing instructions and data required for the processor 72 to operate.

[0078] The processor 72 is configured to execute instructions to implement the method provided by any embodiment of the signal light time domain state detection method of the present application and any non-conflicting combination thereof.

[0079] See also Figure 8 , Figure 8 : is a schematic diagram of the structure of the computer-readable storage medium in the embodiment of the present application. The computer-readable storage medium 81 of the embodiment of the present application stores instruction / program data 82, which, when executed, implements the method provided by any embodiment of the signal light time domain state detection method of the present application and any non-conflicting combination. Among them, the instruction / program data 82 can be formed into a program file and stored in the above-mentioned storage medium 81 in the form of a software product, so that a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) executes all or part of the steps of the methods of each embodiment of the present application. The aforementioned storage medium 81 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or terminal devices such as a computer, a server, a mobile phone, and a tablet.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0081] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0082] The above description is only an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present invention, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for detecting the time domain state of a signal light, characterized in that: The method comprises: Acquire an image sequence, wherein the image sequence includes at least two consecutive frames of images, and the images include a traffic light; Inputting the images in the image sequence into a traffic light time domain state detection model according to the time sequence of the images to obtain the time domain state of the traffic light; The training step of the traffic light time domain state detection model includes: Acquire a sample image sequence, the image sequence including at least a first sample image and a second sample image that are continuous, wherein the first sample image and the second sample image include a signal light; Acquire a first annotation frame of the traffic light in the first sample image, and a second annotation frame of the traffic light in the second sample image; Calculating an intersection-over-union (IoU) ratio of the first annotation frame to the second annotation frame, where the IoU ratio is an overlap ratio of the annotation frames; Obtaining the time domain state of the traffic light corresponding to the intersection-to-combination ratio; The first sample image and the second sample image are input into a traffic light time domain state detection model, and the traffic light time domain state is used as a true value to train the traffic light time domain state detection model.

2. The signal light time domain state detection method according to claim 1, characterized in that: The signal light temporal state detection model includes a feature extraction network and a detection network. Inputting the images in the image sequence into the signal light temporal state detection model in chronological order to obtain the temporal state of the signal light includes: Inputting the images in the image sequence into the feature extraction network in a time sequence of the images to obtain a plurality of time domain feature information; The multiple time domain feature information are input into the detection network to obtain the time domain state of the traffic light.

3. The signal light time domain state detection method according to claim 2, characterized in that: The feature extraction network includes a backbone network and an enhancement network. Inputting images in the image sequence into the feature extraction network in a temporal order of the images to obtain a plurality of time domain feature information includes: Inputting images in the image sequence into the backbone network in a time sequence of the images to obtain a plurality of backbone feature information; The plurality of backbone feature information are input into the enhancement network, and information fusion is performed on the plurality of backbone feature information to obtain the time domain feature information of the traffic light.

4. The method for detecting the time domain state of a signal light according to claim 3, characterized in that: The multiple backbone feature information includes at least first backbone feature information, second backbone feature information, and third backbone feature information of decreasing dimensions. The multiple backbone feature information are input into the enhancement network, and information fusion is performed on the multiple backbone feature information to obtain the time domain information of the traffic light, including: Upsampling the third backbone feature information and fusing it with the second backbone feature information to obtain second fused backbone feature information; upsampling the second fused backbone feature information and fusing it with the first backbone feature information to output first time domain feature information; Downsampling the first time domain feature information and fusing it with the second fused backbone feature information to output second time domain feature information; The third backbone feature information is output to obtain third time domain feature information.

5. The method for detecting the time domain state of a signal light according to claim 4, characterized in that: The backbone network includes at least three residual network layers, and the inputting the images in the image sequence into the backbone network in the time order of the images to obtain a plurality of backbone feature information includes: Using the first residual network layer to perform downsampling feature extraction to obtain the first backbone feature information; Inputting the first backbone feature information into the second residual network layer for downsampling feature extraction to obtain second backbone feature information; The second backbone feature information is input into the third residual network layer for downsampling feature extraction to obtain third backbone feature information.

6. The method for detecting the time domain state of a signal light according to claim 2, characterized in that: The method further comprises: The time domain information of the signal light is detected to obtain the time domain state of the signal light, where the time domain state of the signal light includes any one of: on, off, unchanged, and changed.

7. The signal light time domain state detection method according to claim 1, characterized in that: The acquiring of the time domain state of the traffic light corresponding to the intersection-to-combination ratio includes: If the intersection-over-union ratio is less than or equal to a threshold and only the first marking frame is detected, the time domain state of the traffic light is off; If the intersection-over-union ratio is less than or equal to the threshold and only the second annotation frame is detected, the time domain state of the traffic light is on; If the intersection-over-union ratio is greater than a threshold and the first marking frame and the second marking frame are different, the time domain state of the traffic light is a state change; If the intersection-over-union ratio is greater than a threshold and the first marking box and the second marking box are the same, the time-domain state of the traffic light is unchanged.

8. A signal light detection device, characterized in that: The method comprises a processor configured to execute instructions to implement the signal light time domain state detection method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store instructions / program data, and the instructions / program data can be executed to implement the signal light time-domain state detection method according to any one of claims 1 to 7.

Images