Traffic signal lamp identification and preset network model training method and device
Feature extraction and fusion of images through preset network models, the problems of low traffic light recognition efficiency and poor stability in autonomous driving systems are solved, and more accurate and stable traffic light recognition is achieved, improving the safety and users of urban autonomous driving. Experience.
Patent Information
- Application Number
- CN202410992852.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-08
AI Technical Summary
The existing traffic light recognition methods have problems with low identification efficiency and poor stability in the field of autonomous driving, especially in urban environments, where vehicles are far away from traffic lights and may be blocked, resulting in inaccurate identification results, affecting traffic safety and User experience.
The preset network model is used to extract and fuse the image feature. By fusing the current frame image features with the previous frame image features, and using a deep neural network to identify traffic lights, improving the recognition efficiency and stability.
It improves the accuracy and stability of traffic lights identification, reduces the problem of incomplete information caused by occlusion, and improves the traffic efficiency and safety of autonomous driving systems in urban environments.
Smart Images

Figure CN118918564A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to computer vision technology, and in particular to a method and device for identifying traffic lights and training a preset network model. Background Art
[0002] In the new stage of autonomous driving, urban autonomous driving has become the focus. Among them, traffic light detection, as a major challenge, has become increasingly important against the backdrop of accelerated urbanization. There are two main existing methods for traffic light recognition, one based on visual image recognition and the other based on vehicle-to-vehicle wireless communication technology (V2X). V2X means that the traffic light sends the signal status in real time, and the vehicle receives the signal with low latency. This method requires the transformation of the traffic light, which not only places too high demands on the signal capability of the basic equipment, but also places high demands on network transmission. Based on visual image recognition, the various status information of the traffic light is obtained by detecting and classifying 2D images. The disadvantage of this method is that it makes the simple traffic light recognition function very bloated and reduces scalability. Summary of the invention
[0003] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide a method and device for identifying a traffic light and training a preset network model.
[0004] According to one aspect of an embodiment of the present disclosure, a method for identifying a traffic light is provided, comprising:
[0005] Inputting the first image into a preset network model;
[0006] Performing feature extraction on the first image based on the first subnetwork of the preset network model to obtain first image features;
[0007] The first image feature is fused with the second image feature to obtain a third image feature; the second image feature is an image feature obtained by extracting and fusing features of a previous frame of the first image through the first sub-network;
[0008] The third image feature is processed by the second sub-network in the preset network model to obtain a traffic light recognition result.
[0009] According to another aspect of an embodiment of the present disclosure, a method for training a preset network model is provided, comprising:
[0010] Inputting a first sample image and a second sample image into a preset network model respectively, and outputting a plurality of first recognition data and a plurality of second recognition data respectively through a plurality of recognition branches of the preset network model; wherein the first sample image and the second sample image are adjacent frame images continuously collected based on traffic lights; the plurality of first recognition data correspond to the first sample image, and the plurality of second recognition data correspond to the second sample image;
[0011] determining a network loss based on the plurality of first identification data and the plurality of second identification data;
[0012] The preset network model is trained based on the network loss.
[0013] According to another aspect of the embodiments of the present disclosure, there is provided a traffic light recognition device, comprising:
[0014] An image preprocessing module, used for inputting the first image into a preset network model;
[0015] A feature extraction module, used to extract features of the first image based on the first sub-network of the preset network model to obtain first image features;
[0016] A feature fusion module, used to fuse the first image feature with the second image feature to obtain a third image feature; the second image feature is an image feature obtained by extracting features of a previous frame of the first image through the first sub-network;
[0017] The traffic light recognition module is used to process the third image feature through the second sub-network in the preset network model to obtain a traffic light recognition result.
[0018] According to another aspect of the embodiments of the present disclosure, a training device for a preset network model is provided, comprising:
[0019] A model processing module, used to input a first sample image and a second sample image into a preset network model respectively, and output a plurality of first recognition data and a plurality of second recognition data respectively through a plurality of recognition branches of the preset network model; wherein the first sample image and the second sample image are adjacent frame images collected continuously; the plurality of first recognition data correspond to the first sample image, and the plurality of second recognition data correspond to the second sample image;
[0020] a loss determination module, configured to determine a network loss based on the plurality of first identification data and the plurality of second identification data;
[0021] A network training module is used to train the preset network model based on the network loss.
[0022] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the traffic light recognition method described in any of the above embodiments or the preset network model training method described in any of the above embodiments.
[0023] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising:
[0024] processor;
[0025] a memory for storing instructions executable by the processor;
[0026] The processor is used to read the executable instructions from the memory and execute the instructions to implement the traffic light recognition method described in any of the above embodiments or the preset network model training method described in any of the above embodiments.
[0027] Based on the above-mentioned embodiment of the present disclosure, a method and device for identifying a traffic light and training a preset network model are provided. A first image is input into a preset network model; a first sub-network based on the preset network model performs feature extraction on the first image to obtain a first image feature; the first image feature is fused with the second image feature to obtain a third image feature; the second image feature is an image feature obtained by extracting features of the previous frame of the first image through the first sub-network; the third image feature is processed by the second sub-network in the preset network model to obtain a traffic light recognition result. In this embodiment, a preset network model is applied to predict an image to determine the traffic light recognition result, thereby improving recognition efficiency; and, in this embodiment, by fusing the current frame image feature with the previous frame image feature before the result recognition, the problem of incomplete information in the current frame image due to occlusion and other reasons is solved through feature fusion, thereby improving the stability and accuracy of traffic light recognition.
[0028] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.
[0030] Figure 1is a flow chart of a method for identifying a traffic light provided by an exemplary embodiment of the present disclosure;
[0031] Figure 2 is a schematic diagram of a flow chart of determining a first image in a method for recognizing a traffic light provided by another exemplary embodiment of the present disclosure;
[0032] Figure 3 is a schematic diagram of a second image obtained without mask processing in an optional example of the present disclosure;
[0033] Figure 4 yes Figure 3 A schematic diagram of a first image obtained by masking a second image shown;
[0034] Figure 5 is a flow chart of a method for identifying a traffic light provided by another exemplary embodiment of the present disclosure;
[0035] Figure 6 is a flowchart of a method for training a preset network model provided by an exemplary embodiment of the present disclosure;
[0036] Figure 7 It is a flowchart of determining network loss in a training method of a preset network model provided by an exemplary embodiment of the present disclosure;
[0037] Figure 8 is a schematic structural diagram of a traffic light recognition device provided by an exemplary embodiment of the present disclosure;
[0038] Fig. 9 is a structural schematic diagram of a training device for a preset network model provided by an exemplary embodiment of the present disclosure;
[0039] Fig.10 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0040] In order to explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure. It should be understood that the present disclosure is not limited to the example embodiments described herein.
[0041] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.
[0042] The disclosed embodiments can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.
[0043] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.
[0044] Application Overview
[0045] In the process of realizing the present disclosure, the inventors found that in the field of autonomous driving, especially in urban autonomous driving, traffic light perception and recognition is one of the indispensable functions for traffic at intersections. The research object of perception includes the spatial information of the current frame of the traffic light and the corresponding attribute information (color, type, countdown, etc.), and the perception result of the current frame is processed. After a complex voting strategy, a stable output result in the time dimension is obtained to avoid dangerous behaviors such as sudden braking at a green light or starting at a red light during vehicle driving due to the jump of the perception output.
[0046] However, the existing technology has at least the following problems: although complex voting strategies can improve the success rate of intersection traffic, long voting queues will seriously reduce the efficiency of intersection traffic (for example, a voting queue of 11 frames represents a red light turning green, and the vehicle needs to go through at least 6 frames before it can start). In addition, at intersections with wider roads, the distance between the vehicle and the opposite traffic light may exceed 80 meters, and there may be objects such as wires and leaves that partially block the traffic light. The perception results are difficult to stably and accurately identify, which greatly affects the user's riding experience and the safety of urban traffic.
[0047] Exemplary Methods
[0048] Figure 1FIG. 1 is a flow chart of a method for identifying a traffic light provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Figure 1 As shown, the following steps are included:
[0049] Step 102: input the first image into a preset network model.
[0050] In this embodiment, the first image is an image obtained in any manner. It should be noted that the image may or may not include a traffic light. When the image includes a traffic light, the following steps are performed to obtain an effective traffic light recognition result; and when the image does not include a traffic light, the following steps are still performed, but no effective traffic light recognition result can be obtained. Optionally, the first image is an image including at least one traffic light; for example, an image directly captured by a vehicle-mounted camera, or, for another example, an image captured by a vehicle-mounted camera after preprocessing (for example, cropping, masking, etc.). The preset network model can be any deep neural network that can realize feature extraction and object recognition. This embodiment does not limit the network structure of the preset network model. For example, the preset network model can be a recognition network, etc.
[0051] Step 104: extract features of the first image based on the first subnetwork of the preset network model to obtain first image features.
[0052] In one embodiment, the preset network model includes two parts: a first sub-network and a second sub-network, the first sub-network realizes feature extraction of the image; optionally, the first image feature is a deep feature of the image, for example, a deep semantic feature, etc. Optionally, the first sub-network is a backbone network, for example, the first sub-network includes at least one convolution layer, and features of the image are extracted through at least one convolution layer.
[0053] Step 106: Fusing the first image feature with the second image feature to obtain a third image feature.
[0054] Among them, the second image feature is the image feature obtained by extracting and fusing the feature of the previous frame image of the first image through the first sub-network.
[0055] In this embodiment, the previous frame image may be a previous frame image obtained by the same processing as the first image or based on the same method; the previous frame image is also input into the preset network model, and after the first sub-network of the preset network model outputs the image feature, it is fused with the fusion feature outputted by the previous frame image to obtain the corresponding second image feature. Optionally, when the first image is the first frame image, that is, the first image does not have a corresponding previous frame image, at this time, the second image feature may be an all-0 feature with the same dimension and size as the first image feature, and the third image feature with the same dimension and size as the first image feature is obtained by fusion. The dimension and size of the third image feature in this embodiment are the same as the first image feature. After the third image feature is obtained by fusion, in addition to determining the traffic light recognition result based on the third image feature; the third image feature is also used as the second image feature corresponding to the next frame of the first image, and is fused with the first image feature obtained by the next frame of the image to obtain a new third image feature. By analogy, the third image feature after the fusion of the first image feature and the second image feature of each frame is used as the second image feature corresponding to the next frame of the image, and the fusion is performed in sequence. In this embodiment, the third image feature corresponding to each frame of the first image is integrated with the image feature of the previous frame through feature fusion. By associating the previous and next frames, a relatively accurate prediction result can be output even if there is occlusion in the current frame image, thereby improving the recognition accuracy of traffic lights.
[0056] Step 108: Process the third image feature through the second sub-network in the preset network model to obtain a traffic light recognition result.
[0057] In this embodiment, since the traffic lights have multiple attributes, such as color, category, direction, countdown, etc., in addition, a first image may also include multiple traffic lights, and the attributes also need to include corresponding light identifiers (different identifiers represent different groups of traffic lights, usually a group of traffic lights represent a traffic information, and the same group of traffic lights are usually distributed in a row). Optionally, the second sub-network may include multiple recognition branches or output ends (heads), and multiple attributes corresponding to the traffic lights are output through multiple output ends, each output end corresponding to one attribute; optionally, the second sub-network may also output traffic light recognition results corresponding to multiple attributes based on one output end at the same time, for example, different attributes are represented based on information at different positions in a vector, for example, the output vector is [0001 0011 0010 0101], wherein the first 4 bits represent the color, the 5th to 8th bits represent the category, the 9th to 12th bits represent the direction, the 13th to 16th bits represent the countdown, etc. This example is only an optional example, and the representation of data in a specific embodiment can be set according to a specific application scenario.
[0058] The above-mentioned embodiment of the present disclosure provides a method for identifying a traffic light, which inputs a first image into a preset network model; extracts features of the first image based on the first subnetwork of the preset network model to obtain first image features; fuses the first image features with the second image features to obtain third image features; the second image features are image features obtained by extracting features of the previous frame of the first image through the first subnetwork; and processes the third image features through the second subnetwork in the preset network model to obtain a traffic light recognition result. In this embodiment, a preset network model is applied to predict images to determine traffic light recognition results, thereby improving recognition efficiency; and, in this embodiment, by fusing the current frame image features with the previous frame image features before result recognition, the problem of incomplete information in the current frame image due to occlusion and other reasons is solved through feature fusion, thereby improving the stability and accuracy of traffic light recognition.
[0059] In some optional embodiments, the method provided by the present disclosure may further include:
[0060] The first sample image and the second sample image are respectively input into a preset network model, and a plurality of first recognition data and a plurality of second recognition data are respectively output through a plurality of recognition branches of the preset network model. The first sample image and the second sample image are adjacent frame images continuously collected based on traffic lights, the plurality of first recognition data correspond to the first sample image, and the plurality of second recognition data correspond to the second sample image.
[0061] Based on the plurality of first identification data, the plurality of second identification data, and the plurality of first reference values, a network loss is determined.
[0062] In this embodiment, each first reference value corresponds to an identification branch. The first reference value is used to indicate whether the labels of the previous and next frames are consistent. The first reference value can be determined based on the marked true value label. Optionally, when the true value labels of the previous and next frames are consistent, the first reference value is 1. When they are inconsistent, the first reference value is 0.
[0063] Train the preset network model based on the network loss.
[0064] In this embodiment, the preset network model is trained in combination with two consecutive frames of images, and the first sample image and the second sample image serve as supervisory information for each other. In this embodiment, it is only necessary to mark whether the true value labels of the traffic lights corresponding to the first sample image and the second sample image in different recognition branches are consistent, which reduces the labeling information, greatly reduces the workload of manual labeling, and improves the accuracy of the preset network model.
[0065] Figure 2 FIG. 1 is a flow chart of determining a first image in a method for identifying a traffic light provided by another exemplary embodiment of the present disclosure. Figure 2 As shown, determining the first image may include:
[0066] Step 202 , respectively detect the acquired multiple frames of target images to obtain multiple first detection frames.
[0067] Each frame of the target image corresponds to at least one first detection frame.
[0068] Optionally, the target image can be obtained through an image acquisition device such as a vehicle-mounted camera, and multiple frames of target images can be acquired continuously; each frame of the target image is detected separately (existing detection technology can be used, such as a detection network model, etc.) to obtain at least one first detection frame, the purpose of which is to select the traffic light in the target image. Optionally, since the target image is acquired continuously, a path can be established for the first detection frame, and the position of the first detection frame in the current frame target image can be inferred from the first detection frame of the adjacent frame to obtain the first detection frame corresponding to the current frame target image. The first detection frame can represent its size and position information based on the coordinates of the four corners. For example, the four coordinates of a first detection frame are: (x1, y1), (x1, y2), (x2, y1), (x2, y2).
[0069] Step 204: Perform scaling processing on each first detection frame in the plurality of first detection frames according to the network parameters of the preset network model to obtain a second detection frame.
[0070] Different preset network models can process different image sizes. In this embodiment, the receivable image size (for example, 96 pixels x 96 pixels, etc.) is determined according to the model parameters of the preset network model, and the size of the first detection frame is scaled according to the size so that the size of the second detection frame obtained after the scaling process meets the requirements of the preset network model. For example, after the coordinates of the first detection frame are processed, the four coordinates of the second detection frame are: (x1_1, y1_1), (x1_1, y2_1), (x2_1, y1_1), and (x2_1, y2_1).
[0071] Step 206: determine a plurality of first images based on the plurality of second detection frames.
[0072] Each second detection frame corresponds to a first image.
[0073] Optionally, when the target image includes multiple groups of traffic lights, each group of traffic lights is selected separately by using multiple first detection frames to segment the multiple groups of traffic lights, so that each first detection frame only includes one group of traffic lights, and correspondingly, each second detection frame also only includes one group of traffic lights.
[0074] In this embodiment, at least one preprocessing is performed on the target image to obtain a second detection frame corresponding to the size of the preset network model input image, and the first image is obtained based on the second detection frame, so that the size of the first image meets the input requirements of the preset network model, thereby avoiding the problem of reduced efficiency caused by inputting an image that does not meet its input size to the preset network model, and improving the processing speed of the preset network model.
[0075] Optionally, based on the above embodiment, step 206 may further include:
[0076] A second image is obtained by cropping the corresponding target image based on each second detection frame.
[0077] Optionally, after the second detection frame is determined, based on the coordinates of the four corners corresponding to the second detection frame, an image of corresponding coordinates is re-cropped from the original target image to obtain a second image.
[0078] Mask processing is performed on each second image to obtain a plurality of first images, each of which includes a group of traffic lights.
[0079] In this embodiment, the second image determined based on the second detection frame may include, in addition to a group of traffic lights, other groups of traffic lights or light bulbs of individual traffic lights, and may also include other irrelevant information. In order to improve the processing effect of the preset network model on the first image, this embodiment performs mask processing on the second image to obtain a first image that only includes a group of traffic lights.
[0080] Optionally, the mask processing process can be implemented by the following pseudo code:
[0081] mask_weight=np.zeros((1,input_wh[0]));
[0082] mask_height=np.zeros((1,input_wh[1]));
[0083] mask_weight[:,x1-x1_1:x1_1–x2]=1;
[0084] mask_height[:,y1-y1_1:y1_1–y2]=1;
[0085] mask_height=mask_height.transpose(1,0);
[0086] img=torch.mul(torch.mul(img,mask_weight),mask_height).
[0087] The above pseudo code first sets the pixel values of all positions in the second image to zero, and then sets the portion determined to be the traffic lights (based on the corresponding coordinates obtained by subtracting the scaled coordinates from the original coordinates) to the original image pixels to implement mask processing, so that except for a group of traffic lights in the first image, the rest of the portion is covered by the mask, for example, Figure 3 As shown, it is the second image without mask processing. Figure 4 is the first image after mask processing. Through mask processing, the first image only includes one group of traffic lights, thereby improving the recognition effect of traffic lights.
[0088] Figure 5 FIG. 1 is a flow chart of a method for identifying a traffic light provided by another exemplary embodiment of the present disclosure. Figure 5 As shown, the method provided in this embodiment includes the following steps:
[0089] Step 502: input the first image into a preset network model.
[0090] This step can refer to Figure 1 Step 102 in the illustrated embodiment is understood and will not be described in detail here.
[0091] Step 504: extract features of the first image based on the first subnetwork of the preset network model to obtain first image features.
[0092] This step can refer to Figure 1 Step 104 in the illustrated embodiment is understood and will not be described in detail here.
[0093] Step 506: Establish an association relationship between multiple first images corresponding to the same traffic light.
[0094] Optionally, since the captured target image may include multiple groups of traffic lights, in order to facilitate establishing association relationships for multiple first images, an identifier can be assigned to each group of traffic lights, and different traffic lights can be distinguished based on the identifier; in some optional examples, an association relationship is established between the same traffic lights in multiple frames of target images based on the identifier, ensuring that the fused features in the present embodiment correspond to the same group of traffic lights, thereby avoiding the problem of inaccurate predictions caused by non-corresponding traffic lights.
[0095] Step 508: Determine the previous frame image of the first image based on the association relationship.
[0096] In this embodiment, based on the association relationship of the same traffic light, the image of the traffic light is identified from a plurality of first images corresponding to the previous frame of the target image as the previous frame of the first image.
[0097] Step 510: Fusing the first image feature with the second image feature of the previous frame image to obtain a third image feature.
[0098] In this embodiment, the fusion processing of the first image feature and the second image feature can adopt the fusion method in the prior art, and it is only necessary that the third image feature obtained has the same size as the first image feature. Optionally, the first image feature and the second image feature are added bit by bit.
[0099] Step 512: Process the third image feature through the second sub-network in the preset network model to obtain a traffic light recognition result.
[0100] This step can refer to Figure 1 Step 108 in the illustrated embodiment is understood and will not be described in detail here.
[0101] In this embodiment, feature fusion is performed in an iterative (recurrent) manner, and the world always needs to maintain the fused third image feature of the output of the previous frame (for the first frame, the third image feature is a tensor initialized to all 0s) as the second image feature corresponding to the next frame. The advantage of this embodiment is that the saved third image feature incorporates the information of all previous frame images, and the world only needs to maintain the fused features of the previous frame, which greatly reduces the load on the software side. For the first image feature extracted from the current frame feature, a gated recurrent unit (GRU) is used to fuse the first image feature with the second image feature fused from the previous frame, and then input into the second subnetwork, outputting the traffic light recognition result while updating the third fused feature in the cache, and performing the process recursively in sequence.
[0102] In some optional embodiments, step 506 may include:
[0103] An identification label is assigned to the first image.
[0104] Optionally, when a target image includes multiple groups of traffic lights, in order to distinguish different groups of traffic lights, a unique identification mark (track id) may be allocated to the first image corresponding to each group of traffic lights.
[0105] Based on the identification mark, an association relationship is established between multiple frames of first images corresponding to multiple moments.
[0106] In this embodiment, the Hungarian algorithm can be used to perform data association on the first images corresponding to the same group of traffic lights in different frames based on the identification mark. After the association relationship is established, each identification mark corresponds to multiple first images corresponding to target images of different frames. Therefore, when it is necessary to obtain the corresponding first image of the previous frame for a first image of a certain frame, it can be obtained by searching based on the association relationship, thereby improving the efficiency of inter-frame query.
[0107] In some optional embodiments, step 108 may include:
[0108] The third image feature is processed through the multiple recognition branches of the second sub-network respectively, and multiple traffic light recognition results are output through the multiple output terminals.
[0109] The second sub-network includes multiple identification branches; the identification result includes at least two of the following: traffic light color, traffic light type, timing value and traffic light location information. Each identification branch corresponds to an output terminal, that is, the preset network model includes multiple output terminals, and different traffic light identification results are output through each output terminal.
[0110] Optionally, the recognition results of traffic lights may include but are not limited to: color recognition results, category recognition results, countdown recognition results, and recognition results of each traffic light detection frame and its corresponding attributes (color + category). Correspondingly, in an optional example, the second subnetwork includes four recognition branches, and each recognition branch corresponds to a recognition result. Among them, the color recognition results may include but are not limited to: 5 categories such as red, yellow, green, off, and other. The type recognition results may include but are not limited to: round lights, straight lights, left turn lights, right turn lights, U-turn lights, non-motor vehicle lights, countdown lights, left turn lights, straight left turn lights, straight right turn lights, no entry lights, allowed entry lights, and other lights. The countdown recognition results include but are not limited to: 0-999 digital recognition results, etc.
[0111] In this embodiment, the four recognition branches of the second subnetwork simultaneously output four corresponding recognition results, but the effective recognition results are determined by the number of bulbs included in the traffic light group corresponding to the first image. Optionally, when the number of bulbs included in the traffic light group is 1, the color recognition result, the category recognition result and the countdown recognition result are valid. When the number of bulbs included in the traffic light group is not 1, the countdown recognition result and each signal light detection frame and its corresponding attribute recognition result are valid. Optionally, each signal light detection frame recognition result includes the coordinate information corresponding to each bulb, which is used to distinguish different bulbs; and the attribute recognition result corresponding to each signal light detection frame includes the color recognition result and category recognition result of each bulb respectively; wherein, the color recognition result can also include but is not limited to: red, yellow, green, off, other and other 5 categories. The type recognition result can also include but is not limited to: round light, straight light, left turn light, right turn light, U-turn light, non-motor vehicle light, countdown light, left turn U-turn light, straight left turn light, straight right turn light, no entry light, allowed entry light and other lights.
[0112] Figure 6 FIG. 1 is a flow chart of a method for training a preset network model provided by an exemplary embodiment of the present disclosure. Figure 6 As shown, the training method provided in this embodiment includes the following steps:
[0113] Step 602: Input the first sample image and the second sample image into a preset network model respectively, and output a plurality of first recognition data and a plurality of second recognition data respectively through a plurality of recognition branches of the preset network model.
[0114] The first sample image and the second sample image are adjacent frame images continuously collected based on traffic lights; a plurality of first recognition data correspond to the first sample image, and a plurality of second recognition data correspond to the second sample image. In this embodiment, it is only necessary to mark whether the recognition results of the first sample image and the second sample image in each recognition branch are consistent.
[0115] Optionally, in the present embodiment, the process of obtaining multiple first recognition data based on the preset network model can be obtained by referring to the process provided by any of the above-mentioned traffic light recognition methods, wherein, for example, when the first sample image is the previous frame image of the second sample image, the second sample image is input into the preset network model, and the features of the second sample image are extracted based on the first subnetwork of the preset network model to obtain the second sample image features; the second sample image features are fused with the first sample image features to obtain the third sample image features; the third sample image features are processed by the second subnetwork in the preset network model to obtain multiple second recognition data; the first sample image features are the fused features corresponding to the first sample image, which can be obtained by fusing the image features extracted by the first subnetwork and the fused features corresponding to the previous frame image of the first sample image.
[0116] In this embodiment, the continuously collected adjacent frame images are respectively input into the preset network model, so that the preset network model can identify the traffic light images of adjacent frames. Optionally, in this embodiment, the first sample image and the second sample image only include one group of traffic lights, and the first sample image and the second sample image correspond to the same group of traffic lights. Since they correspond to the same group of traffic lights, the recognition results of the traffic lights have a temporal correlation; therefore, when the annotated true value labels corresponding to an identification branch are consistent, the first recognition data and the second recognition data are consistent in time sequence; in addition, when the annotated true value labels corresponding to an identification branch are inconsistent, there is a property change of the traffic light. For example, in the color recognition branch, the true value label of the previous frame is a green light, and the true value label of the current frame is a red light. Therefore, there is also a difference comparison in the outputs of adjacent frames.
[0117] Step 604, determining network loss based on the plurality of first identification data, the plurality of second identification data and the plurality of first reference values.
[0118] In this embodiment, each first reference value corresponds to an identification branch. The first reference value is used to indicate whether the labels of the previous and next frames are consistent. The first reference value can be determined based on the marked true value label. Optionally, when the true value labels of the previous and next frames are consistent, the first reference value is 1. When they are inconsistent, the first reference value is 0.
[0119] In this embodiment, since the preset network model has multiple identification branches, optionally, a branch network loss is determined based on a set of first identification data and second identification data corresponding to each identification branch, and the network loss is obtained by combining the branch network losses of all identification branches; for example, when the preset network model includes four identification branches, the four branch network losses are added, and the result of the addition is used as the network loss.
[0120] Step 606: training the preset network model based on the network loss.
[0121] In this embodiment, under the monocular camera forward viewing angle, based on the time-series traffic light feature fusion method, time-series consistency supervision and contrast supervision are introduced to improve the stability and accuracy of traffic light recognition in the case of distance or partial occlusion, reduce the complexity of downstream decision-making and planning voting strategies, and improve the efficiency and success rate of intersection traffic. In addition, this embodiment determines the network loss by combining the losses of multiple recognition branches, which improves the training efficiency of the preset network model.
[0122] like Figure 7 As shown in the above Figure 6 Based on the illustrated embodiment, step 604 may include the following steps:
[0123] Step 6041, determining a first loss and a second loss based on a plurality of first identification data, a plurality of second identification data and a plurality of first reference values.
[0124] Optionally, the first sample image and the second sample image may include multiple groups of traffic lights respectively. With reference to the above embodiment, a unique identification mark is assigned to each group of traffic lights to distinguish different groups of traffic lights, and a group of traffic lights in the first sample image is associated with traffic lights in the second sample image based on the identification mark. At this time, a corresponding relationship can be established between the first identification data output by an identification branch of the first sample image and the second identification data output by the identification branch corresponding to the second sample image based on the identification mark, for example, one-to-one correspondence, one first identification data corresponds to one second identification data; for example, the first identification data and the second identification data output by the color identification branch with the same identification mark have a corresponding relationship. Optionally, the first loss is a temporal consistency loss, and the first sub-loss represents the distribution consistency between the first identification data and the corresponding second identification data; at this time, the first sub-loss can be determined based on each first identification data, the corresponding second identification data, and the corresponding first reference value. Each first identification data and the corresponding second identification data determine a first sub-loss. Based on multiple first identification data and multiple second identification data, multiple first sub-losses can be obtained. The multiple first sub-losses are accumulated and summed to obtain the first loss. Optionally, the multiple first sub-losses can also be weighted and summed to obtain the first loss.
[0125] In this embodiment, for traffic lights at adjacent moments, when the first reference value is 1, that is, the true value labels of the previous and next frames in a recognition branch are consistent (for example, in the color recognition branch, the true value labels of the traffic lights in the previous and next frames are consistent), the output of the recognition branch should be as consistent as possible in distribution for the same classification and regression tasks. Taking the color recognition branch as an example, if the previous and next frames are both green lights, then the output of the color recognition branch should be as consistent as possible in distribution, that is, when the colors of the traffic lights in two consecutive frames are the same, it is hoped that the recognition results output by the color recognition branch are the same in two consecutive frames (for example, the recognition results are all green lights); other branches are similar, and there is consistency. For example, the output of the position recognition branch should also be as consistent as possible in distribution, that is, when the positions of the traffic lights in two consecutive frames are the same, it is hoped that the recognition position information output by the position recognition branch is the same in two consecutive frames. Therefore, the first loss in this embodiment is a temporal consistency loss. Optionally, the first sub-loss can be determined based on the KL divergence. Although not every identification branch is consistent (for example, the identification results of the timing identification branch at different times are different), the embodiment of the present disclosure calculates the first sub-loss for each identification branch.
[0126] Optionally, the second loss is a contrast loss, and the second sub-loss indicates the distribution difference between the first identification data and the corresponding second identification data; in this case, the second sub-loss can be determined based on each first identification data and the corresponding second identification data, and the corresponding first reference value. Similar to the first sub-loss, a second sub-loss is determined for each first identification data and the corresponding second identification data, and multiple second sub-losses can be obtained based on multiple first identification data and multiple second identification data. The multiple second sub-losses are accumulated and summed to obtain the second loss. Optionally, the multiple second sub-losses can also be weighted and summed to obtain the second loss.
[0127] In this embodiment, for traffic lights at adjacent moments, when the first reference value is 0, that is, the true value labels of the previous and next frames in a recognition branch are inconsistent (for example, in the color recognition branch, the true value labels of the traffic lights of the previous and next frames are inconsistent), in the same classification and regression task, if the previous frame is a green light and the current frame is a red light, then the output of the color recognition branch should have differences in distribution, that is, when the colors of the traffic lights in two consecutive frames are different, it is hoped that the recognition results output by the color recognition branch are different in the two consecutive frames (for example, the two recognition results are green and red, etc.); other branches are similar, and there are differences. For example, the output of the timing recognition branch should also be as different as possible in distribution, that is, when the timing values of the traffic lights in two consecutive frames are different, it is hoped that the recognition timing value information output by the timing recognition branch is different in the two consecutive frames. Therefore, the second loss of this embodiment is a contrast loss to ensure that there is a degree of distinction between the features corresponding to different frame images. Although not every recognition branch has differences (for example, the recognition results of the type recognition branch at different times are the same), the disclosed embodiment calculates the second sub-loss for each recognition branch.
[0128] A first loss is determined based on the plurality of first sub-losses, and a second loss is determined based on the plurality of second sub-losses.
[0129] Optionally, multiple first sub-losses are accumulated and summed to obtain a first loss; multiple second sub-losses are accumulated and summed to obtain a second loss. In some optional examples, multiple first sub-losses may be weighted and summed, for example, a higher weight value (for example, a first weight value, the specific value is determined by the application scenario) is assigned to the first sub-loss with consistency in the corresponding recognition branch (for example, a color recognition branch, etc.), and a lower weight value (for example, a second weight value, the specific value is determined by the application scenario, and the first weight value is greater than the second weight value) is assigned to the first sub-loss with no consistency in the corresponding recognition branch (for example, a timing recognition branch, etc.). In some optional examples, multiple second sub-losses may be weighted and summed, for example, a higher weight value (for example, a first weight value, the specific value is determined by the application scenario) is assigned to the second sub-loss with differences in the corresponding recognition branch (for example, a color recognition branch, a timing recognition branch, etc.), and a lower weight value (for example, a second weight value, the specific value is determined by the application scenario, and the first weight value is greater than the second weight value) is assigned to the second sub-loss with no differences in the corresponding recognition branch (for example, a type recognition branch, etc.).
[0130] Step 6042, determine the network loss based on the first loss and the second loss.
[0131] Optionally, the first loss and the second loss are accumulated and summed to obtain the network loss; the first loss and the second loss can also be weighted and summed to obtain the network loss; the weight value of the first loss and the weight value of the second loss can be set according to the actual application scenario, or the weight value of the first loss and the weight value of the second loss can be determined according to the empirical value. In this embodiment, by combining the temporal consistency loss and contrast loss of all recognition branches, the consistency and jump of the previous and next frames in the traffic light are learned, and the recognition accuracy of the preset network model obtained by training for the traffic light is improved.
[0132] Optionally, determining the first sub-loss may include:
[0133] A KL divergence calculation is performed based on the first identification data and the second identification data to determine a first divergence.
[0134] Based on the first divergence and the first reference value, a first sub-loss is determined.
[0135] In this embodiment, the first sub-loss is used to measure the consistency of the distribution, which can be implemented by using KL divergence (Kullback-Leibler divergence). Optionally, a first sub-loss can be determined based on the following formula (1):
[0136]
[0137] Wherein, when the first sub-loss corresponds to the color branch, loss1 represents the first sub-loss; y_cur represents the output of the color branch of the current frame; y_pre represents the output of the color branch of the previous frame; and k represents the first reference value. In this embodiment, the above process is performed on each recognition branch respectively to obtain multiple first sub-losses.
[0138] Optionally, the determination of the second sub-loss may include:
[0139] The Euclidean distance between the first recognition data and the corresponding second recognition data is determined to obtain a first distance.
[0140] In this embodiment, the second reference value is used to indicate whether the labels of the previous and next frames are consistent. In this embodiment, the second reference value can be determined based on the marked true value label, and can also be determined based on the first reference value. The second reference value can be obtained by negating the first reference value. Optionally, when the labels of the previous and next frames are consistent, the second reference value is 0, and when they are inconsistent, the second reference value is 1. It can be seen that the second reference value and the first reference value are in a negated relationship. Therefore, in the calculation process, the second reference value can be determined based on the negation of the first reference value. For example, the second reference value is represented as ~k, ~ represents negation, and k represents the first reference value.
[0141] A second sub-loss is determined based on the first distance, the first identification data, the second identification data, and the second reference value.
[0142] The second sub-loss in this embodiment is used to ensure that there is distinction between features corresponding to different label images. Optionally, the second sub-loss can be determined based on the following formula (2):
[0143] loss2={y_pre·dist 2 +[(1-y_pre)·clamp(1-dist, min=0.0)] 2}·(~k)
[0144] Formula (2)
[0145] Wherein, when the second sub-loss corresponds to the color branch, y_cur represents the output of the color branch of the current frame; y_pre represents the output of the color branch of the previous frame; ~k represents the second reference value; dist represents the first distance, specifically representing the distance between y_cur and y_pre (pixel-level features) (e.g., Euclidean distance, etc.). Optionally, dist can be determined based on the following formula (3).
[0146] dist=pairwise_distance(y_pre,u_cur) formula (3)
[0147] Among them, pairwise_distance means calculating the Euclidean distance between the two.
[0148] The training method of the preset network model provided in any of the above embodiments of the present application can be applied to the training of the preset network model in the traffic light recognition method provided in any of the above embodiments. Figure 6 The training method of the preset network model provided is Figure 1 In the embodiment shown, the preset network model in step 102 is trained; for example, using the above Figure 7 The training method of the preset network model provided is Figure 1 In the illustrated embodiment, the preset network model in step 102 is trained, etc.
[0149] Any traffic light recognition method provided in the embodiments of the present disclosure may be executed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server, etc. Alternatively, any traffic light recognition method provided in the embodiments of the present disclosure may be executed by a processor, such as the processor executing any traffic light recognition method mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.
[0150] Exemplary Devices
[0151] Figure 8 FIG. 1 is a schematic diagram of a traffic light recognition device provided by an exemplary embodiment of the present disclosure. Figure 8 As shown, the device provided in this embodiment includes:
[0152] The image preprocessing module 81 is used to input the first image into a preset network model.
[0153] The feature extraction module 82 is used to extract features of the first image based on the first sub-network of the preset network model to obtain first image features.
[0154] The feature fusion module 83 is used to fuse the first image feature with the second image feature to obtain a third image feature; the second image feature is an image feature obtained by extracting features of the previous frame of the first image through the first sub-network.
[0155] The traffic light recognition module 84 is used to process the third image feature through the second sub-network in the preset network model to obtain a traffic light recognition result.
[0156] The above-mentioned embodiment of the present disclosure provides a traffic light recognition device, which inputs a first image into a preset network model; extracts features of the first image based on the first sub-network of the preset network model to obtain first image features; fuses the first image features with the second image features to obtain third image features; the second image features are image features obtained by extracting features of the previous frame of the first image through the first sub-network; and processes the third image features through the second sub-network in the preset network model to obtain a traffic light recognition result. In this embodiment, a preset network model is applied to predict images to determine traffic light recognition results, thereby improving recognition efficiency; and, in this embodiment, by fusing the current frame image features with the previous frame image features before result recognition, the problem of incomplete information in the current frame image due to occlusion and other reasons is solved through feature fusion, thereby improving the stability and accuracy of traffic light recognition.
[0157] In some optional embodiments, the device provided by this embodiment further includes:
[0158] A network training module is used to input a first sample image and a second sample image into a preset network model respectively, and output a plurality of first recognition data and a plurality of second recognition data respectively through a plurality of recognition branches of the preset network model; wherein the first sample image and the second sample image are adjacent frame images continuously collected based on traffic lights, the plurality of first recognition data correspond to the first sample image, and the plurality of second recognition data correspond to the second sample image; based on the plurality of first recognition data, the plurality of second recognition data and the plurality of first reference values, the network loss is determined; and the preset network model is trained based on the network loss.
[0159] In some optional embodiments, the device provided by this embodiment further includes:
[0160] The first detection module is used to detect the collected multiple frames of target images respectively to obtain multiple first detection frames; each frame of the target image corresponds to at least one first detection frame.
[0161] The second detection module is used to perform scaling processing on each first detection frame in the multiple first detection frames according to the network parameters of the preset network model to obtain a second detection frame.
[0162] The first image determination module is used to determine multiple first images based on multiple second detection frames; each second detection frame corresponds to a first image.
[0163] Optionally, the first image determination module is specifically used to crop the second image from the corresponding target image based on each second detection frame; perform mask processing on each second image to obtain multiple first images, each first image including a group of traffic lights.
[0164] In some optional embodiments, the device provided by this embodiment further includes:
[0165] The association establishing module is used to establish an association relationship between multiple first images corresponding to the same traffic light.
[0166] The feature fusion module 83 is specifically used to determine the previous frame image of the first image based on the association relationship; and fuse the first image feature with the second image feature of the previous frame image to obtain a third image feature.
[0167] Optionally, the association establishing module is specifically used to assign an identification mark to the first image; based on the identification mark, establish an association relationship between multiple frames of first images corresponding to multiple moments.
[0168] In some optional embodiments, the second sub-network includes a plurality of identification branches; the identification result includes at least two of the following: the color of the signal light, the type of the signal light, the timing value, and the position information of the signal light;
[0169] The traffic light recognition module 84 is specifically used to process the third image feature through multiple recognition branches of the second sub-network respectively, and output multiple traffic light recognition results through multiple output terminals; each recognition branch corresponds to an output terminal.
[0170] The beneficial technical effects corresponding to the exemplary embodiment of the present device can be found in the corresponding beneficial technical effects of the above exemplary method section, which will not be repeated here.
[0171] Fig. 9 FIG. 1 is a schematic diagram of a structure of a training device for a preset network model provided by an exemplary embodiment of the present disclosure. Fig. 9 As shown, the device provided in this embodiment includes:
[0172] The model processing module 91 is used to input the first sample image and the second sample image into a preset network model respectively, and output a plurality of first recognition data and a plurality of second recognition data respectively through a plurality of recognition branches of the preset network model.
[0173] The first sample image and the second sample image are adjacent frame images collected continuously; the plurality of first recognition data correspond to the first sample image, and the plurality of second recognition data correspond to the second sample image.
[0174] The loss determination module 92 is used to determine the network loss based on a plurality of first identification data, a plurality of second identification data and a plurality of first reference values.
[0175] The network training module 93 is used to train the preset network model based on the network loss.
[0176] In this embodiment, based on the fusion of time-series traffic light features, time-series consistency supervision and comparison supervision are introduced in a single-view perspective to improve the stability and accuracy of traffic light recognition in the case of distance or partial occlusion, reduce the complexity of downstream decision-making and planning voting strategies, and improve the efficiency and success rate of intersection traffic. In addition, this embodiment determines the network loss by combining the losses of multiple recognition branches, which improves the training efficiency of the preset network model.
[0177] In some optional embodiments, the loss determination module 92 includes:
[0178] The loss calculation unit is used to determine the first loss and the second loss based on the plurality of first identification data, the plurality of second identification data and the plurality of first reference values.
[0179] The network loss unit is used to determine the network loss based on the first loss and the second loss.
[0180] Optionally, the first identification data and the second identification data output by the same identification branch have a corresponding relationship;
[0181] The loss calculation unit is specifically used to determine a first sub-loss based on each first identification data and the corresponding second identification data, and the corresponding first reference value; the first sub-loss represents the distribution consistency between the first identification data and the corresponding second identification data; the second sub-loss represents the distribution difference between the first identification data and the second identification data based on each first identification data and the corresponding second identification data, and the corresponding first reference value; the first loss is determined based on multiple first sub-losses, and the second loss is determined based on multiple second sub-losses.
[0182] Optionally, when the loss calculation unit determines the first sub-loss, it is used to perform KL divergence calculation based on the first recognition data and the second recognition data to determine the first divergence; and determine the first sub-loss based on the first divergence and the first reference value.
[0183] Optionally, when the loss calculation unit determines the second sub-loss, it is used to determine the Euclidean distance between the first identification data and the corresponding second identification data to obtain the first distance; based on the first distance, the first identification data, the second identification data and the second reference value, the second sub-loss is determined; the second reference value is determined based on the first reference value.
[0184] The beneficial technical effects corresponding to the exemplary embodiment of the present device can be found in the corresponding beneficial technical effects of the above exemplary method section, which will not be repeated here.
[0185] Exemplary Electronic Devices
[0186] Fig.10 A structural diagram of an electronic device provided in an embodiment of the present disclosure includes at least one processor 11 and a memory 12.
[0187] The processor 11 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.
[0188] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute one or more computer program instructions to implement the traffic signal light recognition method and / or other desired functions of the various embodiments of the present disclosure described above.
[0189] In one example, the electronic device 10 may further include: an input device 13 and an output device 14, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).
[0190] The input device 13 may also include, for example, a keyboard, a mouse, etc.
[0191] The output device 14 can output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.
[0192] Of course, to simplify, Fig.10 Only some of the components related to the present disclosure in the electronic device 10 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 10 may also include any other appropriate components.
[0193] Exemplary computer program products and computer-readable storage media
[0194] In addition to the above-mentioned methods and devices, embodiments of the present disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the traffic light recognition method of various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section.
[0195] The computer program product may be written in any combination of one or more programming languages to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0196] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the traffic light recognition method of various embodiments of the present disclosure described in the above "Exemplary Method" section.
[0197] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium is, for example, but not limited to, a system, device or device including electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0198] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and cannot be considered as necessary for each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, rather than limitation, and the above details do not limit the present disclosure to being implemented by adopting the above specific details.
[0199] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present disclosure claims and their equivalents, the present disclosure is also intended to include these modifications and variations.
Claims
1. A method for identifying a traffic light, comprising: Inputting the first image into a preset network model; Performing feature extraction on the first image based on the first subnetwork of the preset network model to obtain first image features; The first image feature is fused with the second image feature to obtain a third image feature; the second image feature is an image feature obtained by extracting and fusing features of a previous frame of the first image through the first sub-network; The third image feature is processed by the second sub-network in the preset network model to obtain a traffic light recognition result.
2. The method according to claim 1, further comprising: Inputting a first sample image and a second sample image into the preset network model respectively, and outputting a plurality of first recognition data and a plurality of second recognition data respectively through a plurality of recognition branches of the preset network model; wherein the first sample image and the second sample image are adjacent frame images continuously collected based on traffic lights, the plurality of first recognition data correspond to the first sample image, and the plurality of second recognition data correspond to the second sample image; determining a network loss based on the plurality of first identification data, the plurality of second identification data, and a plurality of first reference values; The preset network model is trained based on the network loss.
3. The method according to claim 1, further comprising: Detecting the acquired multiple frames of target images respectively to obtain multiple first detection frames; Each frame of the target image corresponds to at least one of the first detection frames; According to the network parameters of the preset network model, performing scaling processing on each of the first detection frames in the plurality of first detection frames to obtain a second detection frame; Determine a plurality of the first images based on a plurality of the second detection frames; Each of the second detection frames corresponds to one of the first images.
4. The method according to claim 3, wherein: The determining a plurality of the first images based on the plurality of the second detection frames comprises: Based on each of the second detection frames, cropping from the corresponding target image to obtain a second image; Mask processing is performed on each of the second images to obtain a plurality of the first images, each of which includes a group of traffic lights.
5. The method according to any one of claims 1 to 4, before fusing the first image feature with the second image feature to obtain the third image feature, further comprising: Establishing an association relationship between a plurality of the first images corresponding to the same traffic light; The fusing the first image feature with the second image feature to obtain a third image feature includes: Determine a previous frame image of the first image based on the association relationship; The first image feature is fused with the second image feature of the previous frame image to obtain a third image feature.
6. The method according to claim 5, wherein the step of establishing an association relationship between the plurality of first images corresponding to the same traffic light comprises: assigning an identification mark to the first image; Based on the identification mark, an association relationship is established between multiple frames of the first image corresponding to multiple moments.
7. The method according to any one of claims 1 to 4, wherein: The second sub-network includes a plurality of identification branches; the identification result includes at least two of the following: signal light color, signal light type, timing value and signal light location information; The processing of the third image feature by the second sub-network in the preset network model to obtain a traffic light recognition result includes: Processing the third image feature through the multiple recognition branches of the second sub-network respectively, and outputting the multiple traffic light recognition results through multiple output terminals; Each of the identification branches corresponds to an output terminal.
8. A method for training a preset network model, comprising: Inputting a first sample image and a second sample image into a preset network model respectively, and outputting a plurality of first recognition data and a plurality of second recognition data respectively through a plurality of recognition branches of the preset network model; wherein the first sample image and the second sample image are adjacent frame images continuously collected based on traffic lights; the plurality of first recognition data correspond to the first sample image, and the plurality of second recognition data correspond to the second sample image; determining a network loss based on the plurality of first identification data, the plurality of second identification data, and a plurality of first reference values; The preset network model is trained based on the network loss.
9. The method according to claim 8, wherein: The determining of the network loss based on the plurality of first identification data, the plurality of second identification data and the plurality of first reference values comprises: determining a first loss and a second loss based on the plurality of first identification data, the plurality of second identification data, and the plurality of first reference values; The network loss is determined based on the first loss and the second loss.
10. The method according to claim 9, wherein: The first identification data and the second identification data outputted by the same identification branch have a corresponding relationship; The determining the first loss and the second loss based on the plurality of first identification data, the plurality of second identification data and the plurality of first reference values comprises: Determine a first sub-loss based on each of the first identification data and the corresponding second identification data, and the corresponding first reference value; the first sub-loss represents the distribution consistency between the first identification data and the corresponding second identification data; Determine a second sub-loss based on each of the first identification data and the corresponding second identification data, and the corresponding first reference value; the second sub-loss represents the distribution difference between the first identification data and the second identification data; The first loss is determined based on a plurality of the first sub-losses, and the second loss is determined based on a plurality of the second sub-losses.
11. The method according to claim 10, wherein: The determining of the first sub-loss based on each of the first identification data and the corresponding second identification data, and the corresponding first reference value, comprises: Perform KL divergence calculation based on the first recognition data and the second recognition data to determine a first divergence; The first sub-loss is determined based on the first divergence and the first reference value.
12. The method according to claim 10 or 11, wherein: The determining the second sub-loss based on each of the first identification data and the corresponding second identification data, and the corresponding first reference value, comprises: Determine the Euclidean distance between the first identification data and the corresponding second identification data to obtain a first distance; The second sub-loss is determined based on the first distance, the first identification data, the second identification data, and a second reference value; the second reference value is determined based on the first reference value.
13. A traffic light recognition device, comprising: An image preprocessing module, used for inputting the first image into a preset network model; A feature extraction module, used to extract features of the first image based on the first sub-network of the preset network model to obtain first image features; A feature fusion module, used to fuse the first image feature with the second image feature to obtain a third image feature; the second image feature is an image feature obtained by extracting features of a previous frame of the first image through the first sub-network; The traffic light recognition module is used to process the third image feature through the second sub-network in the preset network model to obtain a traffic light recognition result.
14. A training device for a preset network model, comprising: A model processing module, used to input a first sample image and a second sample image into a preset network model respectively, and output a plurality of first recognition data and a plurality of second recognition data respectively through a plurality of recognition branches of the preset network model; wherein the first sample image and the second sample image are adjacent frame images collected continuously; the plurality of first recognition data correspond to the first sample image, and the plurality of second recognition data correspond to the second sample image; a loss determination module, configured to determine a network loss based on the plurality of first identification data, the plurality of second identification data and a plurality of first reference values; A network training module is used to train the preset network model based on the network loss.
15. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the traffic light recognition method described in any one of claims 1 to 7 or the preset network model training method described in any one of claims 8 to 12.
16. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the traffic light recognition method described in any one of claims 1-7 or the preset network model training method described in any one of claims 8-12.
