Traffic light identification model training method, traffic light identification method and related device

Abstract

The embodiment of the invention provides a traffic light recognition model training method, a traffic light recognition method and a related device, and is applied to the technical field of computer target recognition. According to the embodiment of the invention, the difficult traffic light image can be obtained, the shape label corresponding to the difficult traffic light image is obtained based on the difficult traffic light image and N shape classifiers, and the color label corresponding to the difficult traffic light image is obtained based on the difficult traffic light image and N color classifiers. And training to obtain a traffic light recognition model according to the difficult traffic light image, the shape label corresponding to the difficult traffic light image and the color label corresponding to the difficult traffic light image. Thus, the identification capability of the traffic light identification model obtained through training can be improved, the identification accuracy of the traffic light identification model is improved, and if the traffic light identification model is used by an intelligent driving vehicle, the driving safety of the vehicle can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer target recognition technology, and in particular to a traffic light recognition model training method, a traffic light recognition method and related devices. Background Art

[0002] With the continuous development of science and technology, intelligent driving equipment such as autonomous vehicles and assisted driving vehicles have attracted more and more attention. The perception module plays an important role in intelligent driving. The perception module can perceive obstacles, vehicles, pedestrians, traffic signs and other information around the equipment. This perception information is used by the decision module to determine a safe driving path, so that the intelligent driving equipment can drive along the path.

[0003] In the process of intelligent driving, the perception of traffic signals and the like is an important part of the perception process. Traffic lights are devices that control the passage of vehicles and play an important role in ensuring traffic safety. Intelligent driving vehicles usually use traffic light recognition models to identify the location of one or more traffic lights in the driving scene and identify their states (for example, red, green, and yellow). In specific implementations, the traffic light recognition model requires a large amount of labeled traffic light data (i.e., traffic light images) for training. However, for different types of traffic lights and traffic lights under various lighting and weather conditions, there may be some difficult traffic light images that are difficult to label with accurate information. For these difficult traffic light images, the performance of the traffic light recognition model may be poor, and the recognition results may not be accurate enough. When intelligent driving vehicles cannot correctly identify traffic lights, illegal behavior may occur, and even traffic accidents may occur, posing a threat to personal and property safety. Summary of the invention

[0004] The embodiments of the present application provide a traffic light recognition model training method, a traffic light recognition method and related devices, which can improve the recognition ability of the trained traffic light recognition model and improve the recognition accuracy of the traffic light recognition model. If an intelligent driving vehicle uses the traffic light recognition model, the driving safety of the vehicle can be improved.

[0005] In a first aspect, an embodiment of the present application provides a traffic light recognition model training method, the method comprising:

[0006] Get a hard example traffic light image;

[0007] Obtaining a shape label corresponding to the difficult traffic light image based on the difficult traffic light image and N shape classifiers, and obtaining a color label corresponding to the difficult traffic light image based on the difficult traffic image and N color classifiers;

[0008] A traffic light recognition model is trained based on the difficult traffic light image, the shape label corresponding to the difficult traffic light image, and the color label corresponding to the difficult traffic light image.

[0009] The present application can obtain shape labels corresponding to difficult traffic light images based on difficult traffic light images and N shape classifiers, and obtain color labels corresponding to difficult traffic light images based on difficult traffic images and N color classifiers, and then use difficult traffic light images, shape labels corresponding to difficult traffic light images, and color labels corresponding to difficult traffic light images as training samples of a traffic light recognition model to train a traffic light recognition model, which can improve the recognition ability of the traffic light recognition model. For traffic light images acquired by image sensors with poor imaging, the traffic light recognition model can be used to effectively solve the problem that data caused by small targets or blurred targets cannot be effectively labeled, and can improve the recognition accuracy of the traffic light recognition model and enhance the performance of the traffic light recognition model. If an intelligent driving vehicle uses the traffic light recognition model, the driving safety of the autonomous driving vehicle can be improved.

[0010] In a possible implementation of the first aspect, before obtaining a shape label corresponding to the difficult traffic light image based on the difficult traffic light image and N shape classifiers, and obtaining a color label corresponding to the difficult traffic light image based on the difficult traffic image and N color classifiers, the method further includes:

[0011] Obtain N sample sets, each of the N sample sets includes a plurality of traffic light samples, each traffic light sample includes a traffic light image, a shape label corresponding to the traffic light image, and a color label corresponding to the traffic light image, where N is an integer and is greater than 0;

[0012] The N sample sets are respectively input into N shape classifier models and N color classifier models, and N shape classifiers and N color classifiers are obtained through training. One shape classifier among the N shape classifiers and one color classifier among the N color classifiers correspond to one sample set among the N sample sets.

[0013] In this way, by training with multiple sample sets, the recognition capabilities of the trained shape classifier and color classifier can be improved.

[0014] In another possible implementation of the first aspect, inputting the N sample sets into N shape classifier models and N color classifier models respectively to train N shape classifiers and N color classifiers comprises:

[0015] Inputting the N sample sets into the N shape classifier models and the N color classifier models for training respectively;

[0016] The following steps are performed on any one of the N shape classifier models and any one of the N color classifier models:

[0017] Determine a traffic light shape category loss of any one of the shape classifier models and a traffic light color category loss of any one of the color classifier models;

[0018] When the traffic light shape category loss of any one of the shape classifier models is greater than the first threshold, training the any one of the shape classifier models again until the traffic light shape category loss of any one of the shape classifier models is less than or equal to the first threshold;

[0019] When the traffic light color category loss of any one of the color classifier models is greater than the second threshold, the any one of the color classifier models is trained again until the traffic light color category loss of any one of the color classifier models is less than or equal to the second threshold.

[0020] In this way, each time the shape classifier model and the color classifier model are trained, the shape classifier model and the color classifier model will be fine-tuned according to the labeled data. The smaller the loss of the traffic light shape category, the more accurate the classification result of the shape classifier model; the smaller the loss of the traffic light color category, the more accurate the classification result of the color classifier model. Through this process, the decision boundaries of the shape classifier model and the color classifier model are changed, the classification accuracy is improved, and the purpose of continuously optimizing the shape classifier model and the color classifier model is achieved.

[0021] In another possible implementation of the first aspect, obtaining a shape label corresponding to the difficult traffic light image based on the difficult traffic light image and N shape classifiers includes:

[0022] Input the difficult traffic light image into the N shape classifiers to obtain M shape recognition results, where M is an integer, and M is greater than 0 and M is less than or equal to N;

[0023] Based on the M shape recognition results, a shape label corresponding to the difficult traffic light image is obtained.

[0024] In another possible implementation of the first aspect, obtaining a color label corresponding to the difficult traffic light image based on the difficult traffic image and N color classifiers includes:

[0025] Input the difficult traffic light image into the N color classifiers to obtain P color recognition results, where P is an integer, and P is greater than 0 and P is less than or equal to N;

[0026] Based on the P color recognition results, a color label corresponding to the difficult traffic light image is obtained.

[0027] In another possible implementation of the first aspect, inputting the difficult traffic light image into the N shape classifiers to obtain M shape recognition results includes:

[0028] Inputting the difficult example traffic light image into the N shape classifiers to obtain N shape category confidences corresponding to the difficult example traffic light image, each shape classifier in the N shape classifiers corresponding to one shape category confidence among the N shape category confidences;

[0029] According to the N shape category confidences corresponding to the difficult traffic light image, M shape recognition results are obtained, and the shape category confidence corresponding to each of the M shape recognition results is greater than or equal to a third threshold.

[0030] In this way, the shape recognition result with a shape confidence greater than or equal to the third threshold is selected as the shape recognition result for determining the shape label corresponding to the difficult traffic light image, thereby reducing the impact of a large number of shape recognition results with low shape confidence on the shape label corresponding to the difficult traffic light image, and improving the accuracy of determining the shape label corresponding to the difficult traffic light image.

[0031] In another possible implementation of the first aspect, inputting the difficult traffic light image into the N color classifiers to obtain P color recognition results includes:

[0032] Inputting the difficult example traffic light image into the N color classifiers to obtain N color category confidences corresponding to the difficult example traffic light image, each color classifier in the N color classifiers corresponding to one color category confidence in the N color category confidences;

[0033] According to the N color category confidences corresponding to the difficult traffic light image, P color recognition results are obtained, and the color category confidence corresponding to each of the P color recognition results is greater than or equal to a fourth threshold.

[0034] In this way, the color recognition result with a color confidence greater than or equal to the fourth threshold is selected as the color recognition result for determining the color label corresponding to the difficult traffic light image, thereby reducing the impact of the large number of color recognition results with low color confidence on the color label corresponding to the difficult traffic light image, and improving the accuracy of determining the color label corresponding to the difficult traffic light image.

[0035] In another possible implementation of the first aspect, each of the M shape recognition results indicates a result value, and obtaining a shape label corresponding to the difficult traffic light image based on the M shape recognition results includes:

[0036] Based on the quantity distribution characteristics of each result value indicated by the M shape recognition results, voting is performed to obtain a final shape recognition result corresponding to the difficult traffic light image, wherein the result value indicated by the final shape recognition result is the same as the result value indicated by the majority of the shape recognition results in each result value indicated by the M shape recognition results;

[0037] The shape label corresponding to the difficult traffic light image is the shape category corresponding to the final shape recognition result.

[0038] In this way, the majority principle is adopted for voting, the one with the most identical shape recognition results is taken as the final shape recognition result, and the shape category corresponding to the final shape recognition result is output as the shape label corresponding to the difficult traffic light image, which can improve the stability of the system and the accuracy of shape recognition.

[0039] In another possible implementation of the first aspect, each of the P color recognition results indicates a result value, and obtaining a color label corresponding to the difficult traffic light image based on the P color recognition results includes:

[0040] Based on the quantity distribution characteristics of each result value indicated by the P color recognition results, voting is performed to obtain a final color recognition result corresponding to the difficult traffic light image, wherein the result value indicated by the final color recognition result is the same as the result value indicated by the majority of the color recognition results in each result value indicated by the P color recognition results;

[0041] The color label corresponding to the difficult traffic light image is the color category corresponding to the final color recognition result.

[0042] In this way, the majority principle is adopted for voting, the one with the most identical color recognition results is taken as the final color recognition result, and the color category corresponding to the final color recognition result is output as the color label corresponding to the difficult traffic light image, which can improve the stability of the system and the accuracy of color recognition.

[0043] In a second aspect, an embodiment of the present application provides a traffic light recognition method, the method comprising:

[0044] Obtain a traffic light image to be identified;

[0045] Inputting the traffic light image to be identified into a traffic light recognition model to obtain a color recognition result and a shape recognition result corresponding to the traffic light image to be identified;

[0046] The traffic light recognition model is trained using the traffic light recognition model training method described in any one of the first aspects above.

[0047] In a third aspect, an embodiment of the present application provides a processing device, wherein the processing device comprises an acquisition unit and a processing unit.

[0048] The acquisition unit is used to acquire a difficult traffic light image;

[0049] The processing unit is used for:

[0050] Obtaining a shape label corresponding to the difficult traffic light image based on the difficult traffic light image and the N shape classifiers, and obtaining a color label corresponding to the difficult traffic light image based on the difficult traffic image and the N color classifiers;

[0051] A traffic light recognition model is trained based on the shape label corresponding to the difficult traffic light image and the color label corresponding to the difficult traffic light image.

[0052] In a possible implementation of the third aspect, before obtaining a shape label corresponding to the difficult traffic light image based on the difficult traffic light image and N shape classifiers, and obtaining a color label corresponding to the difficult traffic light image based on the difficult traffic image and N color classifiers, the acquisition unit is further used to obtain N sample sets, each sample set in the N sample sets includes multiple traffic light samples, each traffic light sample includes a traffic light image, a shape label corresponding to the traffic light image, and a color label corresponding to the traffic light image, N is an integer and N is greater than 0. The processing unit is also used to input the N sample sets into N shape classifier models and N color classifier models, respectively, to obtain N shape classifiers and N color classifiers through training, and one shape classifier in the N shape classifiers and one color classifier in the N color classifiers correspond to one sample set in the N sample sets.

[0053] In yet another possible implementation manner of the third aspect, the processing unit is further configured to:

[0054] Inputting the N sample sets into the N shape classifier models and the N color classifier models for training respectively;

[0055] The following steps are performed on any one of the N shape classifier models and any one of the N color classifier models:

[0056] Determine a traffic light shape category loss of any one of the shape classifier models and a traffic light color category loss of any one of the color classifier models;

[0057] When the traffic light shape category loss of any one of the shape classifier models is greater than the first threshold, training the any one of the shape classifier models again until the traffic light shape category loss of any one of the shape classifier models is less than or equal to the first threshold;

[0058] When the traffic light color category loss of any one of the color classifier models is greater than the second threshold, the any one of the color classifier models is trained again until the traffic light color category loss of any one of the color classifier models is less than or equal to the second threshold.

[0059] In yet another possible implementation manner of the third aspect, the processing unit is further configured to:

[0060] Input the difficult traffic light image into the N shape classifiers to obtain M shape recognition results, where M is an integer, and M is greater than 0 and M is less than or equal to N;

[0061] Based on the M shape recognition results, a shape label corresponding to the difficult traffic light image is obtained.

[0062] In yet another possible implementation manner of the third aspect, the processing unit is further configured to:

[0063] Input the difficult traffic light image into the N color classifiers to obtain P color recognition results, where P is an integer, and P is greater than 0 and P is less than or equal to N;

[0064] Based on the P color recognition results, a color label corresponding to the difficult traffic light image is obtained.

[0065] In yet another possible implementation manner of the third aspect, the processing unit is further configured to:

[0066] Inputting the difficult example traffic light image into the N shape classifiers to obtain N shape category confidences corresponding to the difficult example traffic light image, each shape classifier in the N shape classifiers corresponding to one shape category confidence among the N shape category confidences;

[0067] According to the N shape category confidences corresponding to the difficult traffic light image, M shape recognition results are obtained, and the shape category confidence corresponding to each of the M shape recognition results is greater than or equal to a third threshold.

[0068] In yet another possible implementation manner of the third aspect, the processing unit is further configured to:

[0069] Inputting the difficult example traffic light image into the N color classifiers to obtain N color category confidences corresponding to the difficult example traffic light image, each color classifier in the N color classifiers corresponding to one color category confidence in the N color category confidences;

[0070] According to the N color category confidences corresponding to the difficult traffic light image, P color recognition results are obtained, and the color category confidence corresponding to each of the P color recognition results is greater than or equal to a fourth threshold.

[0071] In yet another possible implementation manner of the third aspect, the processing unit is further configured to:

[0072] Based on the quantity distribution characteristics of each result value indicated by the M shape recognition results, voting is performed to obtain a final shape recognition result corresponding to the difficult traffic light image, wherein the result value indicated by the final shape recognition result is the same as the result value indicated by the majority of the shape recognition results in each result value indicated by the M shape recognition results;

[0073] The shape label corresponding to the difficult traffic light image is the shape category corresponding to the final shape recognition result.

[0074] In yet another possible implementation manner of the third aspect, the processing unit is further configured to:

[0075] Based on the quantity distribution characteristics of each result value indicated by the P color recognition results, voting is performed to obtain a final color recognition result corresponding to the difficult traffic light image, wherein the result value indicated by the final color recognition result is the same as the result value indicated by the majority of the color recognition results in each result value indicated by the P color recognition results;

[0076] The color label corresponding to the difficult traffic light image is the color category corresponding to the final color recognition result.

[0077] In a fourth aspect, an embodiment of the present application provides another processing device, the processing device comprising an acquisition unit and a processing unit,

[0078] The acquisition unit is used to acquire the traffic light image to be identified;

[0079] The processing unit is used for:

[0080] Inputting the traffic light image to be identified into a traffic light recognition model to obtain a color recognition result and a shape recognition result corresponding to the traffic light image to be identified;

[0081] The traffic light recognition model is trained using the traffic light recognition model training method described in any one of the first aspects above.

[0082] In a fifth aspect, an embodiment of the present application provides a computing device, comprising a processor and a memory; the processor executes instructions / programs stored in the memory so that the computing device implements the method described in any one of the first aspect or any one of the second aspect.

[0083] Optionally, the computing device further includes a communication interface, wherein the communication interface is used to receive and / or send data, and / or the communication interface is used to provide input and / or output for the processor.

[0084] It should be noted that the above embodiments are described by taking a processor (or general-purpose processor) that executes the method by calling a computer instruction / program as an example. In the specific implementation process, the processor may also be a dedicated processor, in which case the computer instruction / program has been pre-loaded in the processor. Optionally, the processor may also include both a dedicated processor and a general-purpose processor.

[0085] Optionally, the processor and the memory may also be integrated into one device, that is, the processor and the memory may also be integrated together.

[0086] In a sixth aspect, the present application provides a vehicle, comprising the aforementioned processing device or computing equipment, so that the vehicle implements the method described in any one of the aforementioned second aspects.

[0087] In the seventh aspect, an embodiment of the present application provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed by a processing device or a computing device, the processing device implements the method described in any one of the first aspect or any one of the second aspect.

[0088] In an eighth aspect, the present application provides a computer program product, which includes computer instructions. When the instructions are executed by a processing device, the processing device implements the method described in any one of the first aspect or any one of the second aspect.

[0089] Optionally, the computer program product may be a software installation package or an image file. When the aforementioned method is required, the computer program product may be obtained and executed on a computing device.

[0090] The beneficial effects of the technical solutions provided in the second to eighth aspects of the present application can refer to the beneficial effects of the technical solution of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] The following is a brief introduction to the drawings required for describing the embodiments.

[0092] Figure 1 It is a schematic diagram of the architecture of a traffic light recognition system provided in an embodiment of the present application;

[0093] Figure 2 It is a flow chart of a traffic light recognition model training method provided in an embodiment of the present application;

[0094] Figure 3 It is a flow chart of a traffic light recognition method provided in an embodiment of the present application;

[0095] Figure 4 is a structural schematic diagram of a processing device provided in an embodiment of the present application;

[0096] Figure 5 is a structural schematic diagram of another processing device provided in an embodiment of the present application;

[0097] Figure 6 It is a structural diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0098] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0099] The following is an introduction to the system architecture of the embodiment of the present application. It should be noted that the system architecture and business scenarios described in this application are intended to more accurately illustrate the technical solution of this application, and do not constitute a limitation on the technical solution provided by this application. A person of ordinary skill in the art will know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solution provided by this application is also applicable to similar technical problems.

[0100] See also Figure 1 , Figure 1 Schematic diagram of the architecture of a traffic light recognition system provided by an embodiment of the present application. Figure 1 As shown, the traffic light recognition system includes a vehicle 10 and a traffic light 20. The vehicle 10 includes an image sensor 101 and a processing device 102.

[0101] The image sensor 101 may be a visible light image sensor or an infrared image sensor, etc., and may capture a traffic light image including the traffic light 20, and send the captured traffic light image to the processing device 102. Exemplarily, the image sensor 101 may be a visible light camera or an infrared camera, etc.

[0102] The processing device 102 is a module with data processing and control capabilities, and is used to process relevant data of the vehicle 10 and control the vehicle 10 to complete relevant functions. Exemplarily, the processing device 102 can obtain a traffic light image from the image sensor 101, input the traffic light image into the traffic light recognition model, and obtain a color recognition result and a corresponding shape recognition result corresponding to the traffic light image. Exemplarily, the processing device 102 obtains that the color recognition result corresponding to the traffic light image is red and the corresponding shape recognition result is straight, that is, the straight signal light of the traffic light at this time is red, and the processing device 102 can control the vehicle 10 to stop going straight.

[0103] As a possible implementation, the processing device 102 may be a physical device, for example, the processing device 102 may include one or more of the following modules: a central processing unit (CPU), a microprocessor (MPU), an application specific-integrated circuit (ASIC), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), a coprocessor (to assist the central processor in completing corresponding processing and applications), a microcontroller unit (MCU), and / or an electronic control unit (ECU). Of course, the above description is based on the example that the processing device 102 is an on-board device. In some solutions, the processing device 102 may be a physical device set outside the vehicle, such as a server, a cloud, or a host. As a possible implementation, the processing device 102 may be a software module, such as a virtual machine, software, program code, or a container.

[0104] In the above-mentioned manner, it is mentioned that the processing device 102 can be arranged outside the vehicle. It should be understood that when the processing device 102 is arranged outside the vehicle 10, the vehicle 10 and the processing device 102 can be connected in communication, for example, the two are directly connected through a wired communication method, such as a tangible medium such as a metal wire or an optical fiber, or the two are indirectly connected through a wireless communication method, such as ultra-wideband (UWB) technology, long term evolution (LTE) communication technology, fifth generation mobile communication technology (5th generation mobile networks or 5th generation wireless systems, 5th-Generation, referred to as 5G or 5G technology), global system for mobile communications (GSM), general packet radio service (GPRS), or universal mobile telecommunications system (UMTS).

[0105] Optionally, the foregoing is an exemplary vehicle provided for ease of understanding of the present solution and is not intended to limit the application scenarios of the present solution. The solution provided in the present application is also applicable to similar devices having traveling capabilities and data processing capabilities, such as logistics robots, etc.

[0106] In addition, the present application does not limit the purpose, power supply method, classification, model, etc. of the vehicle 10. For example, the vehicle 10 can be a passenger car, a bus, a truck, a rescue vehicle, a special vehicle (such as a mud tanker, a tanker, a water truck, a fire truck, etc.). For another example, the vehicle 10 can be a pure electric vehicle, a fuel vehicle, a hybrid vehicle, etc.

[0107] The traffic light 20 is a signal light for directing traffic operation, and is generally composed of a red light, a green light, and a yellow light. Among them, the red light indicates that passage is prohibited, the green light indicates that passage is permitted, and the yellow light indicates a warning. Optionally, the traffic light 20 may also include shapes, such as straight ahead, left turn, right turn, and U-turn. Exemplarily, if the shape of the traffic light 20 is straight ahead and the color is green, it means that the vehicle 10 can go straight ahead.

[0108] As a possible implementation, the embodiment of the present application can obtain a traffic light image including a traffic light 20 through an image sensor 101, and the processing device 102 inputs the traffic light image into a traffic light recognition model to obtain a color recognition result and a corresponding shape recognition result corresponding to the traffic light image, so that the processing device 102 controls the driving of the vehicle 10. Among them, the traffic light recognition model is trained based on N sample sets, shape labels corresponding to difficult traffic light images, and color labels corresponding to difficult traffic light images, N is an integer and N is greater than 0, each sample set in the N sample sets includes multiple traffic light samples, and each traffic light sample includes a traffic light image, a shape label corresponding to the traffic light image, and a color label corresponding to the traffic light image. In this way, the present application takes into account traffic lights under different weather conditions, different traffic conditions, and different lighting conditions. For these difficult traffic light images, the present application can improve the recognition accuracy of the traffic light recognition model and improve the performance of the traffic light recognition model. If the intelligent driving vehicle uses the traffic light recognition model, the driving safety of the vehicle can be improved.

[0109] The method of the embodiment of the present application is described in detail below.

[0110] See also Figure 2 , Figure 2 1 is a flow chart of a traffic light recognition model training method provided in an embodiment of the present application. Optionally, the method can be applied to a processing device.

[0111] like Figure 2 The traffic light recognition model training method shown may include multiple steps from step S201 to step S203. It should be understood that for the convenience of description, this application is described in the order of step S201 to step S203, and is not intended to limit the execution to the above order. The embodiment of the application does not limit the execution order, execution time, execution number, etc. of the above one or more steps. Steps S201 to step S203 are as follows:

[0112] Step S201: The processing device obtains a difficult traffic light image.

[0113] A processing device is a device with data processing capabilities. Exemplarily, the processing device may be a mobile phone, a tablet computer, a personal computer, a vehicle, a server, a cloud, or a host, etc. It may also be a virtual device, such as a virtual machine, software, program code, or a container, etc.

[0114] A difficult traffic light image refers to a traffic light image that is marked as a difficult example. Optionally, the annotations here may be manual annotations and / or pre-annotations by a deep learning model. As a possible implementation, the processing device acquires collected road images, extracts traffic light images from these road images, and pre-annotates traffic light images that may be difficult to accurately identify as difficult traffic light images through manual annotations and / or deep learning models, such as traffic light images with blurred images, resolutions below a first threshold, traffic light images with complex backgrounds manually judged, or traffic light images with obstructed traffic lights. The first threshold is predefined or set, for example, the first threshold is 60%.

[0115] Step S202: The processing device obtains a shape label corresponding to the difficult traffic light image based on the difficult traffic light image and N shape classifiers, and obtains a color label corresponding to the difficult traffic light image based on the difficult traffic image and N color classifiers.

[0116] As a possible implementation, the processing device inputs the difficult traffic light image into N shape classifiers to obtain M shape recognition results, and obtains the shape label corresponding to the difficult traffic light image based on the M shape recognition results. Wherein, M is an integer, and M is greater than 0, and M is less than or equal to N.

[0117] In one possible implementation, the difficult traffic light image is input into N shape classifiers to obtain M shape recognition results. There are many possible implementation methods, and several possible implementation methods are introduced below.

[0118] In the first implementation, a difficult traffic light image is input into N shape classifiers to obtain a shape recognition result of each shape classifier. In this case, M is set equal to N, and the shape recognition result of each shape classifier is considered valid. For example, a difficult traffic light image is input into 5 shape classifiers to obtain a shape recognition result of each shape classifier.

[0119] For ease of understanding, Table 1 shows the shape recognition results output by each shape classifier.

[0120] Table 1

[0121] Shape Classifier A1 B1 C1 D1 E1 Shape recognition results straight Turn right Turn right straight Turn right

[0122] As shown in Table 1, the shape recognition result of the shape classifier A1 is going straight, and the shape recognition result of the shape classifier E1 is turning right.

[0123] In the second implementation mode, a difficult traffic light image is input into N shape classifiers to obtain N shape category confidences corresponding to the difficult traffic light image, and M shape recognition results are obtained based on the N shape category confidences corresponding to the difficult traffic light image. Each of the N shape classifiers corresponds to one shape category confidence among the N shape category confidences, and the shape category confidence corresponding to each of the M shape recognition results is greater than or equal to a third threshold. The third threshold is a predefined or set value, for example, the third threshold is 68%. Exemplarily, a difficult traffic light image is input into 5 shape classifiers to obtain 5 shape category confidences corresponding to the difficult traffic light image. It should be noted that in the present application, each shape classifier outputs the maximum shape category confidence.

[0124] For ease of understanding, Table 2 shows the shape category confidence output by each shape classifier.

[0125] Table 2

[0126]

[0127] As shown in Table 2, the shape category confidence of shape classifier A1 is 80% for straight-ahead movement, and the shape category confidence of shape classifier E1 is 60% for right-turn movement.

[0128] Exemplarily, if the third threshold is 50%, the five shape classifiers can obtain three shape recognition results. For ease of understanding, Table 3 shows the three shape recognition results.

[0129] Table 3

[0130]

[0131] As shown in Table 3, the shape recognition result of the shape classifier A1 is a straight line, and the shape recognition result of the shape classifier D1 is a straight line.

[0132] In a third implementation mode, the processing device integrates all shape classifiers together for recognition. For example, the processing device obtains shape recognition results of all shape classifiers.

[0133] Each shape recognition result satisfies the following formula (1):

[0134]

[0135] Among them, classifier i1 is the shape recognition result of the i-th shape classifier, conf i1is the shape category confidence of the i-th shape classifier. That is, only when the shape category confidence of the shape classifier is greater than or equal to 0.5, its shape recognition result is 1, and the other shape recognition results are 0. Combining the above Table 2 and formula (1), the following Table 4 can be obtained.

[0136] Table 4

[0137]

[0138] As shown in Table 4, the shape recognition result of the shape classifier A1 is a straight line with a value of 1, and the shape recognition result of the shape classifier D1 is a straight line with a value of 1.

[0139] It should be understood that the above-mentioned embodiments are illustrative examples, and there may be more embodiments in the specific implementation process. In addition, multiple embodiments may be combined.

[0140] Furthermore, the processing device obtains a shape label corresponding to the difficult traffic light image based on the M shape recognition results.

[0141] In a possible implementation, there are multiple possible implementation methods for obtaining the shape label corresponding to the difficult traffic light image based on M shape recognition results. Several possible implementation methods are introduced below.

[0142] Implementation method 1: Each of the M shape recognition results indicates a result value, and based on the quantity distribution characteristics of each result value indicated by the M shape recognition results, a final shape recognition result corresponding to the difficult traffic light image is obtained by voting, and the shape label corresponding to the difficult traffic light image is the shape category corresponding to the final shape recognition result. The result value indicated by the final shape recognition result is the same as the result value indicated by the majority of the shape recognition results in each result value indicated by the M shape recognition results.

[0143] Exemplarily, the result values ​​are the same or the number of results with the same result value is higher than a preset condition, and the preset condition may be that the number is greater than a fifth threshold, or the number is the largest, or the proportion is greater than a sixth threshold, or the proportion is greater than 50%. The fifth threshold and the sixth threshold may be predefined or set values.

[0144] The characteristics of data distribution can be described from three aspects: central tendency, deviation from the center, skewness and kurtosis. For example, here we take the central tendency as an example and select the mode of the data as the result value indicating the final processing result. The mode is the data value that appears most frequently in a set of data. A set of data may have no mode or multiple modes. When data has no mode or multiple modes, it is considered that the vote fails, the vote cannot be output and the system alarm is issued.

[0145] Exemplarily, in combination with the five shape recognition results obtained in the first embodiment (ie, the five shape recognition results shown in Table 1), Table 5 shows a result value table of these shape recognition results.

[0146] Table 5

[0147]

[0148] As shown in Table 5, the result value corresponding to going straight is R1, and the result value corresponding to turning right is R2.

[0149] For ease of understanding, Table 6 shows the quantity distribution and proportion distribution of each result value.

[0150] Table 6

[0151] Result value R1 R2 Quantity distribution 2 3 Proportional Distribution 40％ 60％

[0152] As shown in Table 6, the number of R1s is 2, accounting for 40%, and the number of R2s is 3, accounting for 60%.

[0153] Exemplarily, the quantity distribution may be the number of times each result value appears, and the proportion distribution may be the proportion of the number of times each result value appears in the total number of times all result values ​​appear.

[0154] It can be seen from Table 6 that the result value indicated by most shape recognition results is R2, so the result value indicated by the final shape recognition result is R2, so the shape label corresponding to the difficult traffic light image is the shape category corresponding to the final shape recognition result, that is, the shape label corresponding to the difficult traffic light image is R2 and the corresponding shape category is right turn.

[0155] As another example, the three shape recognition results obtained in the second embodiment above (i.e., the three shape recognition results shown in Table 3) are combined to determine the result value corresponding to each shape recognition result, which are "the shape recognition result of shape classifier A1 is R1 for going straight", "the shape recognition result of shape classifier D1 is R1 for going straight", and "the shape recognition result of shape classifier E1 is R2 for turning right". It can be seen that the result value indicated by most shape recognition results is R1, so the result value indicated by the final shape recognition result is R1, so the shape label corresponding to the difficult traffic light image is the shape category corresponding to the final shape recognition result, that is, the shape label corresponding to the difficult traffic light image is R1, and the shape category corresponding to the shape label is R1.

[0156] Optionally, the processing device combines the shape recognition results obtained in the third embodiment above, votes on all the shape recognition results, and selects the category with the most votes as the final shape recognition result.

[0157] The final shape recognition result satisfies the following formula (2):

[0158]

[0159] value, m is the total number of shape recognition results for each category, classifier i1 That is, the processing device determines the total number of shape recognition results of each category as 1, selects the category corresponding to the shape recognition result with the largest total number as the shape label m corresponding to the difficult example traffic light image label1 .

[0160] Combining with the above Table 4, it can be obtained that the total number of shape recognition results of the straight category being 1 is 2, and the total number of shape recognition results of the right turn category being 1 is 1, so the processing device selects the straight category as the shape label corresponding to the difficult example traffic light image.

[0161] In this way, the majority principle is adopted for voting or voting, and the one with more identical shape recognition results is taken as the final shape recognition result. The shape category corresponding to the final shape recognition result is output as the shape label corresponding to the difficult traffic light image, which can improve the stability of the system and the accuracy of shape recognition.

[0162] Implementation method 2: For the shape recognition results whose corresponding shape category confidence is greater than or equal to the seventh threshold among the M shape recognition results, the weight is set to 2 (it should be noted that it can be set to any other value in actual use), and the weight of the remaining shape recognition results whose shape category confidence is less than the seventh threshold is set to 1, and the total weight of the same shape recognition results is determined, and the shape recognition results with the same shape recognition results with a large total weight are used as the final shape recognition results, and the shape category corresponding to the final shape recognition result is used as the shape label corresponding to the difficult traffic light image. Among them, the seventh threshold is a predefined or set value.

[0163] Exemplarily, combining the five shape category confidences obtained in the above-mentioned second embodiment and the three shape recognition results obtained (i.e., the five shape category confidences shown in Table 3), the seventh threshold is set to 80%, and Table 7 shows a weight value table of these shape recognition results.

[0164] Table 7

[0165]

[0166] As shown in Table 7, the total weight value of going straight is 3, and the total weight value of turning right is 1. Going straight is selected as the shape label corresponding to the difficult traffic light image.

[0167] It should be understood that the above implementations are illustrative examples, and there may be more implementations during the specific implementation process. In addition, multiple implementations may be combined.

[0168] In one possible implementation, the difficult traffic light image is input into N color classifiers to obtain P color recognition results. There are many possible implementations, similar to the above shape recognition results. Several possible implementations are introduced below.

[0169] In the first implementation, a difficult traffic light image is input into N color classifiers to obtain a color recognition result of each color classifier. In this case, P is set equal to N, and the color recognition result of each color classifier is considered valid. For example, a difficult traffic light image is input into 5 color classifiers to obtain a color recognition result of each color classifier.

[0170] For ease of understanding, Table 8 shows the color recognition results output by each color classifier.

[0171] Table 8

[0172] Color Sorter A2 B2 C2 D2 E2 Color recognition results red green green red green

[0173] As shown in Table 8, the color recognition result of the color classifier A2 is red, and the color recognition result of the color classifier E2 is green.

[0174] Embodiment 2: Input a difficult traffic light image into N color classifiers to obtain N color category confidences corresponding to the difficult traffic light image, and obtain P color recognition results based on the N color category confidences corresponding to the difficult traffic light image. Each of the N color classifiers corresponds to one color category confidence among the N color category confidences, and the color category confidence corresponding to each of the P color recognition results is greater than or equal to a fourth threshold. The fourth threshold is a predefined or set value, for example, the fourth threshold is 68%. Exemplarily, input a difficult traffic light image into 5 color classifiers to obtain 5 color category confidences corresponding to the difficult traffic light image. It should be noted that in the present application, each color classifier outputs the maximum color category confidence.

[0175] For ease of understanding, Table 9 shows the color category confidence output by each color classifier.

[0176] Table 9

[0177]

[0178] As shown in Table 9, the color category confidence of color classifier A2 is 80% for red, and the color category confidence of color classifier E2 is 60% for green.

[0179] Exemplarily, if the fourth threshold is 50%, the five color classifiers can obtain three color recognition results. For ease of understanding, Table 10 shows the three color recognition results.

[0180] Table 10

[0181]

[0182] As shown in Table 10, the color recognition result of the color classifier A2 is red, and the color recognition result of the color classifier D2 is red.

[0183] In a third implementation mode, the processing device integrates all color classifiers together for recognition. For example, the processing device obtains color recognition results of all color classifiers.

[0184] Each color recognition result satisfies the following formula (3):

[0185]

[0186] Among them, classifier i2 is the color recognition result of the i-th color classifier, conf i2 is the color category confidence of the i-th color classifier. That is, only when the color category confidence of the color classifier is greater than or equal to 0.5, its color recognition result is 1, and the rest of the color recognition results are 0. Combining the above Table 9 and formula (3), the following Table 11 can be obtained.

[0187] Table 11

[0188]

[0189] As shown in Table 11, the color recognition result of color classifier A2 is red, with a value of 1, and the color recognition result of color classifier D2 is red, with a value of 1.

[0190] It should be understood that the above-mentioned embodiments are illustrative examples, and there may be more embodiments in the specific implementation process. In addition, multiple embodiments may be combined.

[0191] Furthermore, the processing device obtains a color label corresponding to the difficult traffic light image based on the P color recognition results.

[0192] In one possible implementation, there are multiple possible implementation methods for obtaining the color label corresponding to the difficult traffic light image based on P color recognition results. Several possible implementation methods are introduced below.

[0193] Implementation method 1: Each color recognition result in P color recognition results indicates a result value, and based on the quantity distribution characteristics of each result value indicated by the P color recognition results, a final color recognition result corresponding to the difficult traffic light image is obtained by voting, and the color label corresponding to the difficult traffic light image is the color category corresponding to the final color recognition result. The result value indicated by the final color recognition result is the same as the result value indicated by the majority of the color recognition results in each result value indicated by the P color recognition results.

[0194] Exemplarily, the mode of the data is selected here as the result value indicated by the final processing result.

[0195] Exemplarily, in combination with the five color recognition results obtained in the first embodiment (ie, the five color recognition results shown in Table 1), Table 12 shows a result value table of these color recognition results.

[0196] Table 12

[0197]

[0198] As shown in Table 12, the result value corresponding to red is R3, and the result value corresponding to green is R4.

[0199] For ease of understanding, Table 13 shows the quantity distribution and proportion distribution of each result value.

[0200] Table 13

[0201] Calculation result value R3 R4 Quantity distribution 2 3 Proportional Distribution 40％ 60％

[0202] As shown in Table 13, the number of R3 is 2, accounting for 40%, and the number of R4 is 3, accounting for 60%.

[0203] It can be seen from Table 13 that the result value indicated by most color recognition results is R4, so the result value indicated by the final color recognition result is R4, so the color label corresponding to the difficult traffic light image is the color category corresponding to the final color recognition result, that is, the color label corresponding to the difficult traffic light image is R4 and the corresponding color category is green.

[0204] As another example, the three color recognition results obtained in the above-mentioned second embodiment (i.e., the three color recognition results shown in Table 3) are combined to determine the result value corresponding to each color recognition result, which are "the color recognition result of color classifier A2 is red, the result value is R3", "the color recognition result of color classifier D2 is red, the result value is R3", and "the color recognition result of color classifier E2 is green, the result value is R4". It can be seen that the result value indicated by most color recognition results is R3, so the result value indicated by the final color recognition result is R3, so the color label corresponding to the difficult traffic light image is the color category corresponding to the final color recognition result, that is, the color label corresponding to the difficult traffic light image is R3, and the corresponding color category is red.

[0205] Optionally, the processing device votes on all color recognition results in combination with the color recognition results obtained in the third embodiment, and selects the category with the most votes as the final color recognition result.

[0206] The final color recognition result satisfies the following formula (4):

[0207]

[0208] value, m is the total number of color recognition results for each category, classifier i2 That is, the processing device determines the total number of color recognition results of each category that are 1, and selects the category corresponding to the color recognition result with the largest total number as the color label m corresponding to the difficult traffic light image. label2 .

[0209] Combining the above Table 11, it can be obtained that the total number of color recognition results of category red being 1 is 2, and the total number of color recognition results of category green being 1 is 1, so the processing device selects red as the color label corresponding to the difficult example traffic light image.

[0210] In this way, the majority principle is adopted for voting, the one with the most identical color recognition results is taken as the final color recognition result, and the color category corresponding to the final color recognition result is output as the color label corresponding to the difficult traffic light image, which can improve the stability of the system and the accuracy of color recognition.

[0211] Implementation method two: For the color recognition results whose corresponding color category confidence is greater than or equal to the seventh threshold among the P color recognition results, the weight is set to 2 (it should be noted that it can be set to any other value in actual use), and the weight of the remaining color recognition results whose color category confidence is less than the seventh threshold is set to 1. The total weight of the same color recognition results is determined, and the color recognition result with a large total weight of the same color recognition result is used as the final color recognition result, and the color category corresponding to the final color recognition result is used as the color label corresponding to the difficult traffic light image.

[0212] Exemplarily, combining the 5 color category confidences obtained in the above-mentioned embodiment 2 and the 3 color recognition results obtained (i.e., the 5 color category confidences shown in Table 10), the seventh threshold is set to 80%, and Table 14 shows a weight value table of these color recognition results.

[0213] Table 14

[0214]

[0215] As shown in Table 14, the total weight value of red is 3, the total weight value of green is 1, and red is selected as the color label corresponding to the difficult traffic light image.

[0216] It should be understood that the above implementations are illustrative examples, and there may be more implementations during the specific implementation process. In addition, multiple implementations may be combined.

[0217] Step S203: The processing device trains a traffic light recognition model according to the difficult traffic light image, the shape label corresponding to the difficult traffic light image, and the color label corresponding to the difficult traffic light image.

[0218] As a possible implementation method, the processing device inputs the difficult traffic light image, the shape label corresponding to the difficult traffic light image, and the color label corresponding to the difficult traffic light image into the deep learning model for training, and obtains a traffic light recognition model that can recognize traffic light images. In this way, the training samples of the traffic light recognition model include the difficult traffic light image, the shape label corresponding to the difficult traffic light image, and the color label corresponding to the difficult traffic light image, which can improve the recognition ability of the traffic light recognition model. For traffic light images obtained by image sensors with poor imaging, the traffic light recognition model can be used to recognize them, which can effectively solve the problem that data caused by small targets or blurred targets cannot be effectively labeled, and can improve the recognition accuracy of the traffic light recognition model, enhance the performance of the traffic light recognition model, and thus improve the driving safety of autonomous driving vehicles.

[0219] Optionally, the trained traffic light recognition model can be compressed and quantized and then deployed to an on-board embedded processor of a vehicle, so that the vehicle can complete the traffic light recognition task.

[0220] Optional, Figure 2 The traffic light recognition model training method shown further includes step S204 and step S205 before step S202. Step S204 and step S205 are introduced below.

[0221] Step S204: The processing device obtains N sample sets.

[0222] Each sample set of the N sample sets includes multiple traffic light samples, each traffic light sample includes a traffic light image, a shape label corresponding to the traffic light image, and a color label corresponding to the traffic light image, and N is an integer and N is greater than 0.

[0223] Exemplarily, through manual labeling, the shape label corresponding to the traffic light image and the color label corresponding to the traffic light image are manually labeled. For example, the shape label of the traffic light image is labeled as straight, and the color label of the traffic light image is labeled as green. For some traffic light images that cannot be manually determined, these images are manually labeled as difficult traffic light images.

[0224] As another example, traffic light images are pre-labeled by a deep learning model, and these traffic light images are input into the deep learning model, and the shape recognition result and color recognition result of each traffic light image are output. For example, if the shape recognition result of the output traffic light image is a left turn, and the color recognition result of the output traffic light image is red, then the shape label of the traffic light image is labeled as a left turn, and the color label of the traffic light image is labeled as red. If the deep learning model outputs the shape recognition result of a certain traffic light image, and the probability of the traffic light image being predicted as any shape is the same, it can be considered that the traffic light image is "uncertain" for the deep learning model and is very likely to be misclassified. The traffic light image can be labeled as a difficult traffic light image. Optionally, the deep learning model is trained based on some traffic light images that have been successfully labeled with shape labels and color labels and their corresponding labels.

[0225] Furthermore, these traffic light images that have been successfully annotated with shape labels and color labels and their corresponding labels are used as traffic light samples, and the processing device obtains N sample sets from these traffic light samples through a sampling method with replacement. Exemplarily, the sampling method can be a bootstrap sampling method. Each sample set contains multiple traffic light samples, that is, each sample set includes a traffic light image, a shape label corresponding to the traffic light image, and a color label corresponding to the traffic light image. Optionally, the N sample sets do not include difficult traffic light images.

[0226] Step S205: the processing device inputs the N sample sets into N shape classifier models and N color classifier models respectively, and trains to obtain N shape classifiers and N color classifiers.

[0227] Among them, one shape classifier among the N shape classifiers and one color classifier among the N color classifiers correspond to one sample set among the N sample sets, that is, one sample set can be trained to obtain one shape classifier and one color classifier. Optionally, the N shape classifiers and N color classifiers are trained by bootstrap aggregating (Bagging) method. Among them, the basic idea of ​​Bagging is to generate multiple different training subsets by performing random sampling with replacement (for example, Bootstrap sampling) on ​​the training data, then train a weak model on each subset, and finally combine the prediction results of these sub-models.

[0228] As a possible implementation, the processing device inputs N sample sets into N shape classifier models to train N shape classifiers, and inputs N sample sets into N color classifier models to train N color classifiers. Optionally, the processing device may use a residual network (resnet) with a network depth of 18 as a shape classifier model or a color classifier model for classification training.

[0229] Exemplarily, the processing device inputs N sample sets into N shape classifier models, determines the traffic light shape category loss of any one of the N shape classifier models, and when the traffic light shape category loss of any one of the shape classifier models is greater than a first threshold, trains the any one of the shape classifier models again until the traffic light shape category loss of any one of the shape classifier models is less than or equal to the first threshold. In other words, the processing device determines the traffic light shape category loss of each shape classifier model, and when the traffic light shape category loss is greater than the first threshold, trains the shape classifier model again until the traffic light shape category loss of the shape classifier model is less than or equal to the first threshold. The first threshold may be a predefined or set value. Optionally, the shape classifier model may be repeatedly trained again using the sample set of the first training of the shape classifier model. It should be noted that each time the shape classifier model is trained, the shape classifier model will be fine-tuned according to the labeled data. The smaller the loss of the traffic light shape category, the more accurate the classification result of the shape classifier model. Through this process, the decision boundary of the shape classifier model is changed, the classification accuracy is improved, and the purpose of continuously optimizing the shape classifier model is achieved.

[0230] Exemplarily, the traffic light shape category loss of the shape classifier model satisfies the following formula (5):

[0231]

[0232] Among them, q i1 is the i-th actual shape label corresponding to the traffic light image in the sample set used for shape classifier model training, p i1 is the predicted shape label of the ith actual shape label corresponding to the traffic light image in the sample set used for the shape classifier model training, and n is the total number of shape labels of the traffic light images in the sample set used for the shape classifier model training. Exemplarily, if the sample set used for the shape classifier model training has 20 traffic light images, each traffic light image contains 8 shape labels, then the sample set contains a total of 160 labels, in which case n can be set to 160, and i can be set to be the ith label among the 160 labels.

[0233] Exemplarily, the processing device inputs N sample sets into N color classifier models, determines the traffic light color category loss of any one of the N color classifier models, and when the traffic light color category loss of any one of the color classifier models is greater than a second threshold, trains any one of the color classifier models again until the traffic light color category loss of any one of the color classifier models is less than or equal to the second threshold. In other words, the processing device determines the traffic light color category loss of each color classifier model, and when the traffic light color category loss is greater than the second threshold, trains the color classifier model again until the traffic light color category loss of the color classifier model is less than or equal to the second threshold. The second threshold may be a predefined or set value. Optionally, the color classifier model may be repeatedly trained again using the sample set of the first training of the color classifier model. It should be noted that each time the color classifier model is trained, the color classifier model will be fine-tuned according to the labeled data. The smaller the loss of the traffic light color category, the more accurate the classification result of the color classifier model. Through this process, the decision boundary of the color classifier model is changed, the classification accuracy is improved, and the purpose of continuously optimizing the color classifier model is achieved.

[0234] Exemplarily, the traffic light color category loss of the color classifier model satisfies the following formula (6):

[0235]

[0236] Among them, q i2 is the actual color label of the ith traffic light image in the sample set used for color classifier model training, p i2is the predicted color label of the ith actual color label corresponding to the traffic light image in the sample set used for the color classifier model training, and m is the total number of color labels of the traffic light images in the sample set used for the color classifier model training. For example, if the sample set used for the color classifier model training has 20 traffic light images, each traffic light image contains 6 color labels, then the sample set contains a total of 120 labels, in which case m can be set to 120, and i can be set to be the ith label among the 120 labels.

[0237] Optionally, the processing device can train a traffic light recognition model based on N sample sets, difficult traffic light images, shape labels corresponding to difficult traffic light images, and color labels corresponding to difficult traffic light images, thereby improving the recognition capability of the traffic light recognition model.

[0238] exist Figure 2 In the illustrated embodiment, the processing device can obtain shape labels corresponding to difficult traffic light images based on difficult traffic light images and N shape classifiers, and obtain color labels corresponding to difficult traffic light images based on difficult traffic images and N color classifiers. Then, the processing device can use difficult traffic light images, shape labels corresponding to difficult traffic light images, and color labels corresponding to difficult traffic light images as training samples of a traffic light recognition model to train a traffic light recognition model, which can improve the recognition ability of the traffic light recognition model. For traffic light images acquired by image sensors with poor imaging, the traffic light recognition model can be used to recognize the traffic light images, which can effectively solve the problem that data caused by small targets or blurred targets cannot be effectively labeled, and can improve the recognition accuracy of the traffic light recognition model and enhance the performance of the traffic light recognition model. If an intelligent driving vehicle uses the traffic light recognition model, the driving safety of the vehicle can be improved.

[0239] above Figure 2 There are many possible solutions for the application of the embodiment shown. For ease of understanding, one possible solution is introduced below. It should be understood that Figure 3 For explanations of some of the terms, logic, etc. in the scheme shown, see Figure 2 The embodiment shown.

[0240] See also Figure 3 , Figure 3 1 is a flow chart of a traffic light recognition method provided in an embodiment of the present application. Optionally, the method can be applied to a vehicle, for example, to a processing device on a vehicle. The method comprises the steps of:

[0241] Step S301: The processing device obtains a traffic light image to be identified.

[0242] Optionally, the processing device is a device with data processing capabilities, which may be arranged inside the vehicle, or may be a processing module integrated in the vehicle, such as a microcontroller unit MCU and / or an electronic control unit ECU. In some schemes, the processing device may also be arranged outside the vehicle, such as a server, cloud, or host, etc., or it may be a virtual device, such as a virtual machine, software, program code, or container. Optionally, when the processing device is installed on the vehicle, the functions implemented by the processing device can also be regarded as implemented by the vehicle. Specifically, when the processing device is installed on the vehicle, it can be a functional module, for example, the processing device can be integrated into the central domain controller of the vehicle or the vehicle-mounted embedded processor.

[0243] As a possible implementation, the vehicle includes an image sensor, which acquires the traffic light image to be identified through the image sensor, and transmits the traffic light image to be identified to the processing device, so that the processing device can acquire the traffic light image to be identified.

[0244] Step S302: the processing device inputs the traffic light image to be recognized into the traffic light recognition model to obtain a color recognition result and a shape recognition result corresponding to the traffic light image to be recognized.

[0245] Among them, the traffic light recognition model is based on Figure 2 The traffic light recognition model is trained using the traffic light recognition model training method shown.

[0246] Exemplarily, the traffic light recognition model is trained based on difficult traffic light images, shape labels corresponding to difficult traffic light images, and color labels corresponding to difficult traffic light images. In this way, the training samples of the traffic light recognition model include difficult traffic light images, shape labels corresponding to difficult traffic light images, and color labels corresponding to difficult traffic light images, which can improve the recognition ability of the traffic light recognition model. For traffic light images acquired by image sensors with poor imaging, the traffic light recognition model can be used to recognize them, which can effectively solve the problem that data caused by small targets or blurred targets cannot be effectively labeled, and can improve the recognition accuracy of the traffic light recognition model and the performance of the traffic light recognition model. If an intelligent driving vehicle uses the traffic light recognition model, the driving safety of the vehicle can be improved.

[0247] Exemplarily, the shape label corresponding to the difficult traffic light image is obtained based on M shape recognition results, where M is an integer, and M is greater than 0, and M is less than or equal to N. The M shape recognition results are obtained by inputting the difficult traffic light image into N shape classifiers, and the N shape classifiers are obtained by inputting N sample sets into N shape classifier models for training. The specific training process can be found in the above Figure 2 The relevant contents in will not be repeated here.

[0248] Exemplarily, the color label corresponding to the difficult traffic light image is obtained based on P color recognition results, where P is an integer, and P is greater than 0, and P is less than or equal to N. The P color recognition results are obtained by inputting the difficult traffic light image into N color classifiers, and the N color classifiers are obtained by inputting N sample sets into N color classifier models for training. The specific training process can be found in the above Figure 2 The relevant contents in will not be repeated here.

[0249] Exemplarily, the processing device inputs the traffic light image to be identified into the traffic light recognition model, and obtains that the color recognition result corresponding to the traffic light image to be identified is red, and the corresponding shape recognition result is left turn. The processing device can display "Left turn is red, left turn is prohibited" on the vehicle's display screen, thereby providing feedback to the driver on the identified traffic light status.

[0250] exist Figure 3 In the embodiment shown, the processing device can obtain a traffic light image to be identified, input the traffic light image to be identified into a traffic light recognition model, and obtain a color recognition result and a shape recognition result corresponding to the traffic light image to be identified. In this way, the processing device can quickly complete the traffic light recognition task and improve the user experience. If the intelligent driving vehicle uses the traffic light recognition model, the driving safety of the vehicle can be improved.

[0251] The method of the embodiment of the present application is described in detail above, and the device of the embodiment of the present application is provided below.

[0252] See also Figure 4 , Figure 4 40 is a schematic diagram of a processing device provided in an embodiment of the present application. The processing device 40 may include an acquisition unit 401 and a processing unit 402. The processing device 40 is used to implement the aforementioned traffic light recognition model training method, for example Figure 2 The traffic light recognition model training method in the illustrated embodiment.

[0253] It should be noted that the division of the above multiple units is only a logical division based on functions, and does not limit the specific structure of the processing device 40. In a specific implementation, some functional modules may be subdivided into more small functional modules, and some functional modules may be combined into one functional module.

[0254] In a possible implementation, the acquisition unit 401 is used to acquire a difficult traffic light image. The processing unit 402 is used to obtain a shape label corresponding to the difficult traffic light image based on the difficult traffic light image and N shape classifiers, and to obtain a color label corresponding to the difficult traffic light image based on the difficult traffic image and N color classifiers, and to train a traffic light recognition model based on the shape label corresponding to the difficult traffic light image and the color label corresponding to the difficult traffic light image.

[0255] In a possible implementation, before obtaining the shape label corresponding to the difficult traffic light image based on the difficult traffic light image and N shape classifiers, and obtaining the color label corresponding to the difficult traffic light image based on the difficult traffic image and N color classifiers, the acquisition unit 401 is further used to obtain N sample sets, each sample set in the N sample sets includes multiple traffic light samples, each traffic light sample includes a traffic light image, a shape label corresponding to the traffic light image, and a color label corresponding to the traffic light image, and N is an integer and N is greater than 0. The processing unit 402 is also used to input the N sample sets into N shape classifier models and N color classifier models, respectively, and train to obtain N shape classifiers and N color classifiers. Among them, one shape classifier in the N shape classifiers and one color classifier in the N color classifiers correspond to one sample set in the N sample sets.

[0256] In a possible implementation, the processing unit 402 is further configured to input the N sample sets into N shape classifier models and N color classifier models for training, and perform the following steps on any one of the N shape classifier models and any one of the N color classifier models:

[0257] Determine the traffic light shape category loss of any shape classifier model and the traffic light color category loss of any color classifier model. When the traffic light shape category loss of any shape classifier model is greater than a first threshold, train any shape classifier model again until the traffic light shape category loss of any shape classifier model is less than or equal to the first threshold. When the traffic light color category loss of any color classifier model is greater than a second threshold, train any color classifier model again until the traffic light color category loss of any color classifier model is less than or equal to the second threshold.

[0258] In a possible implementation, the processing unit 402 is further used to input the difficult traffic light image into N shape classifiers to obtain M shape recognition results, where M is an integer, and M is greater than 0, and M is less than or equal to N, and based on the M shape recognition results, a shape label corresponding to the difficult traffic light image is obtained.

[0259] In a possible implementation, the processing unit 402 is further used to input the difficult traffic light image into N color classifiers to obtain P color recognition results, where P is an integer, and P is greater than 0, and P is less than or equal to N, and based on the P color recognition results, a color label corresponding to the difficult traffic light image is obtained.

[0260] In a possible implementation, the processing unit 402 is further used to input the difficult example traffic light image into N shape classifiers to obtain N shape category confidences corresponding to the difficult example traffic light image, each of the N shape classifiers corresponds to one shape category confidence of the N shape category confidences, and based on the N shape category confidences corresponding to the difficult example traffic light image, M shape recognition results are obtained, and the shape category confidence corresponding to each of the M shape recognition results is greater than or equal to a third threshold.

[0261] In a possible implementation, the processing unit 402 is further used to input the difficult traffic light image into N color classifiers to obtain N color category confidences corresponding to the difficult traffic light image, each of the N color classifiers corresponds to one color category confidence of the N color category confidences, and based on the N color category confidences corresponding to the difficult traffic light image, P color recognition results are obtained, and the color category confidence corresponding to each of the P color recognition results is greater than or equal to a fourth threshold.

[0262] In one possible implementation, the processing unit 402 is further used to vote to obtain a final shape recognition result corresponding to the difficult traffic light image based on the quantity distribution characteristics of each result value indicated by the M shape recognition results, the result value indicated by the final shape recognition result is the same as the result value indicated by the majority of the shape recognition results in each result value indicated by the M shape recognition results, and the shape label corresponding to the difficult traffic light image is the shape category corresponding to the final shape recognition result.

[0263] In a possible implementation, the processing unit 402 is further used to vote to obtain a final color recognition result corresponding to the difficult traffic light image based on the quantity distribution characteristics of each result value indicated by the P color recognition results, the result value indicated by the final color recognition result is the same as the result value indicated by the majority of the color recognition results among each result value indicated by the P color recognition results, and the color label corresponding to the difficult traffic light image is the color category corresponding to the final color recognition result.

[0264] It should be noted that the above modules (acquisition unit 401 and processing unit 402) are used to execute the relevant steps of the above method. For example, acquisition unit 401 is used to execute the relevant content of step S201, and processing unit 402 is used to execute the relevant content of steps S202 to S204.

[0265] See also Figure 5 , Figure 5 is a schematic diagram of the structure of another processing device provided in an embodiment of the present application. The processing device 50 may include an acquisition unit 501 and a processing unit 502. The processing device 50 is used to implement the aforementioned traffic light recognition method, for example Figure 3 The traffic light recognition method in the illustrated embodiment.

[0266] It should be noted that the division of the above multiple units is only a logical division based on functions, and does not limit the specific structure of the processing device 50. In a specific implementation, some functional modules may be subdivided into more small functional modules, and some functional modules may be combined into one functional module.

[0267] In a possible implementation, the acquisition unit 501 is used to acquire a traffic light image to be recognized. The processing unit 502 is used to input the traffic light image to be recognized into a traffic light recognition model to obtain a color recognition result and a shape recognition result corresponding to the traffic light image to be recognized. The traffic light recognition model is based on Figure 2 The traffic light recognition model is trained using the traffic light recognition model training method shown.

[0268] It should be noted that the above modules (acquisition unit 501 and processing unit 502) are used to execute the relevant steps of the above method. For example, acquisition unit 501 is used to execute the relevant content of step S301, and processing unit 502 is used to execute the relevant content of step S302.

[0269] Figure 6 The figure shows a schematic diagram of the structure of a computing device provided in an embodiment of the present application. A computing device is a device with processing capabilities, and the device here can be a physical device, such as a server (such as a rack server), a host, etc., or a virtual device, such as a virtual machine, a container, etc.

[0270] like Figure 6 As shown, the computing device 60 includes: a processor 601 and a memory 602 and one or more programs, and may include a communication interface 603. It should be understood that the present application does not limit the number of processors and memories in the computing device 60.

[0271] The processor 601 is a module for performing calculations, and may include a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), a digital signal processor (DSP), a microcontroller unit (MCU), or one or more integrated circuits for controlling the execution of the above program.

[0272] The memory 602 is used to provide a storage space, and the storage space can optionally store application data, user data, operating system, computer programs, etc. The memory 602 may include a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, and may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0273] The memory 602 may exist independently and be connected to the processor 601 via a bus. The memory 602 may also be integrated with the processor 601 .

[0274] The communication interface 603 is used to provide information input or output for the at least one processor. And / or, the communication interface 603 can be used to receive data sent externally and / or send data to the outside. The communication interface 603 can be a wired link interface including an Ethernet cable, etc., or a wireless link (Wi-Fi, Bluetooth, general wireless transmission and other wireless communication technologies, etc.) interface. Optionally, the communication interface 603 can also include a transmitter (such as a radio frequency transmitter, an antenna, etc.) coupled to the interface, or a receiver, etc.

[0275] In the embodiment of the present application, the one or more programs are stored in the memory 602 in the form of program codes and are configured to be executed by the processor 601. The programs include instructions for implementing the steps in the aforementioned traffic light recognition model training method or traffic light recognition method. Figure 2 The traffic light recognition model training method shown or Figure 3 That is, the memory 602 stores executable instructions, and the processor 601 executes the executable instructions to implement the aforementioned traffic light recognition model training method or traffic light recognition method, for example Figure 2 The traffic light recognition model training method or Figure 3That is, the memory 602 stores instructions for executing the traffic light recognition model training method or the traffic light recognition method.

[0276] Alternatively, executable instructions are stored in the memory 602, and the processor 601 executes the executable instructions to respectively implement the functions of one or more units (or devices) in the aforementioned acquisition unit 401 and the processing unit 402, thereby realizing the traffic light recognition model training method, or the processor 601 executes the executable instructions to respectively implement the functions of one or more units (or devices) in the aforementioned acquisition unit 501 and the processing unit 502, thereby realizing the traffic light recognition method.

[0277] The embodiment of the present application further provides a vehicle, the vehicle comprising the aforementioned processing device 50 or the aforementioned computing device 60, the vehicle being used to implement the aforementioned traffic light recognition method, for example Figure 3 The traffic light recognition method in the embodiment of the present invention.

[0278] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. The computer program instructions are used to implement the aforementioned traffic light recognition model training method or traffic light recognition method, for example Figure 2 The traffic light recognition model training method shown or Figure 3 The traffic light recognition method shown.

[0279] The present application also provides a computer-readable storage medium. The computer-readable storage medium includes instructions for implementing the aforementioned traffic light recognition model training method or traffic light recognition method, for example Figure 2 The traffic light recognition model training method shown or Figure 3 The traffic light recognition method shown.

[0280] The computer-readable storage medium may be any available medium that can be stored by the processing device, or a data storage device such as a data center that contains one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk).

[0281] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0282] The "at least one" mentioned in the embodiments of the present application refers to one or more, and "plurality" refers to two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, (a and b), (a and c), (b and c), or (a and b and c), where a, b, c can be single or multiple. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can be represented by: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0283] Furthermore, unless otherwise specified, the ordinal numbers such as "first" and "second" used in the embodiments of the present application are used to distinguish multiple objects, and are not used to limit the order, timing, priority or importance of multiple objects. For example, the first threshold and the second threshold are only for the convenience of description, and do not represent the difference in the deployment order, importance, etc. of the first threshold and the second threshold.

[0284] Those skilled in the art will appreciate that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk, or an optical disk, etc.

[0285] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A traffic light recognition model training method, characterized in that: The method comprises: Get a hard example traffic light image; Obtaining a shape label corresponding to the difficult traffic light image based on the difficult traffic light image and N shape classifiers, and obtaining a color label corresponding to the difficult traffic light image based on the difficult traffic image and N color classifiers; A traffic light recognition model is trained based on the difficult traffic light image, the shape label corresponding to the difficult traffic light image, and the color label corresponding to the difficult traffic light image.

2. The method according to claim 1, characterized in that Before obtaining a shape label corresponding to the difficult traffic light image based on the difficult traffic light image and N shape classifiers, and obtaining a color label corresponding to the difficult traffic light image based on the difficult traffic image and N color classifiers, the method further includes: Obtain N sample sets, each of the N sample sets includes a plurality of traffic light samples, each traffic light sample includes a traffic light image, a shape label corresponding to the traffic light image, and a color label corresponding to the traffic light image, where N is an integer and is greater than 0; The N sample sets are respectively input into N shape classifier models and N color classifier models, and N shape classifiers and N color classifiers are obtained through training. One shape classifier among the N shape classifiers and one color classifier among the N color classifiers correspond to one sample set among the N sample sets.

3. The method according to claim 2, characterized in that The N sample sets are input into N shape classifier models and N color classifier models respectively, and N shape classifiers and N color classifiers are obtained by training, including: Inputting the N sample sets into the N shape classifier models and the N color classifier models for training respectively; The following steps are performed on any one of the N shape classifier models and any one of the N color classifier models: Determine a traffic light shape category loss of any one of the shape classifier models and a traffic light color category loss of any one of the color classifier models; When the traffic light shape category loss of any one of the shape classifier models is greater than the first threshold, training the any one of the shape classifier models again until the traffic light shape category loss of any one of the shape classifier models is less than or equal to the first threshold; When the traffic light color category loss of any one of the color classifier models is greater than the second threshold, the any one of the color classifier models is trained again until the traffic light color category loss of any one of the color classifier models is less than or equal to the second threshold.

4. The method according to any one of claims 1 to 3, characterized in that: The obtaining a shape label corresponding to the difficult traffic light image based on the difficult traffic light image and N shape classifiers includes: Input the difficult traffic light image into the N shape classifiers to obtain M shape recognition results, where M is an integer, and M is greater than 0 and M is less than or equal to N; Based on the M shape recognition results, a shape label corresponding to the difficult traffic light image is obtained.

5. The method according to any one of claims 1 to 3, characterized in that: The obtaining a color label corresponding to the difficult traffic light image based on the difficult traffic image and N color classifiers includes: Input the difficult traffic light image into the N color classifiers to obtain P color recognition results, where P is an integer, and P is greater than 0 and P is less than or equal to N; Based on the P color recognition results, a color label corresponding to the difficult traffic light image is obtained.

6. The method according to claim 4, characterized in that The step of inputting the difficult traffic light image into the N shape classifiers to obtain M shape recognition results includes: Inputting the difficult example traffic light image into the N shape classifiers to obtain N shape category confidences corresponding to the difficult example traffic light image, each shape classifier in the N shape classifiers corresponding to one shape category confidence among the N shape category confidences; According to the N shape category confidences corresponding to the difficult traffic light image, M shape recognition results are obtained, and the shape category confidence corresponding to each of the M shape recognition results is greater than or equal to a third threshold.

7. The method according to claim 5, characterized in that The step of inputting the difficult traffic light image into the N color classifiers to obtain P color recognition results includes: Inputting the difficult example traffic light image into the N color classifiers to obtain N color category confidences corresponding to the difficult example traffic light image, each color classifier in the N color classifiers corresponding to one color category confidence in the N color category confidences; According to the N color category confidences corresponding to the difficult traffic light image, P color recognition results are obtained, and the color category confidence corresponding to each of the P color recognition results is greater than or equal to a fourth threshold.

8. The method according to claim 4, characterized in that Each of the M shape recognition results indicates a result value, and obtaining a shape label corresponding to the difficult traffic light image based on the M shape recognition results includes: Based on the quantity distribution characteristics of each result value indicated by the M shape recognition results, voting is performed to obtain a final shape recognition result corresponding to the difficult traffic light image, wherein the result value indicated by the final shape recognition result is the same as the result value indicated by the majority of the shape recognition results in each result value indicated by the M shape recognition results; The shape label corresponding to the difficult traffic light image is the shape category corresponding to the final shape recognition result.

9. The method according to claim 5, characterized in that Each of the P color recognition results indicates a result value, and obtaining a color label corresponding to the difficult traffic light image based on the P color recognition results includes: Based on the quantity distribution characteristics of each result value indicated by the P color recognition results, voting is performed to obtain a final color recognition result corresponding to the difficult traffic light image, wherein the result value indicated by the final color recognition result is the same as the result value indicated by the majority of the color recognition results in each result value indicated by the P color recognition results; The color label corresponding to the difficult traffic light image is the color category corresponding to the final color recognition result.

10. A traffic light recognition method, characterized in that: The method comprises: Obtain a traffic light image to be identified; The traffic light image to be identified is input into a traffic light recognition model to obtain a color recognition result and a shape recognition result corresponding to the traffic light image to be identified; the traffic light recognition model is trained using the traffic light recognition model training method as described in any one of claims 1 to 9.

11. A computing device, characterized in that: The computing device includes a processor and a memory, wherein a program is stored in the memory, and the processor executes the program so that the computing device implements the method according to any one of claims 1 to 9, or the processor executes the program so that the computing device implements the method according to claim 10.

12. A vehicle, characterized in that: The vehicle comprises the computing device of claim 11 , and the vehicle is configured to implement the method of claim 10 .

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, wherein the computer program includes instructions for executing the method according to any one of claims 1 to 10.

Images