Traffic light status recognition method, device, vehicle and storage medium

Abstract

The present disclosure relates to the field of autonomous driving, and specifically to a traffic light state recognition method, device, vehicle and storage medium. The traffic light state recognition method collects multiple frames of traffic light state images within a preset time period through an image acquisition device on the vehicle; determines the state timing information of the traffic light ahead within the preset time period based on the multiple frames of traffic light state images; determines the current target traffic light state based on the state timing information, the target traffic light image at the current moment and the driving state information of the vehicle ahead, which can effectively improve the accuracy of the traffic light state recognition result, reduce the probability of the unmanned vehicle violating traffic regulations, thereby effectively improving the driving safety performance of the unmanned vehicle, and is conducive to improving the user experience of the unmanned vehicle.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving, and in particular to a method, device, vehicle, and storage medium for identifying traffic light status. Background Art

[0002] Currently, the perception algorithms used in autonomous driving can accurately identify the status of traffic lights in each frame of an image. However, statistics on the recognition results of traffic light status over a period of time show that the accuracy of traffic light status recognition is still relatively low, and it is easy for vehicles to violate traffic regulations due to incorrect traffic light status recognition. This is very detrimental to improving the vehicle user experience. Summary of the Invention

[0003] To overcome the problems existing in the related art, the present disclosure provides a traffic light status recognition method, device, vehicle and storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for identifying a traffic light state is provided, which is applied to a vehicle. The method includes:

[0005] When it is determined that the distance between the vehicle and the location of the traffic light ahead is less than or equal to a preset distance threshold, capturing a plurality of frames of traffic light status images within a preset time period by an image capture device on the vehicle;

[0006] Determining, based on the multiple frames of traffic light state images, state timing information of the traffic light ahead within the preset duration, the state timing information including the traffic light state corresponding to each time point within the preset duration, the traffic light state including at least one of a red light on / off state, a yellow light on / off state, and a green light on / off state;

[0007] Collecting a target traffic light image at the current moment and driving status information of the vehicle ahead of the vehicle;

[0008] The current target traffic light state is determined according to the state timing information, the target traffic light image and the driving state information.

[0009] Optionally, determining the state timing information of the traffic light ahead within the preset time period according to the multiple frames of traffic light state images includes:

[0010] Identifying the traffic light state in each frame of the traffic light state image to obtain the traffic light state corresponding to each image acquisition time point within the preset time period;

[0011] The traffic light state corresponding to each time point within the preset time period is determined according to the traffic light state corresponding to each image acquisition time point within the preset time period to obtain the state timing information.

[0012] Optionally, determining the current target traffic light state according to the state timing information, the target traffic light image, and the driving state information includes:

[0013] The state timing information, the target traffic light image, and the driving state information are input into a first preset recognition model to obtain the target traffic light state output by the first preset recognition model.

[0014] Optionally, the first preset recognition model is pre-trained in the following manner:

[0015] Acquire multiple groups of historical sample data corresponding to multiple traffic light intersections, each group of the historical sample data including state time sequence information, historical traffic light images, and historical driving state information of the traffic light intersection within the historical preset time period;

[0016] The first initial model is trained using the multiple groups of historical sample data as the first training data to obtain the first preset recognition model, wherein the first training data includes the traffic light status annotation data within a specified time period after the preset time period in history.

[0017] Optionally, determining the current target traffic light state according to the state timing information, the target traffic light image, and the driving state information includes:

[0018] Inputting the state time series information into a second preset recognition model to obtain predicted state time series information within a target time period after the preset time length output by the second preset recognition model;

[0019] Determining a first traffic light signal state corresponding to a current time point according to the predicted state time series information;

[0020] Acquiring a second traffic light signal state in the target traffic light image;

[0021] determining a third traffic light signal state according to the driving state information of the preceding vehicle;

[0022] The target traffic light state is determined according to the first traffic light signal state, the second traffic light signal state, and the third traffic light signal state.

[0023] Optionally, determining the target traffic light state according to the first traffic light signal state, the second traffic light signal state, and the third traffic light signal state includes:

[0024] Obtaining target weights of the first traffic light signal state, the second traffic light signal state, and the third traffic light signal state;

[0025] The target traffic light state is determined according to the first traffic light signal state, the second traffic light signal state, and the third traffic light signal state using the target weight.

[0026] Optionally, the second preset recognition model is pre-trained in the following manner:

[0027] Acquire multiple time series state sample information corresponding to multiple traffic light intersections in a historical time period, wherein the time series state sample information includes state annotation data within a target time period;

[0028] The second initial model is trained using the plurality of time series state sample information as second training data to obtain the second preset recognition model.

[0029] According to a second aspect of an embodiment of the present disclosure, a traffic light state recognition device is provided, which is applied to a vehicle. The device includes:

[0030] The first determining module is configured to, when determining that the distance between the vehicle and the location of the traffic light ahead is less than or equal to a preset distance threshold, capture, by an image capture device on the vehicle, a plurality of frames of traffic light status images within a preset time period;

[0031] a second determining module configured to determine, based on the multiple frames of traffic light state images, state timing information of the traffic light ahead within the preset duration, the state timing information including the traffic light state corresponding to each time point within the preset duration, the traffic light state including at least one of a red light on / off state, a yellow light on / off state, and a green light on / off state;

[0032] an acquisition module configured to acquire a target traffic light image at a current moment and driving status information of a vehicle ahead of the vehicle;

[0033] The third determining module is configured to determine the current target traffic light state according to the state timing information, the target traffic light image and the driving state information.

[0034] Optionally, the second determining module is configured to:

[0035] Identifying the traffic light state in each frame of the traffic light state image to obtain the traffic light state corresponding to each image acquisition time point within the preset time period;

[0036] The traffic light state corresponding to each time point within the preset time period is determined according to the traffic light state corresponding to each image acquisition time point within the preset time period to obtain the state timing information.

[0037] Optionally, the third determining module is configured to:

[0038] The state timing information, the target traffic light image and the driving state information are input into a first preset recognition model to obtain the target traffic light state output by the first preset recognition model.

[0039] Optionally, the first preset recognition model is pre-trained in the following manner:

[0040] Acquire multiple groups of historical sample data corresponding to multiple traffic light intersections, each group of the historical sample data including state time sequence information, historical traffic light images, and historical driving state information of the traffic light intersection within the historical preset time period;

[0041] The first initial model is trained using the multiple groups of historical sample data as the first training data to obtain the first preset recognition model, wherein the first training data includes the traffic light status annotation data within a specified time period after the preset time period in history.

[0042] Optionally, the third determining module is configured to:

[0043] Inputting the state time series information into a second preset recognition model to obtain predicted state time series information within a target time period after the preset time length output by the second preset recognition model;

[0044] Determining a first traffic light signal state corresponding to a current time point according to the predicted state time series information;

[0045] Acquiring a second traffic light signal state in the target traffic light image;

[0046] determining a third traffic light signal state according to the driving state information of the preceding vehicle;

[0047] The target traffic light state is determined according to the first traffic light signal state, the second traffic light signal state, and the third traffic light signal state.

[0048] Optionally, the third determining module is configured to:

[0049] Obtaining target weights of the first traffic light signal state, the second traffic light signal state, and the third traffic light signal state;

[0050] The target traffic light state is determined according to the first traffic light signal state, the second traffic light signal state, and the third traffic light signal state using the target weight.

[0051] Optionally, the second preset recognition model is pre-trained in the following manner:

[0052] Acquire multiple time series state sample information corresponding to multiple traffic light intersections in a historical time period, wherein the time series state sample information includes state annotation data within a target time period;

[0053] The second initial model is trained using the plurality of time series state sample information as second training data to obtain the second preset recognition model.

[0054] According to a third aspect of an embodiment of the present disclosure, a traffic light status recognition vehicle is provided, comprising:

[0055] processor;

[0056] a memory for storing processor-executable instructions;

[0057] Wherein, the processor is configured to:

[0058] Implement the steps of the method described in the first aspect above.

[0059] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the method described in the first aspect above are implemented.

[0060] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0061] The image acquisition device on the vehicle can capture multiple frames of traffic light status images within a preset time period; determine the status timing information of the traffic light ahead within the preset time period based on the multiple frames of traffic light status images; and determine the current target traffic light status based on the status timing information, the target traffic light image at the current moment, and the driving status information of the vehicle ahead. This can effectively improve the accuracy of traffic light status recognition results, reduce the probability of unmanned vehicles violating traffic regulations, and thus effectively improve the driving safety performance of unmanned vehicles, which is conducive to improving the user experience of unmanned vehicles.

[0062] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0064] Figure 1 is a flow chart of a method for identifying a traffic light state according to an exemplary embodiment of the present disclosure;

[0065] Figure 2is a schematic diagram of state timing information shown in an exemplary embodiment of the present disclosure;

[0066] Figure 3 According to this disclosure Figure 1 The illustrated embodiment shows a flow chart of a method for identifying a traffic light state;

[0067] Figure 4 is a schematic diagram of a model structure shown in an exemplary embodiment of the present disclosure;

[0068] Figure 5 is a block diagram of a traffic light state recognition device shown in an exemplary embodiment of the present disclosure;

[0069] Figure 6 It is a functional block diagram of a vehicle shown in an exemplary embodiment. DETAILED DESCRIPTION

[0070] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0071] It should be noted that all actions of acquiring signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0072] Before describing the specific embodiments of the present disclosure in detail, the following describes the application scenarios of the present disclosure. The present disclosure can be applied to vehicles, particularly unmanned vehicles or autonomous vehicles. An unmanned vehicle will be used as an example for illustration. Current unmanned driving technology requires perception algorithms to analyze the surrounding environment based on data acquired by various sensors. Perception can be likened to the human eye. Current perception algorithms understand fixed images well, but their ability to understand changing scenes is limited. For example, if a traffic light is red and both the preceding and following frames are red, it can be determined that the current light is red. The vehicle must obey traffic regulations and must stop within the stop line and not move. However, if the light is flashing yellow, at least one of the preceding and following frames captured by the image acquisition device may be black. Alternatively, a black light may occur during the transition from red to yellow, yellow to green, or red to green, green to yellow, etc. Current perception algorithms for unmanned vehicles are generally unable to accurately determine the state of a flashing traffic light, making it prone to driving in violation of traffic regulations due to incorrect traffic light state recognition. This is one of the reasons why the accuracy of current traffic light state recognition results is low. This is not only detrimental to improving vehicle safety performance, but also greatly detrimental to improving the vehicle user experience.

[0073] In order to solve the above technical problems, the present disclosure provides a traffic light state recognition method, device, vehicle and storage medium. The traffic light state recognition method collects multiple frames of traffic light state images within a preset time period through an image acquisition device on the vehicle; determines the state timing information of the traffic light in front within the preset time period based on the multiple frames of traffic light state images, and determines the current target traffic light state based on the state timing information, the target traffic light image at the current moment and the driving state information of the vehicle in front. This method can effectively improve the accuracy of the traffic light state recognition result, reduce the probability of unmanned vehicles violating traffic regulations, thereby effectively improving the driving safety performance of unmanned vehicles, and is conducive to improving the user experience of unmanned vehicles.

[0074] The technical solution of the present disclosure is described in detail below with reference to specific embodiments.

[0075] Figure 1 FIG. 1 is a flow chart of a method for identifying a traffic light state according to an exemplary embodiment of the present disclosure; FIG. Figure 1 As shown, the traffic light state recognition method can be applied to vehicles, including:

[0076] Step 101: When it is determined that the distance between the vehicle and the location of the traffic light ahead is less than or equal to a preset distance threshold, a multi-frame traffic light status image within a preset time period is captured by an image capture device on the vehicle.

[0077] The traffic light status image may be an image in a traffic light status video, and the image acquisition device may include a vehicle-mounted camera.

[0078] For example, when it is determined that the distance between the vehicle and the location of the traffic light ahead is less than or equal to 500 meters, a 15-second traffic light status video is obtained, and the traffic light status video includes multiple frames of traffic light status images, and each frame of the traffic light status image includes an image that can reflect the traffic light status. The traffic light status can be at least one of the red light bright or dark state, the yellow light bright or dark state, and the green light bright or dark state.

[0079] Step 102 : determining the state timing information of the traffic light ahead within the preset duration based on the multiple frames of traffic light state images, wherein the state timing information includes the state of the traffic light corresponding to each time point within the preset duration.

[0080] The traffic light state includes at least one of a red light on / off state, a yellow light on / off state, and a green light on / off state, and the preset duration may be one third or one half of a traffic light state cycle.

[0081] In this step, the traffic light status in each frame of the traffic light status image can be identified to obtain the traffic light status corresponding to each image acquisition time point within the preset time length; the traffic light status corresponding to each time point within the preset time length is determined based on the traffic light status corresponding to each image acquisition time point within the preset time length to obtain the status timing information.

[0082] The above-mentioned implementation method of determining the traffic light state corresponding to each time point within the preset time length based on the traffic light state corresponding to each image acquisition time point within the preset time length may be, when two adjacent image acquisition time points are both in the same traffic light state (for example, both are in the state of red light on, green light and yellow light off), then the traffic light state corresponding to the two image acquisition time points is used as the traffic light state corresponding to each time point between the two adjacent image acquisition time points. When the traffic light states corresponding to two adjacent image acquisition time points are different (for example, the traffic light state of the previous image acquisition time point is red light on, green light and yellow light off, and the traffic light state of the next image acquisition time point is yellow light on, green light and red light off), then the traffic light state corresponding to the middle time point between the two adjacent image acquisition time points is determined to be the switching state between the traffic light state at the previous image acquisition time point and the traffic light state at the next image acquisition time point (for example, the state of switching from red light to yellow light), and the time length of the middle time point from the two previous and next image acquisition time points is the same, thereby obtaining the state timing information within the preset time length, and the state timing information may be as follows: Figure 2 The state timing diagram shown in Figure 2FIG. 1 is a schematic diagram of state timing information according to an exemplary embodiment of the present disclosure.

[0083] It should be noted that the traffic light state in the traffic light state image may be identified using a perception algorithm in the prior art. The perception algorithm may be a neural network algorithm, or may include other algorithms in addition to the neural network algorithm.

[0084] Step 103 : collecting the target traffic light image at the current moment and the driving status information of the vehicle ahead of the vehicle.

[0085] Among them, the target traffic light image is the traffic light image at the current moment captured by the image acquisition device on the vehicle, the front vehicle is a vehicle that is traveling in the same direction as the vehicle, is located in front of the vehicle, and is in the same lane as the vehicle, and the driving status information at least includes the forward driving speed.

[0086] Step 104 : Determine the current target traffic light state according to the state timing information, the target traffic light image, and the driving state information.

[0087] In this step, a possible implementation method is: inputting the state timing information, the target traffic light image and the driving state information into a first preset recognition model to obtain the target traffic light state output by the first preset recognition model.

[0088] The first preset recognition model is pre-trained in the following manner:

[0089] Acquire multiple groups of historical sample data corresponding to multiple traffic light intersections, each group of the historical sample data includes state timing information, historical traffic light images, and historical driving state information of the traffic light intersection within a historical preset time length; use the multiple groups of historical sample data as first training data to train a first initial model to obtain the first preset recognition model, and the first training data includes traffic light state annotation data within a specified time length after the historical preset time length.

[0090] It should be noted that the historical traffic light image is a traffic light image at a historical moment collected by the vehicle. For example, if the state time series information within the preset time period of the history is a time series diagram from 15:01:20 on January 2, 2021 to 15:01:30 on January 2, 2021, then the traffic light image at the historical moment can be a traffic light image collected within the target time period (for example, 20 minutes) after 15:01:30 on January 2, 2021; the historical driving state information is the driving state information of the vehicle in front of the vehicle that collected the historical traffic light image at the historical moment (for example, the moment when the historical traffic light image was collected, or the time specified from the moment when the historical traffic light image was collected). In addition, it should be noted that the first initial model can be a neural network model or other machine learning models in the prior art.

[0091] Another possible implementation may include Figure 3 The steps shown are: Figure 3 According to this disclosure Figure 1 The embodiment shown is a flow chart of a method for identifying a traffic light state; Figure 3 As shown:

[0092] Step 1041 : input the state time series information into a second preset recognition model to obtain predicted state time series information within a target time period after the preset time length output by the second preset recognition model.

[0093] The second preset recognition model is pre-trained in the following manner:

[0094] Acquire multiple time series state sample information corresponding to multiple traffic light intersections within a historical time period, where the time series state sample information includes state annotation data within a target time period; use the multiple time series state sample information as second training data to perform model training on the second initial model to obtain the second preset recognition model.

[0095] It should be noted that the second initial model can be a Transformer model, such as Figure 4 As shown, Figure 4 This is a schematic diagram of a model structure shown in an exemplary embodiment of the present disclosure. Figure 4 As shown, the Transformer model may include an Encoder and a Dncoder. The Transformer model may be used to predict time series, and the state annotation data within the target time period may include the traffic light state annotation data at each time point within the target time period.

[0096] Step 1042: Determine the first traffic light signal state corresponding to the current time point according to the predicted state timing information.

[0097] Among them, the predicted state timing information is used to characterize the traffic light state at each time point after the preset time length. After determining the predicted state timing information after the preset time length in the historical time, the predicted state timing information includes the predicted traffic light state at each time point after the preset time length in the historical time. The current time point is one of multiple time points after the preset time length, so the first traffic signal state corresponding to the current time point can be determined based on the predicted state timing information.

[0098] For example, Figure 2 Taking the state timing shown as an example, if the timing state information of the preset duration is the part before t2 in the figure, the predicted state timing information is the part after t2, and if the current time point is t5, the first traffic light signal state corresponding to the current time point t5 can be determined from the predicted state timing information.

[0099] Step 1043: Acquire the second traffic light signal state in the target traffic light image.

[0100] In this step, the traffic light state in the target traffic light image may be identified by an image recognition algorithm, that is, the second traffic light signal state is obtained.

[0101] Step 1044: Determine the third traffic light signal state according to the driving state information of the vehicle ahead.

[0102] In this step, when the forward driving speed in the driving status information of the vehicle ahead is greater than a preset speed threshold, the third traffic light signal state can be determined to be a state where the green light is on and the yellow and red lights are off; when the forward driving speed in the driving status information of the vehicle ahead is greater than zero but less than or equal to the preset speed threshold, the third traffic light signal state can be determined to be a state where the yellow light is on and the green and red lights are off; and when the forward driving speed in the driving status information of the vehicle ahead is zero, the third traffic light signal state can be determined to be a state where the red light is on and the green and yellow lights are off.

[0103] Step 1045 : Determine the target traffic light state according to the first traffic light signal state, the second traffic light signal state, and the third traffic light signal state.

[0104] In this step, the target weights of the first traffic light signal state, the second traffic light signal state, and the third traffic light signal state can be obtained; and the target traffic light state is determined according to the first traffic light signal state, the second traffic light signal state, and the third traffic light signal state through the target weights.

[0105] For example, if 1 is used to represent the state where the green light is on and the red and yellow lights are off, 0 is used to represent the state where the red light is on and the yellow and green lights are off, and 0 is used to represent the state where the yellow light is on and the red and green lights are off; after obtaining the target weight of the first traffic light signal state as q1, the target weight of the second traffic light signal state as q2, and the target weight of the third traffic light signal state as q3, the first traffic light signal state, the second traffic light signal state, and the third traffic light signal are weighted and summed according to q1, q2, and q3 to obtain a result value of the weighted summation. If the result value is greater than or equal to a preset threshold, it is determined that the target traffic light state is the state where the green light is on and the red and yellow lights are off. If the result value is less than the preset threshold, it is determined that the target traffic light state is a state other than the green light, which may be the state where the red light is on and the yellow and green lights are off, or the state where the yellow light is on and the red and green lights are off.

[0106] The above technical solution collects multiple frames of traffic light status images within a preset time period through the image acquisition device on the vehicle; determines the status timing information of the traffic light ahead within the preset time period based on the multiple frames of traffic light status images; determines the current target traffic light status based on the status timing information, the target traffic light image at the current moment and the driving status information of the vehicle ahead, which can effectively improve the accuracy of traffic light status recognition results, reduce the probability of unmanned vehicles violating traffic regulations, thereby effectively improving the driving safety performance of unmanned vehicles and helping to improve the user experience of unmanned vehicles.

[0107] Figure 5 is a block diagram of a traffic light state recognition device shown in an exemplary embodiment of the present disclosure; Figure 5 As shown, the traffic light state recognition device is applied to a vehicle and may include:

[0108] The first determining module 501 is configured to, when determining that the distance between the vehicle and the location of the traffic light ahead is less than or equal to a preset distance threshold, capture, by an image capture device on the vehicle, multiple frames of traffic light status images within a preset time period;

[0109] A second determining module 502 is configured to determine, based on the multiple frames of traffic light state images, state timing information of the traffic light ahead within the preset duration, the state timing information including the traffic light state corresponding to each time point within the preset duration, the traffic light state including at least one of a red light on / off state, a yellow light on / off state, and a green light on / off state;

[0110] The acquisition module 503 is configured to acquire the target traffic light image at the current moment and the driving status information of the vehicle ahead of the vehicle;

[0111] The third determining module 504 is configured to determine the current target traffic light state according to the state timing information, the target traffic light image and the driving state information.

[0112] The above technical solution collects multiple frames of traffic light status images within a preset time period through the image acquisition device on the vehicle; determines the status timing information of the traffic light ahead within the preset time period based on the multiple frames of traffic light status images; determines the current target traffic light status based on the status timing information, the target traffic light image at the current moment and the driving status information of the vehicle ahead, which can effectively improve the accuracy of traffic light status recognition results, reduce the probability of unmanned vehicles violating traffic regulations, thereby effectively improving the driving safety performance of unmanned vehicles and helping to improve the user experience of unmanned vehicles.

[0113] Optionally, the second determining module 502 is configured to:

[0114] Identify the traffic light state in each frame of the traffic light state image to obtain the traffic light state corresponding to each image acquisition time point within the preset time period;

[0115] The traffic light state corresponding to each time point within the preset time period is determined according to the traffic light state corresponding to each image acquisition time point within the preset time period to obtain the state timing information.

[0116] Optionally, the third determining module 504 is configured to:

[0117] The state timing information, the target traffic light image and the driving state information are input into a first preset recognition model to obtain the target traffic light state output by the first preset recognition model.

[0118] Optionally, the first preset recognition model is pre-trained in the following manner:

[0119] Acquire multiple sets of historical sample data corresponding to multiple traffic light intersections, each set of the historical sample data including state time sequence information, historical traffic light images, and historical driving state information of the traffic light intersection within a preset historical time period;

[0120] The first initial model is trained using the multiple groups of historical sample data as first training data to obtain the first preset recognition model. The first training data includes historical traffic light status annotation data within a specified time period after the preset time period.

[0121] Optionally, the third determining module 504 is configured to:

[0122] Inputting the state time series information into a second preset recognition model to obtain predicted state time series information within a target time period after the preset time length output by the second preset recognition model;

[0123] Determine the first traffic light signal state corresponding to the current time point according to the predicted state time series information;

[0124] Obtaining a second traffic light signal state in the target traffic light image;

[0125] determining a third traffic light signal state according to the driving state information of the vehicle ahead;

[0126] The target traffic light state is determined according to the first traffic light signal state, the second traffic light signal state, and the third traffic light signal state.

[0127] Optionally, the third determining module 504 is configured to:

[0128] Obtaining target weights of the first traffic light signal state, the second traffic light signal state, and the third traffic light signal state;

[0129] The target traffic light state is determined according to the first traffic light signal state, the second traffic light signal state, and the third traffic light signal state through the target weight.

[0130] Optionally, the second preset recognition model is pre-trained in the following manner:

[0131] Obtain multiple time series state sample information corresponding to multiple traffic light intersections in a historical time period, where the time series state sample information includes state annotation data within a target time period;

[0132] The second initial model is trained using the plurality of time series state sample information as second training data to obtain the second preset recognition model.

[0133] The above technical solution collects multiple frames of traffic light status images within a preset time period through the image acquisition device on the vehicle; determines the status timing information of the traffic light ahead within the preset time period based on the multiple frames of traffic light status images; determines the current target traffic light status based on the status timing information, the target traffic light image at the current moment and the driving status information of the vehicle ahead, which can effectively improve the accuracy of traffic light status recognition results, reduce the probability of unmanned vehicles violating traffic regulations, thereby effectively improving the driving safety performance of unmanned vehicles and helping to improve the user experience of unmanned vehicles.

[0134] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0135] See Figure 6 , Figure 6 6 is a functional block diagram of a vehicle, illustrating an exemplary embodiment. Vehicle 600 can be configured for either fully or partially autonomous driving. For example, vehicle 600 can obtain environmental information about its surroundings through perception system 620 and, based on analysis of the environmental information, derive an autonomous driving strategy to achieve fully autonomous driving, or present the analysis results to the user to achieve partially autonomous driving.

[0136] Vehicle 600 may include various subsystems, such as an infotainment system 610, a perception system 620, a decision control system 630, a drive system 640, and a computing platform 650. Alternatively, vehicle 600 may include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of vehicle 600 may be interconnected via wired or wireless means.

[0137] In some embodiments, infotainment system 610 may include a communication system 611 , an entertainment system 612 , and a navigation system 613 .

[0138] The communication system 611 may include a wireless communication system that can communicate wirelessly with one or more devices directly or via a communication network. For example, the wireless communication system can use 3G cellular communication, such as CDMA, EVDO, GSM / GPRS, or 4G cellular communication, such as LTE. Or 5G cellular communication. The wireless communication system can use WiFi to communicate with a wireless local area network (WLAN). In some embodiments, the wireless communication system can use an infrared link, Bluetooth, or ZigBee to communicate directly with the device. Other wireless protocols, such as various vehicle communication systems, for example, the wireless communication system may include one or more dedicated short range communications (DSRC) devices, which may include public and / or private data communications between vehicles and / or roadside stations.

[0139] The entertainment system 612 may include a display device, a microphone and speakers. Users can listen to the radio and play music in the car based on the entertainment system; or connect the mobile phone to the vehicle and project the mobile phone screen on the display device. The display device can be touch-sensitive and the user can operate it by touching the screen.

[0140] In some cases, the user's voice signal can be obtained through a microphone, and based on the analysis of the user's voice signal, the user can control certain aspects of the vehicle 600, such as adjusting the temperature inside the vehicle, etc. In other cases, music can be played to the user through a speaker.

[0141] The navigation system 613 may include a map service provided by a map provider, thereby providing navigation for the vehicle 600. The navigation system 613 may be used in conjunction with the vehicle's global positioning system 621 and inertial measurement unit 622. The map service provided by the map provider may be a two-dimensional map or a high-precision map.

[0142] The perception system 620 may include several sensors that sense information about the environment surrounding the vehicle 600. For example, the perception system 620 may include a global positioning system 621 (the global positioning system may be a GPS system, or a BeiDou system or other positioning system), an inertial measurement unit (IMU) 622, a lidar 623, a millimeter wave radar 624, an ultrasonic radar 625, and a camera 626. The perception system 620 may also include sensors of the internal systems of the monitored vehicle 600 (for example, an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, direction, speed, etc.). Such detection and recognition are key functions for the safe operation of the vehicle 600.

[0143] The global positioning system 621 is used to estimate the geographic location of the vehicle 600 .

[0144] The inertial measurement unit 622 is used to sense the posture change of the vehicle 600 based on inertial acceleration. In some embodiments, the inertial measurement unit 622 can be a combination of an accelerometer and a gyroscope.

[0145] LiDAR 623 utilizes laser light to sense objects in the environment in which vehicle 600 is located. In some embodiments, LiDAR 623 may include one or more laser sources, a laser scanner, and one or more detectors, among other system components.

[0146] The millimeter wave radar 624 uses radio signals to sense objects in the surrounding environment of the vehicle 600. In some embodiments, in addition to sensing objects, the millimeter wave radar 624 can also be used to sense the speed and / or heading of the objects.

[0147] The ultrasonic radar 625 may sense objects around the vehicle 600 using ultrasonic signals.

[0148] The camera device 626 is used to capture image information of the surrounding environment of the vehicle 600. The camera device 626 may include a monocular camera, a binocular camera, a structured light camera, a panoramic camera, etc. The image information obtained by the camera device 626 may include static images or video stream information.

[0149] The decision control system 630 includes a computing system 631 that analyzes and makes decisions based on the information obtained by the perception system 620. The decision control system 630 also includes a vehicle controller 632 that controls the power system of the vehicle 600, as well as a steering system 633, throttle 634 and braking system 635 for controlling the vehicle 600.

[0150] The computing system 631 can be operated to process and analyze various information obtained by the perception system 620 in order to identify targets, objects and / or features in the environment surrounding the vehicle 600. Targets may include pedestrians or animals, and objects and / or features may include traffic signals, road boundaries and obstacles. The computing system 631 may use object recognition algorithms, Structure from Motion (SFM) algorithms, video tracking and other technologies. In some embodiments, the computing system 631 can be used to map the environment, track objects, estimate the speed of objects, and so on. The computing system 631 can analyze the various information obtained and derive a control strategy for the vehicle.

[0151] The vehicle controller 632 can be used to coordinate and control the vehicle's power battery and engine 641 to improve the power performance of the vehicle 600.

[0152] The steering system 633 is operable to adjust the forward direction of the vehicle 600. For example, in one embodiment, it may be a steering wheel system.

[0153] The throttle 634 is used to control the operating speed of the engine 641 and thereby control the speed of the vehicle 600 .

[0154] Braking system 635 is used to control the deceleration of vehicle 600. Braking system 635 can use friction to slow down wheels 644. In some embodiments, braking system 635 can convert the kinetic energy of wheels 644 into electric current. Braking system 635 can also take other forms to slow the rotation speed of wheels 644 and thus control the speed of vehicle 600.

[0155] Drive system 640 may include components that provide powered motion for vehicle 600. In one embodiment, drive system 640 may include an engine 641, an energy source 642, a transmission system 643, and wheels 644. Engine 641 may be an internal combustion engine, an electric motor, an air compression engine, or another combination of engines, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air compression engine. Engine 641 converts energy source 642 into mechanical energy.

[0156] Examples of energy source 642 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 642 can also provide energy to other systems of vehicle 600.

[0157] The transmission system 643 can transmit mechanical power from the engine 641 to the wheels 644. The transmission system 643 may include a gearbox, a differential, and a drive shaft. In one embodiment, the transmission system 643 may also include other components, such as a clutch. The drive shaft may include one or more shafts that can be coupled to one or more wheels 644.

[0158] Some or all functions of vehicle 600 are controlled by a computing platform 650. Computing platform 650 may include at least one processor 651 that can execute instructions 653 stored in a non-transitory computer-readable medium such as memory 652. In some embodiments, computing platform 650 may also be a plurality of computing devices that control individual components or subsystems of vehicle 600 in a distributed manner.

[0159] The processor 651 may be any conventional processor, such as a commercially available CPU. Alternatively, the processor 651 may include a graphics processor (GPU), a field programmable gate array (FPGA), a system on chip (SOC), an application specific integrated circuit (ASIC), or a combination thereof. Figure 6Functionally, processor, memory, and other elements of the computer in the same block are illustrated, but those of ordinary skill in the art will appreciate that the processor, computer, or memory may in fact comprise a plurality of processors, computers, or memories that may or may not be stored in the same physical housing. For example, the memory may be a hard drive or other storage medium that is positioned in a housing that is different from the computer. Therefore, reference to a processor or computer will be understood to include reference to a collection of processors or computers or memories that may or may not operate in parallel. Different from using a single processor to perform the steps described herein, some assemblies such as steering assembly and deceleration assembly may each have their own processor that only performs the calculations relevant to the functions specific to the assembly.

[0160] In the embodiment of the present disclosure, the processor 651 may execute the above-mentioned traffic light state recognition method.

[0161] In various aspects described herein, the processor 651 can be located remotely from the vehicle and in wireless communication with the vehicle. In other aspects, some of the processes described herein are performed on a processor disposed within the vehicle while others are performed by a remote processor, including taking the necessary steps to perform a single maneuver.

[0162] In some embodiments, the memory 652 may include instructions 653 (e.g., program logic) that are executable by the processor 651 to perform various functions of the vehicle 600. The memory 652 may also include additional instructions, including instructions for sending data to, receiving data from, interacting with, and / or controlling one or more of the infotainment system 610, the perception system 620, the decision control system 630, and the drive system 640.

[0163] In addition to instructions 653, memory 652 may also store data such as road maps, route information, the vehicle's location, direction, speed, and other such vehicle data, as well as other information. This information may be used by vehicle 600 and computing platform 650 during operation of vehicle 600 in autonomous, semi-autonomous, and / or manual modes.

[0164] The computing platform 650 may control functions of the vehicle 600 based on input received from various subsystems, such as the drive system 640, the perception system 620, and the decision control system 630. For example, the computing platform 650 may utilize input from the decision control system 630 to control the steering system 633 to avoid an obstacle detected by the perception system 620. In some embodiments, the computing platform 650 may be operable to provide control over many aspects of the vehicle 600 and its subsystems.

[0165] Alternatively, one or more of the above components may be installed or associated separately from the vehicle 600. For example, the memory 652 may be partially or completely separate from the vehicle 600. The above components may be communicatively coupled together in a wired and / or wireless manner.

[0166] Optionally, the above components are just an example. In actual applications, the components in the above modules may be added or deleted according to actual needs. Figure 6 It should not be understood as limiting the embodiments of the present disclosure.

[0167] An autonomous vehicle traveling on a road, such as vehicle 600 above, can identify objects in its surroundings to determine adjustments to its current speed. Objects can be other vehicles, traffic control devices, or other types of objects. In some examples, each identified object can be considered independently, and the speed adjustment to be made to the autonomous vehicle can be determined based on its respective characteristics, such as its current speed, acceleration, and distance from the vehicle.

[0168] Optionally, the vehicle 600 or a sensing and computing device associated with the vehicle 600 (e.g., computing system 631, computing platform 650) can predict the behavior of the identified objects based on the characteristics of the identified objects and the state of the surrounding environment (e.g., traffic, rain, ice on the road, etc.). Optionally, each identified object depends on the behavior of each other, so the behavior of all identified objects can also be considered together to predict the behavior of a single identified object. The vehicle 600 can adjust its speed based on the predicted behavior of the identified objects. In other words, the autonomous vehicle can determine what stable state the vehicle will need to adjust to (e.g., accelerate, decelerate, or stop) based on the predicted behavior of the objects. In this process, other factors can also be considered to determine the speed of the vehicle 600, such as the lateral position of the vehicle 600 in the road it is traveling on, the curvature of the road, the proximity of static and dynamic objects, etc.

[0169] In addition to providing instructions to adjust the speed of the autonomous vehicle, the computing device may also provide instructions to modify the steering angle of vehicle 600 so that the autonomous vehicle follows a given trajectory and / or maintains a safe lateral and longitudinal distance from objects near the autonomous vehicle (e.g., vehicles in adjacent lanes on the road).

[0170] The vehicle 600 may be any type of vehicle, such as a car, a truck, a motorcycle, a bus, a ship, an airplane, a helicopter, an RV, a train, etc., and the present disclosure does not impose any particular limitation thereto.

[0171] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program executable by a programmable device, and the computer program has a code portion for executing the above-mentioned traffic light state recognition method when executed by the programmable device.

[0172] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the present disclosure. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0173] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for identifying traffic light status, characterized in that: Applied to a vehicle, the method comprises: When it is determined that the distance between the vehicle and the location of the traffic light ahead is less than or equal to a preset distance threshold, capturing a plurality of frames of traffic light status images within a preset time period by an image capture device on the vehicle; Determining, based on the multiple frames of traffic light state images, state timing information of the traffic light ahead within the preset duration, the state timing information including the traffic light state corresponding to each time point within the preset duration, the traffic light state including at least one of a red light on / off state, a yellow light on / off state, and a green light on / off state; Collecting a target traffic light image at the current moment and driving status information of the vehicle ahead of the vehicle; determining a current target traffic light state according to the state timing information, the target traffic light image, and the driving state information; The determining the current target traffic light state according to the state timing information, the target traffic light image, and the driving state information includes: The state timing information, the target traffic light image and the driving state information are input into a first preset recognition model to obtain the target traffic light state output by the first preset recognition model.

2. The method according to claim 1, characterized in that The determining, based on the multiple frames of traffic light status images, the status timing information of the traffic light ahead within the preset time period includes: Identifying the traffic light state in each frame of the traffic light state image to obtain the traffic light state corresponding to each image acquisition time point within the preset time period; The traffic light state corresponding to each time point within the preset time period is determined according to the traffic light state corresponding to each image acquisition time point within the preset time period to obtain the state timing information.

3. The method according to claim 1, characterized in that The first preset recognition model is pre-trained in the following manner: Acquire multiple sets of historical sample data corresponding to multiple traffic light intersections, each set of the historical sample data including state time sequence information, historical traffic light images, and historical driving state information of the traffic light intersection within the historical preset time period; The first initial model is trained using the multiple groups of historical sample data as the first training data to obtain the first preset recognition model, wherein the first training data includes the traffic light status annotation data within a specified time period after the preset time period in history.

4. The method according to claim 1, wherein The determining of the current target traffic light state according to the state timing information, the target traffic light image and the driving state information further includes: Inputting the state time series information into a second preset recognition model to obtain predicted state time series information within a target time period after the preset time length output by the second preset recognition model; Determining a first traffic light signal state corresponding to a current time point according to the predicted state time series information; Acquiring a second traffic light signal state in the target traffic light image; determining a third traffic light signal state according to the driving state information of the preceding vehicle; The target traffic light state is determined according to the first traffic light signal state, the second traffic light signal state, and the third traffic light signal state.

5. The method according to claim 4, characterized in that The determining the target traffic light state according to the first traffic light signal state, the second traffic light signal state, and the third traffic light signal state includes: Obtaining target weights of the first traffic light signal state, the second traffic light signal state, and the third traffic light signal state; The target traffic light state is determined according to the first traffic light signal state, the second traffic light signal state, and the third traffic light signal state using the target weight.

6. The method according to claim 4, characterized in that The second preset recognition model is pre-trained in the following manner: Acquire multiple time series state sample information corresponding to multiple traffic light intersections in a historical time period, wherein the time series state sample information includes state annotation data within a target time period; The second initial model is trained using the plurality of time series state sample information as second training data to obtain the second preset recognition model.

7. A traffic light status recognition device, characterized in that: Applied to a vehicle, the device comprises: The first determining module is configured to, when determining that the distance between the vehicle and the location of the traffic light ahead is less than or equal to a preset distance threshold, capture, by an image capture device on the vehicle, a plurality of frames of traffic light status images within a preset time period; a second determining module configured to determine, based on the multiple frames of traffic light state images, state timing information of the traffic light ahead within the preset duration, the state timing information including the traffic light state corresponding to each time point within the preset duration, the traffic light state including at least one of a red light on / off state, a yellow light on / off state, and a green light on / off state; an acquisition module configured to acquire a target traffic light image at a current moment and driving status information of a vehicle ahead of the vehicle; a third determining module, configured to determine a current target traffic light state according to the state timing information, the target traffic light image, and the driving state information; The third determination module is configured to input the state timing information, the target traffic light image and the driving state information into a first preset recognition model to obtain the target traffic light state output by the first preset recognition model.

8. A traffic light status recognition vehicle, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: Implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images