Traffic light detection method, device and electronic equipment

Abstract

The present disclosure provides a traffic light detection method, device and electronic device, which relate to the field of artificial intelligence, especially to the field of autonomous driving and intelligent transportation. The specific implementation scheme is: the server can identify the display data of the traffic light at each moment based on the video collected by the unmanned vehicle during the entire driving stage, that is, the third time series data, and the server can detect whether the traffic light display is correct based on the first time series data from the traffic light, the second time series data obtained by the unmanned vehicle based on the real-time video picture collected during driving, and the third time series data. It can be regarded as the server performing two tests on the display data of the traffic light, which can improve the accuracy and efficiency of traffic light detection.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to a traffic light detection method, device, and electronic device, which can be used in the fields of autonomous driving and intelligent transportation. Background Art

[0002] In the field of autonomous driving, unmanned vehicles can identify the color and countdown of traffic lights at intersections while driving, and then accurately control driving operations based on the identified information. The accuracy of traffic lights has a significant impact on the driving safety of unmanned vehicles. If the color of traffic lights is inaccurate, it will cause unmanned vehicles to make wrong driving operations, thus causing traffic accidents.

[0003] Therefore, there is an urgent need for a traffic light detection method to detect whether the display of the traffic light is correct. Summary of the invention

[0004] The present disclosure provides a traffic light detection method, device and electronic device, which can detect whether the traffic light display is correct.

[0005] According to the first aspect of the present disclosure, a traffic light detection method is provided. The method includes: receiving first time series data, second time series data, and video from an unmanned vehicle, the video being a video collected by the unmanned vehicle during the entire driving stage, the first time series data being: the display data from the traffic light received by the unmanned vehicle during the entire driving stage, the second time series data being: the display data of the traffic light obtained by the unmanned vehicle based on the video; obtaining third time series data according to the video, the third time series data including the display data of the traffic light; obtaining a detection result of whether the display of the traffic light is correct according to the first time series data, the second time series data, and the third time series data; and outputting the detection result.

[0006] In the disclosed embodiment, the server can detect whether the traffic light display is correct based on the first time series data from the traffic light, the second time series data obtained based on the real-time video images collected during the driving of the unmanned vehicle, and the third time series data obtained based on the video. This can be regarded as the server performing two tests on the display data of the traffic light, which can improve the accuracy of the traffic light detection.

[0007] According to the second aspect of the present disclosure, a traffic light detection method is provided. The method includes: receiving display data of a traffic light from a roadside device during driving, wherein the display data of the traffic light in the roadside device comes from the traffic light; collecting video images and the position information of the unmanned vehicle; mapping the time of receiving the display data with the display data to obtain first time series data; obtaining second time series data according to the video images and position information collected at different times; and sending the first time series data, the second time series data, and the video collected by the unmanned vehicle during the entire driving stage to a service end at the end of driving.

[0008] According to a third aspect of the present disclosure, a traffic light detection device is provided, wherein a transceiver unit is used to receive first time series data, second time series data, and a video from an unmanned vehicle, wherein the video is a video collected by the unmanned vehicle during the entire driving stage, and the first time series data is: the display data of the traffic light received by the unmanned vehicle during the entire driving stage, and the second time series data is: the display data of the traffic light obtained by the unmanned vehicle based on the video; a processing unit is used to obtain third time series data according to the video, and the third time series data includes the display data of the traffic light; and according to the first time series data, the second time series data, and the third time series data, obtain a detection result of whether the traffic light display is correct; an output unit is used to output the detection result.

[0009] According to a fourth aspect of the present disclosure, a traffic light detection device is provided, wherein, during driving, a transceiver unit is used to receive display data of a traffic light from a roadside device, and the display data of the traffic light in the roadside device comes from the traffic light; a shooting unit is used to capture video images; and a positioning unit is used to capture and position information of an unmanned vehicle; a processing unit is used to map the time of receiving the display data with the display data to obtain first time series data; and, according to the video images and position information collected at different times, obtain second time series data; at the end of driving, the transceiver unit is also used to send the first time series data, the second time series data, and the video collected by the unmanned vehicle during the entire driving stage to a server.

[0010] According to the fifth aspect of the present disclosure, an electronic device is provided, which may be the above-mentioned server or a vehicle-mounted terminal in an unmanned vehicle, and the electronic device includes: at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the electronic device executes the above-mentioned first and second aspects of the method.

[0011] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the methods of the first and second aspects described above are implemented.

[0012] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, and at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect and the second aspect.

[0013] According to the technology disclosed in the present invention, the problem of driving safety caused by incorrect traffic lights is solved. The server compares and verifies the first time series data from the traffic light, the second time series data obtained based on the real-time video images collected by the unmanned vehicle during driving, and the third time series data obtained based on the video, to detect whether the traffic light display is correct. It can be regarded as the server performing two tests on the display data of the traffic light, thereby improving the accuracy of the traffic light detection.

[0014] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0016] Figure 1 A schematic diagram of a scenario applicable to the embodiment of the present disclosure;

[0017] Figure 2 A schematic diagram of the structure of an unmanned vehicle in an embodiment of the present disclosure;

[0018] Figure 3 A flow chart of an embodiment of a traffic light detection method provided by an embodiment of the present disclosure;

[0019] Figure 4 A schematic diagram of time series data provided by an embodiment of the present disclosure;

[0020] Figure 5 A flow chart of another embodiment of the traffic light detection method provided by the embodiment of the present disclosure;

[0021] Figure 6 A schematic diagram of video processing provided by an embodiment of the present disclosure;

[0022] Figure 7A schematic diagram of obtaining a capture time of a target video frame provided in an embodiment of the present disclosure;

[0023] Fig. 8A A schematic diagram of a video processing flow provided by an embodiment of the present disclosure;

[0024] Figure 8B A schematic diagram of detecting a traffic light based on time series data provided by an embodiment of the present disclosure;

[0025] Figure 8C Another schematic diagram of detecting a traffic light based on time series data provided by an embodiment of the present disclosure;

[0026] Fig.8D Another schematic diagram of detecting a traffic light based on time series data provided by an embodiment of the present disclosure;

[0027] Fig. 8E Another schematic diagram of detecting a traffic light based on time series data provided by an embodiment of the present disclosure;

[0028] Fig. 9 A flow chart of another embodiment of the traffic light detection method provided by the embodiment of the present disclosure;

[0029] Fig.10 A schematic diagram of an unmanned vehicle acquiring time series data provided by an embodiment of the present disclosure;

[0030] Fig.11 A schematic diagram of an unmanned vehicle in a driving process provided by an embodiment of the present disclosure;

[0031] Fig.12 A flow chart of another embodiment of the traffic light detection method provided by the embodiment of the present disclosure;

[0032] Fig.13 A schematic diagram of the structure of a traffic light detection device provided in an embodiment of the present disclosure;

[0033] Fig.14 Another structural schematic diagram of a traffic light detection device provided in an embodiment of the present disclosure;

[0034] Fig.15 A schematic diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0036] A traffic light detection method is urgently needed in the prior art to detect whether the display of the traffic light is correct, so as to ensure the driving safety of the unmanned vehicle. At present, the traffic light can be detected by manually detecting whether the display of the traffic light is correct, but this method has the problem of low efficiency.

[0037] In the field of unmanned driving, the traffic lights set on the road are vehicle to X (V2X) traffic lights. V2X traffic lights can interact with vehicles, and vehicles can obtain the display data of V2X traffic lights. Therefore, the embodiment of the present disclosure provides a traffic light detection method, which is applied in the field of artificial intelligence, and is specifically used in the field of autonomous driving and intelligent transportation. The method detects whether the display of the V2X traffic light is correct by comparing the display data from the V2X traffic light and the display data of the V2X traffic light recognized by the vehicle based on the collected video, so as to improve the detection efficiency. In order to improve the detection accuracy of the traffic light in the embodiment of the present disclosure, after the vehicle collects the video, the display data of the traffic light can be identified by a third-party device based on the video collected by the vehicle to further detect whether the display of the traffic light is accurate, so as to improve the detection accuracy. In addition, when a traffic light display error is detected, the user can be promptly prompted to inspect the traffic light to ensure the accurate display of the traffic light.

[0038] In one embodiment, the display data of the traffic light may include information such as the light color of the traffic light and the countdown of the traffic light.

[0039] Figure 1 Schematic diagram of a scenario applicable to the embodiment of the present disclosure. Figure 1 The scene may include: traffic lights 11, road side units (RSU) 12, vehicles 13, and third-party equipment 14.

[0040] The traffic light 11 in the embodiment of the present disclosure is a V2X traffic light. The V2X traffic light 11 can communicate with the roadside device 12. For example, the V2X traffic light 11 can send the display data of the V2X traffic light to the roadside device 12. It should be understood that the traffic light described in the following embodiments refers to a V2X traffic light. In one embodiment, a communication unit can be provided in both the traffic light 11 and the roadside device 12 to realize the communication between the traffic light 11 and the roadside device 12.

[0041] The roadside device 12 may be, for example, a street lamp, a signal pole for installing a traffic light, or a device installed on a signal pole, or a camera installed on the roadside, etc. The embodiment of the present disclosure does not limit the form of the roadside device. The roadside device 12 may communicate with the vehicle 13, such as the roadside device 12 may send the display data of the traffic light to the vehicle 13.

[0042] The vehicle 13 may be an unmanned vehicle or a non-unmanned vehicle. The following embodiments are described using an unmanned vehicle as an example. The vehicle 13 may receive traffic light display data from the roadside device 13. In one embodiment, the unmanned vehicle may be understood as a traffic light detection vehicle, which is specifically used to detect whether the traffic light display is correct.

[0043] The third-party device 14 can be a driverless service end, a traffic light service end, or a separately deployed device, etc. The service end can be an electronic device such as a server or a computer. The disclosed embodiment does not limit the form of the third-party device. The third-party device 14 can identify the display data of the traffic light based on the video collected by the vehicle 13, and then detect whether the display of the traffic light 11 is correct. In the following embodiments, the third-party device 14 is represented by a service end.

[0044] Figure 2 FIG. 1 is a schematic diagram of a structure of an unmanned vehicle in an embodiment of the present disclosure. Figure 2, the unmanned vehicle 13 may include: an on-board communication unit (OBU) 131, a positioning device 132, a shooting device 133, a processing device 134, and a storage device 135. Among them, OBU131 is used to realize the communication between the unmanned vehicle 13 and the roadside equipment 12. The positioning device 132 may include but is not limited to: a global navigation satellite system (GNSS), a global positioning system (GPS), etc. The positioning device 132 is used to realize high-precision positioning of the unmanned vehicle 13. The shooting device 133 may include but is not limited to a camera, a video recorder, etc. The shooting device 133 is used to collect video images of the unmanned vehicle 13 during driving to obtain a video. The processing device 134 may include but is not limited to a module with computing functions such as a processor and a chip. The processing device 134 is used to identify and obtain the display data of the traffic light based on the video image. The storage device 135 is used to store the video and the display data of the traffic light recognized by the unmanned vehicle 13.

[0045] The specific actions performed by the unmanned vehicle 13 can refer to the following embodiments. Fig.10 Related description.

[0046] The traffic light detection method provided by the embodiment of the present disclosure is described below in conjunction with specific embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0047] Figure 3 A flow chart of an embodiment of a traffic light detection method provided by the present disclosure. Figure 3 The traffic light detection method provided by the embodiment of the present disclosure may include:

[0048] S301, receiving the first time series data, the second time series data, and the video from the unmanned vehicle, where the video is the video collected by the unmanned vehicle during the entire driving stage. The first time series data is: the display data from the traffic light received by the unmanned vehicle during the entire driving stage, and the second time series data is: the display data of the traffic light obtained by the unmanned vehicle based on the video.

[0049] After the unmanned vehicle completes the driving, it can send the first time series data, the second time series data, and the video to the server, wherein the video is the video collected by the unmanned vehicle during the entire driving stage.

[0050] It should be understood that time series data refers to data columns recorded in chronological order.

[0051] Among them, the first time series data is the display data received from the traffic light by the unmanned vehicle during the entire driving stage. Exemplarily, during the driving process, the unmanned vehicle can receive the display data of the traffic light from the roadside equipment, and the display data of the traffic light of the roadside equipment is sent by the traffic light. In one embodiment, there can be multiple traffic lights on the road, and when the traffic light sends the display data of the traffic light to the roadside equipment, it can carry the identification of the sending traffic light. In this way, when the unmanned vehicle receives the display data of the traffic light from the roadside equipment, it can determine which traffic light the display data is, so that the unmanned vehicle can merge the display data of each traffic light to obtain the first time series data. The first time series data includes: the display data of each traffic light at each moment. The processing process of the unmanned vehicle can refer to the following Fig.10 With reference to the related description in , in the embodiment of the present disclosure, the processing process of “each traffic light” is represented by the processing process of the traffic light.

[0052] The second time series data is the display data of the traffic lights obtained by the unmanned vehicle based on the video during the entire driving stage. It should be understood that the unmanned vehicle can collect video images in real time during driving, and the unmanned vehicle can obtain the display data of the traffic lights in the video images in real time based on the real-time collected video images, so that the unmanned vehicle can obtain the second time series data. The processing process of the unmanned vehicle can refer to the following Fig.10 Related description in .

[0053] Figure 4 A schematic diagram of time series data provided by an embodiment of the present disclosure. Figure 4 In a in , the time series data (such as a period of time) of a traffic light (such as traffic light 1) is used as an example for explanation. The first time series data is: the display data of traffic light 1 received by the unmanned vehicle in real time from traffic light 1, and the second time series data is: the display data of traffic light 1 obtained by the unmanned vehicle based on the video images collected in real time during driving.

[0054] S302, obtaining third time series data according to the video, where the third time series data includes display data of the traffic light.

[0055] The video collected by the unmanned vehicle during the entire driving phase includes: video images at each collection time. In one embodiment, the server can identify the first video frame (first video image) containing the traffic light in the video, and then obtain the display data of the traffic light in the first video frame based on the light color and countdown of the traffic light in the first video frame, and the collection time of the first video frame, that is, the third time series data.

[0056] In one embodiment, if the server can store pictures of traffic lights, the server can compare frame by frame in the video to detect whether the video contains a traffic light. Exemplarily, the video frame can contain multiple objects, such as vehicles, trees, buildings, etc. The server can use the video frame whose similarity between the contained object and the traffic light is greater than a preset similarity (such as 90%) as the first video frame, that is, the first video frame contains the traffic light.

[0057] For example, Figure 4 In a of FIG. 1 , the third time series data obtained based on the video is shown. In one embodiment, because the server can identify the light color and countdown of the traffic light, but cannot distinguish which traffic light it is, the server can align the first time series data and the second time series data based on the acquisition time of the first video frame, such as Figure 4 As shown in b in FIG. 1 , the server can obtain the display data of traffic light 1 based on the collection time.

[0058] S303, obtaining a detection result of whether the traffic light display is correct according to the first time series data, the second time series data, and the third time series data.

[0059] In the disclosed embodiment, the server can compare the first time series data, the second time series data, and the third time series data to detect whether the traffic light display is correct, and then obtain the detection result of whether the traffic light display is correct. Because the second time series data is the display data of the traffic light obtained by the unmanned vehicle based on the video screen collected in real time, and the server also obtains the display data of the traffic light based on the video screen collected by the unmanned vehicle, that is, the display data of the traffic light is obtained based on the secondary calculation of the video. Then, when the server compares the first time series data, the second time series data, and the third time series data, it can be regarded as performing two tests on the display data of the traffic light, thereby improving the accuracy of the traffic light detection.

[0060] In one embodiment, in the first time series data, the second time series data, and the third time series data, if the display data of the traffic light at each moment (such as light color, countdown), etc. are the same, it is determined that the display of the traffic light is correct. In one embodiment, if the display data of the traffic light at the target moment is different in the second time series data and the third time series data, it indicates that the display data recognized by the unmanned vehicle and the service end at the target moment are different, and the accuracy of the display data at the target moment is poor, so the display data at the target moment in the second time series data and the third time series data can be regarded as invalid data, the video at the target moment needs to be re-collected, and the display data at the target moment needs to be calculated. In one embodiment, if the display data of the traffic light at each moment in the second time series data and the third time series data are the same, but the display data of the traffic light at the target moment in the first time series data is different from the display data of the target moment in the second time series data and the third time series data, it can be determined that the display of the traffic light at the target moment is wrong.

[0061] S304, output the detection result.

[0062] After obtaining the test results, the server can output the test results, so that the staff can promptly determine whether the display of the traffic light is correct based on the test results, and then adjust the traffic light in time when the traffic light display is wrong, so that the display data of the adjusted traffic light is consistent with the display data of the traffic in the second time series data and the third time series data, thereby improving driving safety.

[0063] For example, Figure 1 As shown, the server can display the detection result on the server interface, such as the detection result including: the identification of the traffic light (such as traffic light 1), and the result of whether the traffic light display is correct. Figure 1 The following takes the display data error of traffic light 1 at the 10th second as an example for explanation. For example, the interface of the server displays: traffic light 1, 10th second, wrong light color (should be red light, but displayed as green light).

[0064] In one embodiment, the server may also send the test result to other devices with display functions so that the device can display the test result. In the disclosed embodiment, there is no restriction on the way the server outputs the test result, such as the server may also output the test result in voice.

[0065] In the method for traffic light detection provided by the embodiment of the present disclosure, the server can identify the display data of the traffic light at each moment based on the video collected by the unmanned vehicle during the entire driving stage, that is, the third time series data, and the server can detect whether the traffic light display is correct based on the display data from the traffic light (first time series data), the display data of the traffic light obtained by the unmanned vehicle based on the real-time collected video images during driving (second time series data), and the third time series data. It can be regarded as that the server performs two tests on the display data of the traffic light, such as detecting the display of the traffic light based on the display data of the traffic light identified by the unmanned vehicle and the display data of the traffic light identified by the server, which can improve the accuracy of traffic light detection.

[0066] In order to enable readers to more deeply understand the implementation principle of the present disclosure, the following Figure 5-8D right Figure 3 The illustrated embodiment is further refined.

[0067] Figure 5 A flow chart of another embodiment of the traffic light detection method provided by the present disclosure. Figure 5 , S302 in the above embodiment can be replaced by S302A:

[0068] S302A, identifying the traffic lights in the video according to the traffic light recognition model to obtain third time series data.

[0069] The server stores a pre-trained traffic light recognition model, which is used to recognize the color of the traffic light in the video, as well as information such as the countdown. The server can input the video into the traffic light recognition model to obtain the third time series data output by the traffic light recognition model. The traffic light recognition model and its training process are not described in the embodiments of the present disclosure, and reference can be made to the related description of the recognition model (such as face recognition) and the training process in the prior art.

[0070] In one embodiment, because the video carries the acquisition time of each video frame, the third time series data output by the traffic light recognition model may include: the acquisition time of the video frame containing the traffic light, and the display data of the traffic light corresponding to the video frame containing the traffic light.

[0071] If the video collected by the unmanned vehicle during the entire driving phase is long, if the server inputs all the videos into the traffic light recognition model at once, the traffic light recognition model needs to recognize and calculate the first frame to the last frame of the video frame by frame, which takes a long time and is inefficient. Therefore, in one embodiment, the server can slice the video and then input it into the traffic light recognition model, so that the traffic light recognition model can simultaneously recognize and calculate the multiple sliced ​​videos in parallel, which can improve the processing speed and efficiency.

[0072] The server may process the video into segments to obtain at least one video segment. Figure 6 For a 3-minute video, the server may divide the video into three 1-minute video segments. For each video segment, the server may extract frames for each video segment to obtain at least one video frame corresponding to the video segment. For example, for each 1-minute video segment, the server may extract frames every 600ms to obtain 100 video frames.

[0073] The server can input at least one video frame corresponding to each segment into the traffic light recognition model, so that the traffic light recognition model can concurrently identify at least one video frame corresponding to each segment, and then the display data of the traffic light corresponding to at least one target video frame. For example, the server can input 100 video frames corresponding to each 1-minute video segment into the traffic light recognition model, and the server can concurrently identify and calculate the 100 video frames corresponding to each 1-minute video segment. Among them, the video frame containing the traffic light is the target video frame, and the display data of the traffic light in at least one video frame can be obtained through recognition and calculation by the traffic light recognition model. Exemplary, Figure 6 As shown, the first video clip contains 80 target video frames, the second video clip contains 90 target video frames, and the third video clip contains 85 target video frames.

[0074] It should be understood that the display data of the traffic light corresponding to the target video frame can be understood as: the light color and countdown of the traffic light in the target video frame. In one embodiment, the target video frame is the same as the first video frame in S302 above, both of which are video frames containing traffic lights.

[0075] In one embodiment, when the server inputs at least one video frame corresponding to each video segment into the traffic light recognition model, the server may also input the acquisition time of each video frame corresponding to each video segment into the traffic light recognition model. In this embodiment, when the traffic light recognition model outputs the display data of the traffic light corresponding to the target video frame, it may also output the acquisition time of the target video frame. The server may map the acquisition time of the target video frame and the display data of the traffic light corresponding to the target video frame to obtain the third time series data.

[0076] In one embodiment, the server only inputs at least one video frame corresponding to each video clip into the traffic light recognition model, and the traffic light recognition model can output the display data of the traffic light corresponding to the target video frame, but cannot output the acquisition time of the target video frame. In this embodiment, the server can obtain the acquisition time of the target video frame to map the acquisition time of the target video frame and the display data of the traffic light corresponding to the target video frame, thereby obtaining the third time series data.

[0077] In this embodiment, the server can obtain the acquisition time of the target video frame according to the start and end time of the video, the order of the target video frame in the video segment to which it belongs, and the interval between two adjacent video frames when extracting frames. Figure 7 , take a target video frame in the first video clip as an example, the start time of the video is 10:00, and the end time is 10:03, that is, the start time of the first video clip is 10:00, the end time of the first video clip is 10:01, the target video frame is ranked third in the video frames of the first video clip, and the interval between two adjacent video frames when extracting frames is 600ms, then the server can calculate the acquisition time of the target frame as: 10:00+600ms×2=10:00:02. It should be understood that when the traffic light recognition model outputs the display data of the traffic light corresponding to the target video frame, it can also output the number of the target video frame in a group (at least one) of video frames, which is the ranking of the target video frame in the video clip to which it belongs, that is, the number of the video frame in the video clip.

[0078] In one embodiment, "the server segments the video and performs frame extraction on each video segment" can be called video preprocessing. "The server uses a traffic light recognition model to identify the display data of the traffic light" can be called AI recognition of the traffic light. "The server obtains the acquisition time of the target video frame, and maps the acquisition time of the target video frame and the display data of the traffic light corresponding to the target video frame" can be called "the display data of the traffic light corresponds to the acquisition time". Among them, the execution steps of the server in this embodiment can be simplified as follows: Fig. 8A shown.

[0079] In this way, in the embodiment of the present disclosure, the server can obtain the display data of the traffic light corresponding to each target video frame and the acquisition time of each target video frame. The server maps the acquisition time of the target video frame and the display data of the traffic light corresponding to the target video frame to obtain the third timing data.

[0080] In this embodiment, the server can obtain a detection result of whether the traffic light display is correct according to the first time series data, the second time series data, and the third time series data. When there are multiple traffic lights, the detection method of each traffic light is the same. Exemplarily, the multiple traffic lights include a target traffic light, or the traffic lights in the multiple traffic lights are all called target traffic lights.

[0081] The first time series data, the second time series data, and the third time series data all include: the light color of the target traffic light at different times. The first time series data and the second time series data also include the identification of the target traffic light. Because the third time series data cannot distinguish which traffic light is in the video frame, the third time series data may not include the identification of the traffic light. The identification of the target traffic light included in the first time series data and the second time series data may be: the number of the target traffic light, or the position of the target traffic light in the high-precision map, which can be referred to Fig.10 Correspondingly, the detection result includes the result of whether the target traffic light is displayed correctly.

[0082] The following uses the example of a traffic light including a target traffic light to explain how the server detects whether the display of the target traffic light is correct based on the first time series data, the second time series data, and the third time series data.

[0083] First, if the light colors of the target traffic light at the same time are the same in the first time series data, the second time series data, and the third time series data, the server can determine that the target traffic light is displayed correctly. Correspondingly, the detection result includes: the identification of the target traffic light, and information indicating that the target traffic light is displayed correctly. Among them, the information indicating that the target traffic light is displayed correctly can be: text or other forms indicating that the traffic light is displayed correctly, which is not limited by the embodiments of the present disclosure.

[0084] Reference Figure 8B , Figure 8B In the figure, white represents the green light, shadow 1 represents the yellow light, and shadow 2 represents the red light, taking the display data of the target traffic light over a period of time as an example. Figure 8B It can be seen that in the first time series data, the second time series data and the third time series data, the light color of the target traffic light at each same time is consistent (the consistent light color indicates that the countdown is also consistent), then the server can determine that the target traffic light is displayed correctly, such as the server can output the information "Traffic light 1, display correct".

[0085] Secondly, if the light color of the target traffic light at the same time is the same in the second time series data and the third time series data, but the light color of the target traffic light at the first target time in the first time series data is different from the light color of the target traffic light at the first target time in the second time series data and the third time series data, then the server can determine that the target traffic light displays an error at the first target time. Correspondingly, the detection result includes: the identification of the target traffic light, and information indicating that the target traffic light displays an error at the first target time.

[0086] Among them, the light color of the target traffic light at the first target moment is different from the light color of the target traffic light at the first target moment in the second time series data and the third time series data, which can be expressed in two forms. One is that the light color in the first time series data at the first target moment is red (example description), while the light color in the second time series data and the third time series data is green. The other is that the light color in the first time series data at the first target moment is no light color, while the light color in the second time series data and the third time series data is green. Among them, the light color of the first time series data at the first target moment is no light color, which can be understood as: the target traffic light did not send the display data of the traffic light to the unmanned vehicle through the roadside equipment at the first target moment, so the target traffic light displayed in the first time series data at the first target moment does not have information such as light color.

[0087] Reference Figure 8C a and b in Figure 8C a and b in the above respectively represent two forms of "the light color of the target traffic light at the first target time is different from the light color of the target traffic light at the first target time in the second time series data and the third time series data". Figure 8C The a in the figure indicates that at the first target moment, the light color in the first time series data is red, while the light colors in the second time series data and the third time series data are green. In this example, the server can output a detection result such as "the light color of traffic light 1 at 10:01 is wrong".

[0088] Figure 8C b in the figure indicates that at the first target moment, the light color in the first time series data is off, while the light colors in the second time series data and the third time series data are green. In this example, the server can output a detection result such as "traffic light 1 is off at 10:01".

[0089] In one embodiment, the first target time may be a portion of the entire driving phase of the unmanned vehicle, such as Figure 8C In one embodiment, the first target time may also be the duration of the entire driving phase of the unmanned vehicle, that is, the display data of the target traffic light during the entire driving phase of the unmanned vehicle are all wrong.

[0090] Third, if the light colors of the target traffic light at the second target moment are different in the second time series data and the third time series data, it is determined that the data at the second target moment in the second time series data and the third time series data are unavailable. In this embodiment, the server can output a prompt message. The prompt message is used to indicate that the unmanned vehicle needs to re-collect the data at the second target moment, so that the unmanned vehicle and the server can re-identify and calculate the display data of the traffic light at the second target moment.

[0091] Reference Fig.8D a and b in Fig.8D a in the figure shows that in the second time series data and the third time series data, the light colors of the target traffic lights at the same time are different, indicating that the light color of the traffic light recognized by the unmanned vehicle is inconsistent with the light color of the traffic light recognized by the service end. The second time series data and the third time series data are unavailable, and the unmanned vehicle needs to re-collect the data of the second target moment (a period of time of the traffic) so that the unmanned vehicle and the service end can re-recognize and calculate the display data of the traffic light at the second target moment.

[0092] exist Fig.8D In a, although the light colors of the traffic lights at the same time are consistent in the first time series data and the second time series data, the second time series data and the third time series data are not available, and it is still impossible to determine whether the traffic light is correct. From this example, it can be seen that if only the display data of the traffic lights recognized by the unmanned vehicle is used to detect whether the traffic light display is correct, there may be a problem of low detection accuracy. However, in the embodiment of the present disclosure, the display data of the traffic lights recognized by the unmanned vehicle and the display data of the traffic lights recognized by the server are used to detect whether the traffic light display is correct, which can improve the detection accuracy.

[0093] Fig.8D b in the figure shows that in the second time series data and the third time series data, the target traffic light has different light colors at the same time, indicating that the light color of the traffic light recognized by the unmanned vehicle is inconsistent with the light color of the traffic light recognized by the service end, and the second time series data and the third time series data are unavailable. Fig.8D In a, although the light colors of the traffic lights at the same time are consistent in the first time series data and the third time series data, it is still impossible to determine that the traffic lights are correct because the second time series data and the third time series data are unavailable. From this example, it can be seen that if only the display data of the traffic lights recognized by the server is used to detect whether the traffic lights are displayed correctly, there may be a problem of low detection accuracy. However, in the disclosed embodiment, the display data of the traffic lights recognized by the unmanned vehicle and the display data of the traffic lights recognized by the server are used to detect whether the traffic lights are displayed correctly, which can improve the detection accuracy.

[0094] Fourth, in one embodiment, when the target traffic light has the same light color at the same time in the first time series data and the second time series data, the target traffic light has the same light color at the same time in the first time series data, the first time series data, and the second time series data, and also has different light colors at the same time. The server can also detect whether there is a light color jump delay in the target traffic light. The light color jump means: the light color of the target traffic light jumps from the first light color to the second light color, such as the target traffic light jumps from green to yellow, from yellow to red, and from red to green.

[0095] Among them, the server can obtain the time difference of at least three consecutive light colors of the target traffic light. The time difference of the light color change is: the difference between the time when the target traffic light changes from the first light color to the second light color in the first timing data and the second timing data. Because the light color of the target traffic light at the same time is the same in the second timing data and the third timing data, the time difference of the light color change is also equivalent to: the difference between the time when the target traffic light changes from the first light color to the second light color in the first timing data and the second timing data.

[0096] For example, Fig. 8E A schematic diagram of a light color jump delay is shown in FIG. Fig. 8E When the target traffic light changes from green to yellow, the difference between the change time in the first time series data and the change time in the second time series data is 2s. Fig. 8E In the example, the time when the target traffic light in the first time series data changes from green to yellow is 2s later than the time when the target traffic light in the second time series changes from green to yellow. In addition, the time when the target traffic light in the first time series data changes from yellow to red is 2s later than the time when the target traffic light in the second time series changes from yellow to red, and the time when the target traffic light in the first time series data changes from red to green is also 2s later than the time when the target traffic light in the second time series changes from red to green.

[0097] In this way, when the target traffic light changes color three times in a row, the color change time in the first time sequence is 2s later than the color change time in the second time sequence data (third time sequence data), then the server can determine that there is a color change delay in the target traffic light. Among them, the duration of the color change delay of the target traffic light is the change time difference (such as 2s). Accordingly, the server can output a detection result such as "Traffic light 1, there is a 2s change delay".

[0098] It should be understood that corresponding to the light color change delay, the target traffic light may also have an early light color change. In this example, the light color change moment of the target traffic light in the first timing data is 2s earlier than the light color change moment of the target traffic light in the second timing data. For details, please refer to the relevant description of the change delay.

[0099] Because there can be multiple traffic lights, the above uses the target traffic light as an example to illustrate the process of the server detecting whether the target traffic light is displayed correctly. In one embodiment, for the target traffic light, the above problems "three to four" may occur at the same time. In one embodiment, for multiple traffic lights, each traffic light may also have at least one of the problems "one to four".

[0100] In one embodiment, Figure 3 , Figure 5 The process of the embodiment shown can be simplified as follows: Fig. 9 shown.

[0101] In the disclosed embodiment, the server can use a traffic light recognition model to concurrently recognize the traffic lights in each video clip, which can improve the detection efficiency. In addition, in addition to determining whether the traffic light is displayed correctly or incorrectly, the server can also determine when the traffic light is displayed incorrectly, output the specific error time, and can also determine whether the error is a jump delay (or jump early). In this way, the staff can timely use the corresponding error to adjust the traffic light according to the specific display problem of the traffic light to ensure that the traffic light is displayed correctly, thereby improving driving safety.

[0102] The second time series data used in the above embodiment is obtained based on the real-time collected video images during the driving process of the unmanned vehicle. Fig.10 This section describes the process of the unmanned vehicle acquiring the second time series data. Fig.10 , the process of the unmanned vehicle acquiring the second time series data may include:

[0103] S1001, during driving, the unmanned vehicle receives display data of traffic lights from roadside equipment, and the display data of traffic lights in the roadside equipment comes from traffic lights.

[0104] The traffic lights in the embodiments of the present disclosure can be understood as traffic lights to be detected. The unmanned vehicle can travel on a road section (or area) where traffic lights are installed, and collect data during the driving process. For example, Fig.11 The figure shows a road section with traffic lights installed, where multiple traffic lights can be installed at one intersection. In order to detect whether the display of the traffic lights on the road section is correct, the unmanned vehicle can drive on the road section according to the preset driving route. Fig.11 The preset driving route is route 1 as an example for explanation.

[0105] The traffic light can send the data of the traffic light to the roadside equipment in real time, wherein the data of the traffic light may include: the logo of the traffic light, the light color of the traffic light, and the countdown.

[0106] In one embodiment, a traffic light may include multiple display lights (such as lights arranged horizontally or vertically, etc.), so when the traffic light sends the display data of the traffic light to the roadside device, the display data of the traffic light may carry the identification of each display light (such as the position of the display light in the traffic light, such as the middle, left or right, etc.). In one embodiment, a traffic light refers to each display light, and each display light can send its own identification, light color, and countdown to the roadside device. Regardless of how the traffic light sends its own display data, the following is an example of a traffic light representing each display light.

[0107] After receiving the display data from the traffic light, the roadside device can send it to surrounding vehicles through the communication unit in the roadside device. In one embodiment, the unmanned vehicle is within a preset distance range (such as 200m) of the roadside device and can receive the display data of the traffic light from the roadside device.

[0108] In one embodiment, in order to ensure that the unmanned vehicle can collect complete display data of the traffic light, the unmanned vehicle can travel at a lower speed to ensure that more display data from the traffic light can be received in the same length of time, or the unmanned vehicle can circulate several times on the road section where the traffic light is installed to ensure that the unmanned vehicle can collect more display data of the traffic light, so as to accurately detect the traffic light.

[0109] S1002, the unmanned vehicle collects video images and the position information of the unmanned vehicle.

[0110] There is no order distinction between S1002 and S1001, and both can be executed at the same time.

[0111] The unmanned vehicle can collect video images and the position information of the unmanned vehicle during driving. Among them, the unmanned vehicle can use the shooting device 133 in the unmanned vehicle to collect the video image in front of the unmanned vehicle. The position information of the unmanned vehicle includes: the position and phase of the unmanned vehicle. The position of the unmanned vehicle can be the position of the unmanned vehicle in the earth coordinate system (or understood as the position in the high-precision map), and the phase of the unmanned vehicle refers to the direction of the front of the unmanned vehicle. At the same position, the phase of the unmanned vehicle is different, and the video images collected by the unmanned vehicle are different. For example, refer to Fig.11 As shown in the figure, if the front of the unmanned vehicle is facing left, the unmanned vehicle can collect a video image containing traffic light 2, and if the front of the unmanned vehicle is facing forward, the unmanned vehicle can collect a video image containing traffic light 1. For the same position, the position in the high-precision map is the same as the position in the earth coordinate system.

[0112] In the disclosed embodiment, in order to avoid missing the display data of traffic lights, the unmanned vehicle can follow the preset driving route, traverse the intersections on the preset driving route and traverse the phases of the unmanned vehicle to collect video images and the position information of the unmanned vehicle.

[0113] In one embodiment, when the unmanned vehicle is in the middle of the road section and not close to the intersection, the traffic lights are not included in the video images collected by the unmanned vehicle, so the display information of the unmanned vehicle cannot be obtained based on the video image recognition calculation. Therefore, if the unmanned vehicle keeps collecting video images during driving, a lot of useless data will be collected. In order to reduce the energy consumption of the unmanned vehicle and reduce the amount of calculation, the unmanned vehicle can collect video images based on the collected position information of the unmanned vehicle and the high-precision map, if it is determined that the distance between the unmanned vehicle and the intersection ahead of the unmanned vehicle is less than or equal to the preset distance.

[0114] In one embodiment, the unmanned vehicle can capture video footage based on the collected position information of the unmanned vehicle and a high-precision map, if it is determined that the distance between the unmanned vehicle and the center of the intersection ahead of the unmanned vehicle is less than or equal to a preset distance.

[0115] Among them, the preset distance is the distance at which the unmanned vehicle can collect video images containing traffic lights. For example, the preset distance is 60m. Specifically, the high-precision map is marked with the location of each intersection (or the center of each intersection) and the logo of the traffic light installed at each intersection. Therefore, the unmanned vehicle can determine the real-time position of the unmanned vehicle in the high-precision map based on the collected orientation information of the unmanned vehicle, and then the unmanned vehicle can calculate the distance of the unmanned vehicle from the intersection in front (or the center of the intersection) in real time according to its own driving route, and then when the distance is calculated to be less than or equal to the preset distance, the shooting device 133 is controlled to collect the video image in front of the unmanned vehicle.

[0116] S1003, the unmanned vehicle maps the time of receiving the display data of the traffic light with the display data of the traffic light to obtain first time series data.

[0117] The unmanned vehicle can record the time when the display data of the traffic light is received from the roadside equipment, and can map the display data of the traffic light received at the corresponding time to obtain the first time series data. The first time series data includes the display data of the traffic light at each time.

[0118] S1004, the unmanned vehicle obtains second time series data based on the video images and position information collected at different times.

[0119] Taking the first moment included in the different moments as an example, or taking each moment in the different moments as the first moment as an example, the unmanned vehicle can determine that the traffic light in the video image collected at the first moment is the target traffic light based on the position information of the unmanned vehicle collected at the first moment and the high-precision map. Among them, the unmanned vehicle can determine the position of the unmanned vehicle in the high-precision map (such as a position in the road section) based on the position information of the unmanned vehicle collected at the first moment and the high-precision map, and the unmanned vehicle can determine which traffic light in the high-precision map the traffic light collected in front of the unmanned vehicle is, that is, the target traffic light, based on the phase of the unmanned vehicle. Exemplarily, refer to Fig.11 As shown, if the unmanned vehicle is located at intersection 1 and the front of the unmanned vehicle is facing forward, the unmanned vehicle can determine that the traffic light in the collected video image is traffic light 1 (target traffic light) based on the orientation information of the unmanned vehicle.

[0120] After determining the identification of the traffic light (such as the number of the traffic light or the location of the traffic light in China on the high-precision map), the terminal can identify the light color and countdown of the target traffic light according to the traffic light recognition model, and obtain the display data of the target traffic light at the first moment. Among them, the unmanned vehicle can input the video picture (video frame) collected at the first moment into the traffic light recognition model to obtain the display data of the target traffic light at the first moment output by the traffic light recognition model.

[0121] Using the same method, the unmanned vehicle can obtain the display data of the target traffic light at each moment. Using the same method as the target traffic light, the unmanned vehicle can obtain the display data of different traffic lights at each moment, that is, the second time series data.

[0122] In one embodiment, the frequency at which the unmanned vehicle collects position information is 100 Hz, and the frequency at which the traffic light sends the display data of the traffic light to the unmanned vehicle through the roadside equipment is 10 Hz. Therefore, the unmanned vehicle can use a smaller frequency, such as a frequency of 10 Hz, to identify and calculate the display data of the traffic light once through the position information of the unmanned vehicle and the collected video images.

[0123] S1005, at the end of driving, the unmanned vehicle sends the first time series data, the second time series data, and the video collected by the unmanned vehicle during the entire driving stage to the server.

[0124] In the embodiment of the present disclosure, when the unmanned vehicle finishes driving, the first time series data, the second time series data, and the video collected by the unmanned vehicle during the entire driving stage can be sent to the server. Correspondingly, the server can receive the first time series data, the second time series data, and the video from the unmanned vehicle, and then the server can execute the above Figure 3 , Figure 5 Steps in .

[0125] In one embodiment, the entire process from the unmanned vehicle collecting position information and video images to the server outputting the detection results can be simplified as follows: Fig.12 It should be understood that Fig.12 The AI ​​recognition shown in the figure is that the unmanned vehicle uses a traffic light recognition model to identify traffic lights in the video.

[0126] In the disclosed embodiment, when a traffic light needs to be inspected, the unmanned vehicle can traverse the intersection and the phases to obtain the display data of the traffic light as completely as possible to provide data support for the display detection of the traffic light.

[0127] It should be noted that the traffic light recognition model in this embodiment is not a recognition model for a specific traffic light. In the technical solution disclosed in this disclosure, the collection, storage, use, processing, transmission, provision and disclosure of data involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0128] Fig.13 A schematic diagram of a traffic light detection device provided in an embodiment of the present disclosure. The traffic light detection device involved in this embodiment can be the aforementioned server, or a chip applied to the server. The traffic light detection device can be used to execute the actions of the server in the above method embodiment. Fig.13 As shown, the traffic light detection device 1300 may include: a transceiver unit 1301, a processing unit 1302, and an output unit 1303. The processing unit 1302 includes: a time series data acquisition module 13021, a video processing module 13022, and a detection module 13023.

[0129] The transceiver unit 1301 is used to receive the first time series data, the second time series data, and the video from the unmanned vehicle. The video is the video collected by the unmanned vehicle during the entire driving stage. The first time series data is: the display data from the traffic light received by the unmanned vehicle during the entire driving stage. The second time series data is: the display data of the traffic light obtained by the unmanned vehicle based on the video.

[0130] The processing unit 1302 is used to obtain third time series data based on the video, the third time series data includes display data of the traffic light, and obtain a detection result of whether the traffic light display is correct based on the first time series data, the second time series data, and the third time series data.

[0131] The output unit 1303 is used to output the detection result.

[0132] In a possible implementation, the time series data acquisition module 13021 is used to identify the traffic lights in the video according to the traffic light recognition model to obtain the third time series data.

[0133] In one possible implementation, the video processing module 13022 is used to segment the video to obtain at least one video segment; perform frame extraction on the video segment to obtain at least one video frame corresponding to the video segment; input the at least one video frame corresponding to the video segment into a traffic light recognition model to obtain display data of a traffic light corresponding to at least one target video frame, where the target video frame is a video frame containing a traffic light; obtain the acquisition time of the target video frame; map the acquisition time of the target video frame and the display data of the traffic light corresponding to the target video frame to obtain third time series data.

[0134] In a possible implementation, the video processing module 13022 is also used to obtain the acquisition time of the target video frame according to the start and end time of the video, the order of the target video frame in the corresponding video segment, and the interval length between two adjacent video frames when extracting frames.

[0135] In one possible implementation, there is at least one traffic light, and the at least one traffic light includes a target traffic light. The first time series data, the second time series data, and the third time series data all include: the light color of the target traffic light at different times. The first time series data and the second time series data also include the identification detection result of the target traffic light, including the result of whether the target traffic light is displayed correctly.

[0136] If the light colors of the target traffic light at the same time are the same in the first time series data, the second time series data, and the third time series data, the detection module 12023 is used to determine whether the target traffic light is displayed correctly, and the detection result includes: the identification of the target traffic light, and information indicating that the target traffic light is displayed correctly.

[0137] If the light color of the target traffic light at the same time is the same in the second time series data and the third time series data, but the light color of the target traffic light at the first target time in the first time series data is different from the light color of the target traffic light at the first target time in the second time series data and the third time series data, then the detection module 12023 is used to determine that the target traffic light displays an error at the first target time, and the detection result includes: the identification of the target traffic light, and information characterizing that the target traffic light displays an error at the first target time.

[0138] In one possible implementation, when the light colors of the target traffic light at the same time are the same in the first timing data and the second timing data, the detection module 12023 is used to obtain the time difference of at least three consecutive light colors of the target traffic light, and the light color jump time difference is: the difference between the moments when the target traffic light jumps from the first light color to the second light color in the first timing data and the second timing data; and, if the time differences of at least three light colors are the same, it is determined that there is a light color jump delay in the target traffic light, and the duration of the light color jump delay is the jump time difference. The detection result includes: the identification of the target traffic light, and the jump time difference.

[0139] In a possible implementation, if the light colors of the target traffic light at the second target moment are different in the second time series data and the third time series data, the detection module 12023 is used to determine that the data at the second target moment in the second time series data and the third time series data is unavailable.

[0140] The output unit 1303 is also used to output prompt information, where the prompt information is used to instruct the unmanned vehicle to re-collect data at the second target moment.

[0141] The traffic light detection device provided in the embodiment of the present disclosure can execute the actions of the server in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0142] Fig.14 Another structural diagram of the traffic light detection device provided in the embodiment of the present disclosure. The traffic light detection device involved in this embodiment can be the unmanned vehicle mentioned above, or a chip applied to the unmanned vehicle. The traffic light detection device can be used to execute the actions of the unmanned vehicle in the above method embodiment. Fig.14 As shown, the traffic light detection device 1400 may include: a transceiver unit 1401 , a shooting unit 1402 , a positioning unit 1403 , and a processing unit 1404 . The processing unit 1404 may include: a display data acquisition module 14041 and a control module 14042 .

[0143] In one embodiment, the transceiver unit 1401 may include an OBU 131 , the photographing unit 1402 may be a photographing device 133 , the positioning unit 1403 may be a positioning device 132 , and the processing unit 1404 may be a processing device 134 .

[0144] During driving, the transceiver unit 1401 is used to receive display data of the traffic light from the roadside equipment, and the display data of the traffic light in the roadside equipment comes from the traffic light.

[0145] The shooting unit 1402 is used to capture video images.

[0146] The positioning unit 1403 is used to collect the position information of the unmanned vehicle.

[0147] The processing unit 1404 is used to map the time of receiving the display data with the display data to obtain the first time series data; and obtain the second time series data according to the video images and orientation information collected at different times.

[0148] At the end of driving, the transceiver unit 1401 is also used to send the first time series data, the second time series data, and the video collected by the unmanned vehicle during the entire driving stage to the server.

[0149] In a possible implementation, the different moments include a first moment, and the video picture collected at the first moment includes a traffic light. The display data acquisition module 14041 is used to determine that the traffic light in the video picture collected at the first moment is a target traffic light based on the position information of the unmanned vehicle collected at the first moment and the high-precision map; and to identify the light color and countdown of the target traffic light based on the traffic light recognition model to obtain the display data of the target traffic light at the first moment.

[0150] In one possible implementation, the control module 14042 is used to control the unmanned vehicle to follow a preset driving route, traverse intersections on the preset driving route, and traverse the phase of the unmanned vehicle, so that the shooting unit 1402 captures video images and the positioning unit 1403 captures the orientation information of the unmanned vehicle.

[0151] In one possible implementation, the control module 14042 is also used to control the shooting unit 1402 to capture video images if it is determined that the distance between the unmanned vehicle and the intersection ahead of the unmanned vehicle is less than or equal to a preset distance based on the position information of the unmanned vehicle collected by the positioning unit 1403 and the high-precision map.

[0152] The traffic light detection device provided in the embodiment of the present disclosure can execute the actions of the unmanned vehicle in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0153] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0154] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0155] Fig.15A schematic block diagram of an example electronic device 1500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein. In one embodiment, the electronic device can be a service end as in the above embodiment or a vehicle-mounted terminal in an unmanned vehicle.

[0156] like Fig.15 As shown, the device 1500 includes a computing unit 1501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1502 or a computer program loaded from a storage unit 1508 into a random access memory (RAM) 1503. In the RAM 1503, various programs and data required for the operation of the device 1500 can also be stored. The computing unit 1501, the ROM 1502, and the RAM 1503 are connected to each other via a bus 1504. An input / output (I / O) interface 1505 is also connected to the bus 1504.

[0157] A number of components in the device 1500 are connected to the I / O interface 1505, including: an input unit 1506, such as a keyboard, a mouse, etc.; an output unit 1507, such as various types of displays, speakers, etc.; a storage unit 1508, such as a disk, an optical disk, etc.; and a communication unit 1509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1509 allows the device 1500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0158] The computing unit 1501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1501 performs the traffic light detection method in the various embodiments described above. For example, in some embodiments, the traffic light detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1500 via the ROM 1502 and / or the communication unit 1509. When the computer program is loaded into the RAM 1503 and executed by the computing unit 1501, one or more steps of the traffic light detection method described above may be performed. Alternatively, in other embodiments, the computing unit 1501 may be configured to perform the traffic light detection method in any other appropriate manner (e.g., by means of firmware).

[0159] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0160] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0161] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0162] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0163] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0164] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0165] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for detecting a traffic light, comprising: Receiving first time series data, second time series data, and a video from an unmanned vehicle, wherein the video is a video collected by the unmanned vehicle during the entire driving phase, the first time series data is: display data from a traffic light received by the unmanned vehicle during the entire driving phase, and the second time series data is: display data of the traffic light obtained by the unmanned vehicle based on the video; there is at least one traffic light, and the at least one traffic light includes a target traffic light; According to the traffic light recognition model, identify the traffic light in the video, and obtain third time series data, wherein the third time series data includes display data of the traffic light; Obtaining a detection result of whether the traffic light display is correct according to the first time series data, the second time series data, and the third time series data; Outputting the detection result; The first time series data, the second time series data and the third time series data all include: the light colors of the target traffic light at different times; Obtaining a detection result of whether the target traffic light is displayed correctly according to the first time series data, the second time series data, and the third time series data, including: In response to the display data of the target traffic light at the same time being different in the second time series data and the third time series data, determining that the display data at that time in the second time series data and the third time series data is invalid; or, In response to the second timing data and the third timing data, the light color of the target traffic light at the same time is the same, but in the first timing data, the light color of the target traffic light at the first target time is different from the light color of the target traffic light at the first target time in the second timing data and the third timing data, it is determined that the target traffic light displays an error at the first target time.

2. The method according to claim 1, wherein: The step of identifying the traffic light in the video according to the traffic light recognition model to obtain the third time series data includes: Segmenting the video to obtain at least one video segment; Performing frame extraction processing on the video clip to obtain at least one video frame corresponding to the video clip; Inputting at least one video frame corresponding to the video clip into the traffic light recognition model to obtain display data of a traffic light corresponding to at least one target video frame, wherein the target video frame is a video frame including the traffic light; Obtaining the acquisition time of the target video frame; The acquisition time of the target video frame and the display data of the traffic light corresponding to the target video frame are mapped to obtain the third time series data.

3. The method according to claim 2, wherein: The acquiring the acquisition time of the target video frame comprises: The acquisition time of the target video frame is obtained according to the start and end time of the video, the order of the target video frame in the video segment to which it belongs, and the interval length between two adjacent video frames when extracting frames.

4. The method according to any one of claims 1 to 3, wherein the first time series data and the second time series data further include an identification of the target traffic light; The detection result includes whether the target traffic light is displayed correctly; The method further comprises: If the light colors of the target traffic light at the same time are the same in the first time series data, the second time series data, and the third time series data, it is determined that the target traffic light is displayed correctly, and the detection result includes: the identification of the target traffic light, and information characterizing that the target traffic light is displayed correctly.

5. The method according to claim 4, wherein: Obtaining a detection result of whether the target traffic light is displayed correctly according to the first time series data, the second time series data, and the third time series data, including: When the light colors of the target traffic light at the same time are the same in the first time series data and the second time series data, the time difference of the jump of at least three consecutive light colors of the target traffic light is obtained, and the time difference of the jump of the light color is: the difference between the time when the target traffic light jumps from the first light color to the second light color in the first time series data and the second time series data; If the transition time differences of the at least three light colors are the same, it is determined that the target traffic light has a light color transition delay, the duration of the light color transition delay is the transition time difference, and the detection result includes: the identification of the target traffic light and the transition time difference.

6. The method according to claim 4, further comprising: If the light colors of the target traffic light at the second target time are different in the second time series data and the third time series data, it is determined that the data at the second target time in the second time series data and the third time series data is unavailable; Output prompt information, where the prompt information is used to instruct the unmanned vehicle to re-collect data at the second target moment.

7. A method for detecting a traffic light, comprising: During driving, receiving display data of a traffic light from a roadside device, wherein the display data of the traffic light in the roadside device comes from the traffic light; There is at least one traffic light, and the at least one traffic light includes a target traffic light; Collect video images and the position information of the unmanned vehicle; Mapping the time of receiving the display data with the display data to obtain first timing data; Acquire second time series data according to the video images and position information collected at different times; At the end of the driving, the first time series data, the second time series data, and the video collected by the unmanned vehicle during the entire driving phase are sent to the server; The server is used to identify the traffic light in the video according to the traffic recognition model, obtain the third time series data, and obtain the detection result of whether the traffic light is displayed correctly according to the first time series data, the second time series data, and the third time series data; The third time series data includes display data of the traffic light; The first time series data, the second time series data and the third time series data all include: the light colors of the target traffic light at different times; Obtaining a detection result of whether the target traffic light is displayed correctly according to the first time series data, the second time series data, and the third time series data, including: In response to the display data of the target traffic light at the same time being different in the second time series data and the third time series data, determining that the display data at that time in the second time series data and the third time series data is invalid; or, In response to the second timing data and the third timing data, the light color of the target traffic light at the same time is the same, but in the first timing data, the light color of the target traffic light at the first target time is different from the light color of the target traffic light at the first target time in the second timing data and the third timing data, it is determined that the target traffic light displays an error at the first target time.

8. The method according to claim 7, wherein: The different moments include a first moment, the video picture collected at the first moment includes a traffic light, and the acquiring of the second time series data according to the video pictures and the orientation information collected at different moments includes: According to the position information of the unmanned vehicle collected at the first moment and the high-precision map, determining that the traffic light in the video picture collected at the first moment is the target traffic light; According to the traffic light recognition model, the light color and countdown of the target traffic light are recognized to obtain the display data of the target traffic light at the first moment.

9. The method according to claim 7, wherein: The collecting of video images and the position information of the unmanned vehicle includes: According to the preset driving route, the intersections on the preset driving route and the phases of the unmanned vehicle are traversed to collect video images and the position information of the unmanned vehicle.

10. The method according to any one of claims 7 to 9, wherein: The collecting of video images comprises: Based on the collected position information of the unmanned vehicle and the high-precision map, if it is determined that the distance between the unmanned vehicle and the intersection ahead of the unmanned vehicle is less than or equal to a preset distance, a video image is collected.

11. A traffic light detection device, comprising: a transceiver unit, configured to receive first time series data, second time series data, and a video from the unmanned vehicle, wherein the video is a video collected by the unmanned vehicle during the entire driving phase, the first time series data is: display data received by the unmanned vehicle from a traffic light during the entire driving phase, and the second time series data is: display data of the traffic light obtained by the unmanned vehicle based on the video; there is at least one traffic light, and the at least one traffic light includes a target traffic light; A processing unit, configured to identify the traffic light in the video according to a traffic light recognition model, and obtain third time series data, wherein the third time series data includes display data of the traffic light; and obtaining a detection result of whether the traffic light display is correct according to the first time series data, the second time series data, and the third time series data; An output unit, used for outputting the detection result; The first time series data, the second time series data and the third time series data all include: the light colors of the target traffic light at different times; The processing unit is specifically configured to determine that the display data at the same moment in the second time series data and the third time series data is invalid in response to the display data of the target traffic light at the same moment in the second time series data and the third time series data being different; or, In response to the second timing data and the third timing data, the light color of the target traffic light at the same time is the same, but in the first timing data, the light color of the target traffic light at the first target time is different from the light color of the target traffic light at the first target time in the second timing data and the third timing data, it is determined that the target traffic light displays an error at the first target time.

12. A traffic light detection device, comprising: During driving, the transceiver unit is used to receive display data of a traffic light from a roadside device, wherein the display data of the traffic light in the roadside device comes from the traffic light; there is at least one traffic light, and the at least one traffic light includes a target traffic light; A shooting unit, used for capturing video images; as well as, Positioning unit, used to collect the position information of the unmanned vehicle; A processing unit, configured to map the time of receiving the display data with the display data to obtain first time series data; and to obtain second time series data according to the video images and orientation information collected at different times; At the end of driving, the transceiver unit is further used to send the first time series data, the second time series data, and the video collected by the unmanned vehicle during the entire driving stage to the server; The server is used to identify the traffic light in the video according to the traffic recognition model, obtain the third time series data, and obtain the detection result of whether the traffic light is displayed correctly according to the first time series data, the second time series data, and the third time series data; The third time series data includes display data of the traffic light; The first time series data, the second time series data and the third time series data all include: the light colors of the target traffic light at different times; Obtaining a detection result of whether the target traffic light is displayed correctly according to the first time series data, the second time series data, and the third time series data, including: In response to the display data of the target traffic light at the same time being different in the second time series data and the third time series data, determining that the display data at that time in the second time series data and the third time series data is invalid; or, In response to the second timing data and the third timing data, the light color of the target traffic light at the same time is the same, but in the first timing data, the light color of the target traffic light at the first target time is different from the light color of the target traffic light at the first target time in the second timing data and the third timing data, it is determined that the target traffic light displays an error at the first target time.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

15. A computer program product, comprising a computer program, which implements the steps of the method according to any one of claims 1 to 10 when executed by a processor.

Images