Signal light state correction method, device and computer readable storage medium

Abstract

The embodiment of the present invention provides a method, device and computer-readable storage medium for correcting the state of a signal light. The method includes: according to the position of the signal light group contained in each frame of the multi-frame image, associating the signal light groups with the same position in the multi-frame image; obtaining the state of the signal lights in the signal light groups that are mutually associated in the multi-frame image, and determining whether there is a conflict between the states of the signal lights in the signal light groups that are mutually associated between different frames in the multi-frame image; if there is a conflict, correcting the state of the conflicting signal light. Through the present invention, the problem of how to improve the accuracy of signal light state judgment is solved. The scheme outputs the state according to the light group, integrates the signal light state in the multi-frame image, and uses the signal light state in the multi-frame image to correct some erroneous recognitions, thereby improving the accuracy of signal light state judgment.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of video surveillance, and in particular, to a signal light state correction method, device and computer-readable storage medium. Background Art

[0002] With the development of video surveillance technology, the monitoring of urban traffic conditions has become more and more intelligent, and the results of video surveillance have been applied in various fields. For example, video surveillance is used to capture violations, and video surveillance is used to identify traffic lights and pedestrian and vehicle flows at intersections for intelligent navigation.

[0003] The widespread implementation of video surveillance relies on accurate recognition technology. If the recognition result is inaccurate, it will inevitably lead to misjudgment of violation capture or navigation errors. Whether in violation capture, intelligent navigation or other monitoring scenarios involving vehicles and pedestrians on the road, accurate recognition of traffic light status is crucial. Although traffic light status can be recognized based on monitoring images, how to improve the accuracy of traffic light status judgment is currently a problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present invention provide a method, an apparatus and a computer-readable storage medium for correcting the state of a signal light, so as to at least solve the problem of how to improve the accuracy of judging the state of a signal light.

[0005] According to one embodiment of the present invention, a method for correcting the state of a traffic light is provided, comprising: associating the traffic light groups at the same position in the multiple frames of images according to the position of the traffic light groups contained in each frame of the multiple frames of images; obtaining the states of the traffic lights in the mutually related traffic light groups in the multiple frames of images, and determining whether there is a conflict in the states of the traffic lights in the mutually related traffic light groups between different frames of the multiple frames of images; and in case of a conflict, correcting the states of the conflicting traffic lights.

[0006] According to at least one exemplary embodiment, before obtaining the states of signal lights in signal light groups that are mutually associated in the multiple frames of images, the method further includes: identifying the positions of the signal light groups and the positions of the signal lights contained in each frame of image; and determining the signal light group to which each signal light contained in each frame of image belongs based on the positions of the signal light groups and the positions of the signal lights contained in each frame of image.

[0007] According to at least one exemplary embodiment, the position of the signal light group is represented by the coordinate information of the first target frame corresponding to the signal light group, and the position of the signal light is represented by the coordinate information of the second target frame corresponding to the signal light. According to the position of the signal light group and the position of the signal light contained in each frame of the image, determining the signal light group to which each signal light contained in each frame of the image belongs includes: determining a first intersection-and-union ratio between the first target frame and the second target frame based on the coordinate information of the first target frame and the coordinate information of the second target frame; and determining that the signal light corresponding to the second target frame belongs to the signal light group corresponding to the first target frame when the first intersection-and-union ratio is greater than a first threshold.

[0008] According to at least one exemplary embodiment, the position of the signal light group is represented by the coordinate information of the first target frame corresponding to the signal light group, and according to the position of the signal light group contained in each frame of the multi-frame image, the signal light groups in the same position in the multi-frame image are associated, including: determining a second intersection-and-union ratio between the first target frames corresponding to the signal light groups in every two frames of the multi-frame image based on the coordinate information of the first target frame; when the second intersection-and-union ratio is greater than a second threshold, determining that the positions of the signal light groups in the two frames are the same, and associating the signal light groups with the same determined positions.

[0009] According to at least one exemplary embodiment, determining whether there is a conflict in the states of the signal lights in the mutually related signal light groups between different frame images in the multiple frame images includes: determining a light group type of the mutually related signal light groups, wherein the light group type includes: a mixed light group or an ordinary light group, a signal light group in which the number of signal lights lit in one frame image is greater than or equal to 2 is the mixed light group, and a signal light group in which the number of signal lights lit in one frame image is less than 2 is the ordinary light group; in a case where the light group type of the mutually related signal light groups is the ordinary light group, determining whether there is a conflict in the states of the signal lights in the mutually related signal light groups between different frame images in the multiple frame images.

[0010] According to at least one exemplary embodiment, determining whether there is a conflict in the states of the signal lights in the mutually related signal light groups between different frame images in the multiple frame images includes at least one of the following: conflict situation 1: when the colors of the signal lights at different positions in the mutually related signal light groups between different frame images in the multiple frame images are the same, it is determined that there is a conflict in the states of the signal lights in the mutually related signal light groups; conflict situation 2: when the directions of the signal lights at the same position in the mutually related signal light groups between different frame images in the multiple frame images are different, it is determined that there is a conflict in the states of the signal lights in the mutually related signal light groups; conflict situation Form 3: When the colors of the signal lights at the same position in the mutually related signal light groups are different between different frame images in the multiple frame images, it is determined that there is a conflict in the states of the signal lights in the mutually related signal light groups; Conflict situation 4: When, in X consecutive frame images of the multiple frame images, the signal lights at the same position in the mutually related signal light groups of the first frame image and the last frame image are lit and have the same state, but the signal lights at the corresponding positions in one or more frame images between the first frame image and the last frame image are not lit, it is determined that there is a conflict in the states of the signal lights in the mutually related signal light groups, wherein X is a natural number and 3≤X≤the third threshold value.

[0011] According to at least one exemplary embodiment, in the event of a conflict, modifying the state of the conflicting signal light includes at least one of the following: for conflict scenario 1, modifying the color of the conflicting signal light based on the arrangement direction of the mutually related signal light groups and the position of the conflicting signal light; for conflict scenario 2, determining whether there is a non-conflicting pie-shaped signal light within a predetermined area of ​​the signal light group to which the conflicting signal light belongs; if not, modifying the conflicting signal light to a pie-shaped signal light; if so, determining the shape of the conflicting signal light within a predetermined determination range, wherein the predetermined determination range includes other signal light directions except the pie-shaped shape; for conflict scenario 3, in the event that the mutually related signal light groups belong to a vertical light group or a horizontal light group , based on the arrangement direction of the mutually related signal light groups and the position of the conflicting signal light, the color of the conflicting signal light is corrected; for conflict situation 3, when the mutually related signal light group belongs to a single-light group, determine whether there is a non-conflicting signal light within the predetermined area of ​​the signal light group to which the conflicting signal light belongs. If so, the color of the conflicting signal light is corrected based on the color and position of the non-conflicting signal light. If not, the color of the conflicting signal light is determined; for conflict situation 4, based on the states of the signal lights in the mutually related signal light groups of the first frame image and the last frame image, the states of the unlit signal lights in one or more frames between the first frame image and the last frame image are corrected.

[0012] According to at least one exemplary embodiment, in the conflict situation 1 and the conflict situation 3, based on the arrangement direction of the mutually associated signal light groups and the position of the conflicting signal light, correcting the color of the conflicting signal light includes: determining the position of the conflicting signal light in the signal light group to which it belongs based on the position of the conflicting signal light; deriving the correct color of the conflicting signal light based on the arrangement direction of the signal light group and the position of the conflicting signal light in the signal light group; and correcting the color of the conflicting signal light based on the derived correct color.

[0013] According to at least one exemplary embodiment, in the event of a conflict, after correcting the state of the conflicting signal light, the method further includes: outputting the position of the signal light group and the position and corrected state of the signal light contained in each frame of the multiple frames of images.

[0014] According to another embodiment of the present invention, there is provided a signal light state correction device, comprising: an association module, configured to associate signal light groups at the same position in the multiple frames of images according to the position of the signal light groups contained in each frame of the multiple frames of images; a conflict judgment module, configured to obtain the states of signal lights in the mutually related signal light groups in the multiple frames of images, and judge whether there is a conflict in the states of the signal lights in the mutually related signal light groups between different frames of the multiple frames of images; and a state correction module, configured to correct the states of the conflicting signal lights when a conflict exists.

[0015] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0016] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0017] Through the present invention, since the scheme associates the signal light groups with the same position in multiple frames of images, and determines whether there is a conflict in the states of the signal lights in the signal light groups that are mutually associated between different frames in the multiple frames of images, if there is a conflict, the states of the conflicting signal lights are corrected. Therefore, the problem of how to improve the accuracy of signal light state judgment can be solved. The scheme outputs the states according to the light groups, fuses the signal light states in multiple frames of images, and uses the signal light states in the multiple frames of images to correct some erroneous recognitions, thereby improving the accuracy of signal light state judgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a hardware structure block diagram of a monitoring server of a signal light recognition method according to an embodiment of the present invention;

[0019] Figure 2 is a flow chart of a method for correcting a signal light state according to an embodiment of the present invention;

[0020] Figure 3 is a structural block diagram of a signal light state correction device according to an embodiment of the present invention;

[0021] Figure 4 is an exemplary structural frame of the signal light state correction device according to an embodiment of the present invention. Figure 1 ;

[0022] Figure 5 is an exemplary structural frame of the signal light state correction device according to an embodiment of the present invention. Figure 2 ;

[0023] Figure 6 is a detailed flow chart of a method for correcting a signal light state according to an embodiment of the present invention;

[0024] Figure 7 is a schematic diagram of a signal lamp and a lamp assembly according to an embodiment of the present invention;

[0025] Figure 8 is a schematic diagram of a common lamp group and a hybrid lamp group according to an embodiment of the present invention;

[0026] Fig. 9 is a schematic diagram of conflict scenario 1 according to an embodiment of the present invention;

[0027] Fig.10 is a schematic diagram of conflict scenario 2 according to an embodiment of the present invention;

[0028] Fig.11 is a schematic diagram of conflict scenario 3 according to an embodiment of the present invention;

[0029] Fig.12 is a detailed flow chart of conflict handling according to an embodiment of the present invention;

[0030] Fig.13 is a schematic diagram of a single-lamp lamp group according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to realize applications such as traffic penalty or intelligent navigation, it is necessary to detect and identify the traffic lights in the monitoring image. However, errors are inevitable in the detection and identification process. How to improve the accuracy of traffic light status judgment is a problem that needs to be solved urgently.

[0032] To address this problem, an embodiment of the present invention provides a method, device and computer-readable storage medium for correcting the state of a signal light. Based on obtaining the state of a signal light in a single-frame image, the solution fuses the states of the signal lights in multiple frames of images according to the output states of the signal light groups (also referred to as light groups in the following embodiments for simplicity), uses the states of the signal lights in the multiple frames of images to correct some erroneous recognitions, and finally obtains multiple states of different light groups in the multiple frames of images.

[0033] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0035] The method embodiments provided in the embodiments of the present application can be executed in a device or device collection with computing functions such as a monitoring server, a monitoring platform, a monitoring device, etc. Taking running on a monitoring server as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a monitoring server of a signal light recognition method according to an embodiment of the present invention. Figure 1 As shown, the monitoring server may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a central processing unit CPU, a microprocessor MCU or a programmable logic device FPGA and the like) and a memory 104 for storing data, wherein the monitoring server may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the monitoring server. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0036] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the signal light recognition method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the monitoring server via a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0037] The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network or a wired network provided by a communication provider of the monitoring server. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with other devices in the Internet by wireless means, and can also be a transceiver port, which is used to communicate with other devices in the Internet by wired means.

[0038] In this embodiment, a method for correcting the state of a traffic light running on a computing device such as a monitoring server, a monitoring platform, or a monitoring device is provided. Figure 2 is a flow chart of a method for correcting a signal light state according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0039] Step S202 , according to the position of the signal light group contained in each frame of the multiple frames of images, the signal light groups at the same position in the multiple frames of images are associated.

[0040] Since the position and angle of the monitoring equipment in the scene of traffic video monitoring are often relatively fixed, while the signal lights and signal light groups themselves are absolutely fixed, the positions of the signal light groups and signal lights identified from each frame image are also relatively fixed, and the signal light groups and signal lights contained therein can be bound based on the positions of the signal light groups and signal lights in each frame image. According to at least one exemplary embodiment, before step S202, the method may further include:

[0041] Identifying the position of the signal light group and the position of the signal light included in each frame of image;

[0042] The signal light group to which each signal light contained in each frame of image belongs is determined according to the position of the signal light group contained in each frame of image and the position of the signal light.

[0043] Through this solution, the signal light group and the signal lights contained therein can be bound, that is, the signal lights are divided into the signal light groups to which they belong, and subsequent operations can be performed on the basis of the signal light group.

[0044] According to at least one exemplary embodiment, the position of the signal light group is represented by the coordinate information of the first target box corresponding to the signal light group, for example, box1 (xmin1, ymin1, xmax1, ymax1), which is composed of the coordinates of the upper left and lower right points of the first target box, and the position of the signal light is represented by the coordinate information of the second target box corresponding to the signal light, for example, box2 (xmin2, ymin2, xmax2, ymax2), which is composed of the coordinates of the upper left and lower right points of the second target box. In a specific implementation, according to the position of the signal light group and the position of the signal light contained in each frame of the image, determining the signal light group to which each signal light contained in each frame of the image belongs may include:

[0045] Determine a first intersection-over-union ratio between the first target frame and the second target frame based on the coordinate information of the first target frame and the coordinate information of the second target frame;

[0046] When the first intersection-over-union ratio is greater than a first threshold, it is determined that the signal light corresponding to the second target frame belongs to the signal light group corresponding to the first target frame.

[0047] Through this solution, the signal light group to which a certain signal light belongs can be quickly determined, thereby quickly implementing the binding of the signal light group and the signal lights contained therein.

[0048] Since the position and angle of the monitoring equipment in the scene of traffic video monitoring are often relatively fixed, while the traffic lights and traffic light groups themselves are absolutely fixed, the position of the same traffic light and traffic light group in each frame of monitoring image is relatively fixed. Therefore, the traffic light groups in different frames can be associated based on the position of the traffic light group in each frame. According to at least one exemplary embodiment, the position of the traffic light group is represented by the coordinate information of the first target box corresponding to the traffic light group, for example, box1 (xmin1, ymin1, xmax1, ymax1), which is composed of the coordinates of the upper left and lower right points of the first target box. Accordingly, in a specific implementation, step S202, based on the position of the traffic light group contained in each frame of the multiple frames, the process of associating the traffic light groups with the same position in the multiple frames may include:

[0049] Determine, based on the coordinate information of the first target frame, a second intersection-and-union ratio between the first target frames corresponding to the signal light groups in every two frames of the multiple images;

[0050] When the second intersection-over-union ratio is greater than a second threshold, it is determined that the positions of the signal light groups in the two frames of images are the same, and the signal light groups with the same positions are associated.

[0051] Through this solution, it is possible to quickly determine whether the positions of the traffic light groups in different frame images are the same, thereby quickly realizing the association between the traffic light groups at the same positions between different frame images. The determination of whether the traffic lights conflict in subsequent operations is based on the mutually associated traffic light groups in different frame images.

[0052] Step S204, obtaining states of the signal lights in the mutually related signal light groups in the multiple image frames, and determining whether there is a conflict between the states of the signal lights in the mutually related signal light groups between different image frames in the multiple image frames.

[0053] Traffic lights appear in light groups in real scenes. According to whether the color of the same traffic light in a light group changes, it can be divided into mixed light groups and ordinary light groups. The solution in this embodiment can identify conflicts between ordinary light groups. According to at least one exemplary embodiment, determining whether there is a conflict between the states of the traffic lights in the mutually related traffic light groups between different frames of images in the multiple frames of images in step S204 may include:

[0054] Determine the light group type of the mutually associated signal light groups, wherein the light group type includes: a mixed light group or a common light group, a signal light group whose number of lighted light groups in one frame of image is greater than or equal to 2 is the mixed light group, and a signal light group whose number of lighted light groups in one frame of image is less than 2 is the common light group;

[0055] In a case where the light group type of the mutually associated signal light groups is the common light group, it is determined whether there is a conflict between the states of the signal lights in the mutually associated signal light groups between different frame images in the multiple frame images.

[0056] According to at least one exemplary embodiment, determining in step S204 whether there is a conflict between the states of the signal lights in the mutually associated signal light groups between different frame images in the multiple frame images may include at least one of the following:

[0057] Conflict situation 1: In real scenes, the colors of the traffic lights at different positions in the mutually related traffic light groups are generally different. For example, the three traffic lights in an ordinary traffic light group are red, yellow, and green respectively. Once it is found that the colors of the traffic lights at the same position in the same traffic light group are different in different frames, it indicates that there is a conflict in the recognition results of the traffic lights in different frames. In other words, when the colors of the traffic lights at different positions in the mutually related traffic light groups are the same between different frames in the multiple frames, it can be determined that there is a conflict in the states of the traffic lights in the mutually related traffic light groups.

[0058] Conflict situation 2: In real scenes, the directions of the traffic lights at the same position in the mutually related traffic light groups are generally the same, and generally should not change even in different frame images. If the directions of the traffic lights at the same position in different frame images are different, for example, one frame identifies the traffic lights as round lights, and another frame identifies the traffic lights as left turn lights, etc., then it indicates that there is a conflict in the recognition results of the traffic lights in different frame images. In other words, when the directions of the traffic lights at the same position in the mutually related traffic light groups are different between different frame images in the multiple frame images, it is determined that there is a conflict in the states of the traffic lights in the mutually related traffic light groups.

[0059] Conflict situation 3: In real scenes, the colors of the traffic lights at the same position in the mutually related traffic light groups are generally the same, and generally should not change even in different frame images. If the colors of the traffic lights at the same position in different frame images are different, for example, one frame is identified as red, and another frame is identified as yellow, etc. (it is easy to misidentify the red and yellow lights), it indicates that there is a conflict in the recognition results of the traffic lights in different frame images. In other words, when the colors of the traffic lights at the same position in the mutually related traffic light groups are different between different frame images in the multiple frame images, it is determined that there is a conflict in the states of the traffic lights in the mutually related traffic light groups.

[0060] Conflict situation 4: In real scenes, the capture time interval of the image where the traffic light is located is relatively short, and it is impossible to realize a cyclic change of a color within a few frame times. Therefore, if in multiple frames of images within a short period of time, the states of the traffic lights in the previous frame and the next frame are consistent, but the traffic light in the middle frame is not lit, it indicates that the state of the traffic light in the middle frame is missed, which indicates that there is a conflict in the recognition results of the traffic lights in different frames. That is to say, in the continuous X frames of the multiple frames, the traffic lights at the same position in the mutually related traffic light group of the first frame and the last frame are lit and have the same state, but the traffic lights at the corresponding positions in one or more frames between the first frame and the last frame are not lit, it is determined that there is a conflict in the states of the traffic lights in the mutually related traffic light group, where X is a natural number and 3≤X≤the third threshold.

[0061] Step S206: if there is a conflict, modify the state of the conflicting traffic light.

[0062] This embodiment provides a correction scheme for the several conflict situations determined in step S204. According to at least one exemplary embodiment, in the case of a conflict, the correction of the state of the conflicting signal light may include at least one of the following:

[0063] For conflict situation 1, the color of the conflicting signal light is modified based on the arrangement direction of the mutually related signal light groups and the position of the conflicting signal light.

[0064] According to at least one exemplary embodiment, in the conflict situation 1, based on the arrangement direction of the mutually associated signal light groups and the position of the conflicting signal light, correcting the color of the conflicting signal light may include:

[0065] Based on the position of the conflicting signal light, determining the position of the conflicting signal light in the signal light group to which it belongs;

[0066] Deducing a correct color of the conflicting signal light based on an arrangement direction of the signal light group and a position of the conflicting signal light in the signal light group;

[0067] The color of the conflicting signal light is corrected based on the derived correct color.

[0068] For a specific correction example of conflict situation 1, see the processing example based on an actual traffic light recognition scenario given below.

[0069] For conflict situation 2, determine whether there is a non-conflicting pancake-shaped traffic light within the predetermined area of ​​the traffic light group to which the conflicting traffic light belongs. If not, correct the conflicting traffic light to a pancake-shaped traffic light. If so, determine the shape of the conflicting traffic light within a predetermined determination range, wherein the predetermined determination range includes directions of traffic lights other than the pancake shape.

[0070] For a specific correction example of conflict situation 2, see the processing example based on an actual traffic light recognition scenario given below.

[0071] For conflict scenario 3, when the mutually related signal light groups belong to a vertical light group or a horizontal light group, the color of the conflicting signal light is corrected based on the arrangement direction of the mutually related signal light groups and the position of the conflicting signal light.

[0072] According to at least one exemplary embodiment, in the conflict situation 3, based on the arrangement direction of the mutually associated signal light groups and the position of the conflicting signal light, correcting the color of the conflicting signal light may include:

[0073] Based on the position of the conflicting signal light, determining the position of the conflicting signal light in the signal light group to which it belongs;

[0074] Deducing a correct color of the conflicting signal light based on an arrangement direction of the signal light group and a position of the conflicting signal light in the signal light group;

[0075] The color of the conflicting signal light is corrected based on the derived correct color.

[0076] For a specific correction example of conflict situation 3, see the processing example based on an actual traffic light recognition scenario given below.

[0077] For conflict scenario 3, when the mutually related signal light groups belong to a single-light group, determine whether there is a non-conflicting signal light within the predetermined area of ​​the signal light group to which the conflicting signal light belongs. If so, correct the color of the conflicting signal light based on the color and position of the non-conflicting signal light. If not, determine the color of the conflicting signal light.

[0078] For a specific correction example of conflict situation 3, see the processing example based on an actual traffic light recognition scenario given below.

[0079] For conflict scenario 4, based on the states of the signal lights in the interrelated signal light groups of the first frame image and the last frame image, the states of the unlit signal lights in one or more frames between the first frame image and the last frame image are corrected.

[0080] For a specific correction example of conflict situation 4, see the processing example based on an actual traffic light recognition scenario given below.

[0081] According to at least one exemplary embodiment, after step S206, the method may further include: outputting the position of the signal light group and the position and corrected state of the signal light contained in each frame of the multiple frames of images. For example, there are N frames of images in total, and after the light groups are associated, it is known that there are M light groups, and the information finally obtained is M light groups, each light group has N states, and to determine whether the vehicle runs a red light, it is necessary to determine the changes in the N states of all light groups.

[0082] Through the scheme of the embodiment of the present invention, since the signal light groups in the same position in multiple frames of images are associated, and it is determined whether there is a conflict in the states of the signal lights in the signal light groups that are mutually associated between different frames in the multiple frames of images, if there is a conflict, the states of the conflicting signal lights are corrected. Therefore, the problem of how to improve the accuracy of signal light state judgment can be solved. The scheme outputs the state according to the light group, fuses the signal light states in multiple frames of images, and uses the signal light states in the multiple frames of images to correct some erroneous recognitions, thereby improving the accuracy of signal light state judgment.

[0083] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0084] In this embodiment, a signal light state correction device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made are not repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0085] Figure 3 is a structural block diagram of a signal light state correction device according to an embodiment of the present invention. Figure 3 As shown, the signal light state correction device includes: an association module 32, a conflict judgment module 34, and a state correction module 36, which are described in detail below.

[0086] (1) The associating module 32 is configured to associate the signal light groups at the same position in the multiple image frames according to the position of the signal light groups included in each image frame in the multiple image frames.

[0087] Since the position and angle of the monitoring equipment in the scene of traffic video monitoring are often relatively fixed, while the traffic lights and traffic light groups themselves are absolutely fixed, the position of the same traffic light and traffic light group in each frame of monitoring image is relatively fixed. Therefore, the traffic light groups in different frames can be associated based on the position of the traffic light group in each frame. According to at least one exemplary embodiment, the position of the traffic light group is represented by the coordinate information of the first target box corresponding to the traffic light group, for example, box1 (xmin1, ymin1, xmax1, ymax1), which is composed of the coordinates of the upper left and lower right points of the first target box. Accordingly, in a specific implementation, the association module 32 can be set to:

[0088] Determine, based on the coordinate information of the first target frame, a second intersection-and-union ratio between the first target frames corresponding to the signal light groups in every two frames of the multiple images;

[0089] When the second intersection-over-union ratio is greater than a second threshold, it is determined that the positions of the signal light groups in the two frames of images are the same, and the signal light groups with the same positions are associated.

[0090] Through this solution, it is possible to quickly determine whether the positions of the traffic light groups in different frame images are the same, thereby quickly realizing the association between the traffic light groups at the same positions between different frame images. The determination of whether the traffic lights conflict in subsequent operations is based on the mutually associated traffic light groups in different frame images.

[0091] (2) The conflict determination module 34 is configured to obtain the states of the signal lights in the mutually related signal light groups in the multiple image frames, and determine whether there is a conflict between the states of the signal lights in the mutually related signal light groups between different image frames in the multiple image frames.

[0092] Traffic lights appear in groups in real scenes. According to whether the color of the same traffic light in a group changes, it can be divided into mixed groups and ordinary groups. The solution in this embodiment can identify conflicts between ordinary groups. According to at least one exemplary embodiment, the conflict judgment module 34 can be set as:

[0093] Determine the light group type of the mutually associated signal light groups, wherein the light group type includes: a mixed light group or a common light group, a signal light group whose number of lighted light groups in one frame of image is greater than or equal to 2 is the mixed light group, and a signal light group whose number of lighted light groups in one frame of image is less than 2 is the common light group;

[0094] In a case where the light group type of the mutually associated signal light groups is the common light group, it is determined whether there is a conflict between the states of the signal lights in the mutually associated signal light groups between different frame images in the multiple frame images.

[0095] According to at least one exemplary embodiment, the conflict determination module 34 may be configured to determine at least one of the following conflict situations.

[0096] Conflict situation 1: In real scenes, the colors of the traffic lights at different positions in the mutually related traffic light groups are generally different. For example, the three traffic lights in an ordinary traffic light group are red, yellow, and green respectively. Once it is found that the colors of the traffic lights at the same position in the same traffic light group are different in different frames, it indicates that there is a conflict in the recognition results of the traffic lights in different frames. In other words, when the colors of the traffic lights at different positions in the mutually related traffic light groups are the same between different frames in the multiple frames, it can be determined that there is a conflict in the states of the traffic lights in the mutually related traffic light groups.

[0097] Conflict situation 2: In real scenes, the directions of the traffic lights at the same position in the mutually related traffic light groups are generally the same, and generally should not change even in different frame images. If the directions of the traffic lights at the same position in different frame images are different, for example, one frame identifies the traffic lights as round lights, and another frame identifies the traffic lights as left turn lights, etc., then it indicates that there is a conflict in the recognition results of the traffic lights in different frame images. In other words, when the directions of the traffic lights at the same position in the mutually related traffic light groups are different between different frame images in the multiple frame images, it is determined that there is a conflict in the states of the traffic lights in the mutually related traffic light groups.

[0098] Conflict situation 3: In real scenes, the colors of the traffic lights at the same position in the mutually related traffic light groups are generally the same, and generally should not change even in different frame images. If the colors of the traffic lights at the same position in different frame images are different, for example, one frame is identified as red, and another frame is identified as yellow, etc. (it is easy to misidentify the red and yellow lights), it indicates that there is a conflict in the recognition results of the traffic lights in different frame images. In other words, when the colors of the traffic lights at the same position in the mutually related traffic light groups are different between different frame images in the multiple frame images, it is determined that there is a conflict in the states of the traffic lights in the mutually related traffic light groups.

[0099] Conflict situation 4: In real scenes, the capture time interval of the image where the traffic light is located is relatively short, and it is impossible to realize a cyclic change of a color within a few frame times. Therefore, if in multiple frames of images within a short period of time, the states of the traffic lights in the previous frame and the next frame are consistent, but the traffic light in the middle frame is not lit, it indicates that the state of the traffic light in the middle frame is missed, which indicates that there is a conflict in the recognition results of the traffic lights in different frames. That is to say, in the continuous X frames of the multiple frames, the traffic lights at the same position in the mutually related traffic light group of the first frame and the last frame are lit and have the same state, but the traffic lights at the corresponding positions in one or more frames between the first frame and the last frame are not lit, it is determined that there is a conflict in the states of the traffic lights in the mutually related traffic light group, where X is a natural number and 3≤X≤the third threshold.

[0100] (3) The state correction module 36 is configured to correct the state of the conflicting traffic light when there is a conflict.

[0101] This embodiment provides a correction scheme for several conflict situations determined by the conflict determination module 34. According to at least one exemplary embodiment, the state correction module 36 may be configured to perform at least one of the following correction operations.

[0102] For conflict situation 1, the color of the conflicting signal light is modified based on the arrangement direction of the mutually related signal light groups and the position of the conflicting signal light.

[0103] According to at least one exemplary embodiment, in the conflict situation 1, the state correction module 36 may be configured as follows:

[0104] Based on the position of the conflicting signal light, determining the position of the conflicting signal light in the signal light group to which it belongs;

[0105] Deducing a correct color of the conflicting signal light based on an arrangement direction of the signal light group and a position of the conflicting signal light in the signal light group;

[0106] The color of the conflicting signal light is corrected based on the derived correct color.

[0107] For a specific correction example of conflict situation 1, see the processing example based on an actual traffic light recognition scenario given below.

[0108] For conflict situation 2, determine whether there is a non-conflicting pancake-shaped traffic light within the predetermined area of ​​the traffic light group to which the conflicting traffic light belongs. If not, correct the conflicting traffic light to a pancake-shaped traffic light. If so, determine the shape of the conflicting traffic light within a predetermined determination range, wherein the predetermined determination range includes directions of traffic lights other than the pancake shape.

[0109] For a specific correction example of conflict situation 2, see the processing example based on an actual traffic light recognition scenario given below.

[0110] For conflict scenario 3, when the mutually related signal light groups belong to a vertical light group or a horizontal light group, the color of the conflicting signal light is corrected based on the arrangement direction of the mutually related signal light groups and the position of the conflicting signal light.

[0111] According to at least one exemplary embodiment, in the conflict situation 3, the state correction module 36 may be configured as follows:

[0112] Based on the position of the conflicting signal light, determining the position of the conflicting signal light in the signal light group to which it belongs;

[0113] Deducing a correct color of the conflicting signal light based on an arrangement direction of the signal light group and a position of the conflicting signal light in the signal light group;

[0114] The color of the conflicting signal light is corrected based on the derived correct color.

[0115] For a specific correction example of conflict situation 3, see the processing example based on an actual traffic light recognition scenario given below.

[0116] For conflict scenario 3, when the mutually related signal light groups belong to a single-light group, determine whether there is a non-conflicting signal light within the predetermined area of ​​the signal light group to which the conflicting signal light belongs. If so, correct the color of the conflicting signal light based on the color and position of the non-conflicting signal light. If not, determine the color of the conflicting signal light.

[0117] For a specific correction example of conflict situation 3, see the processing example based on an actual traffic light recognition scenario given below.

[0118] For conflict scenario 4, based on the states of the signal lights in the interrelated signal light groups of the first frame image and the last frame image, the states of the unlit signal lights in one or more frames between the first frame image and the last frame image are corrected.

[0119] For a specific correction example of conflict situation 4, see the processing example based on an actual traffic light recognition scenario given below.

[0120] In traffic video surveillance scenarios, the position and angle of monitoring equipment are often relatively fixed, while traffic lights and traffic light groups themselves are absolutely fixed. Therefore, the positions of traffic light groups and traffic lights identified from each frame of the image are also relatively fixed. The traffic light groups and the traffic lights contained therein can be bound based on the positions of the traffic light groups and traffic lights in each frame of the image. Figure 4 is an exemplary structural frame of the signal light state correction device according to an embodiment of the present invention. Figure 1 ,like Figure 4 As shown, the device includes Figure 3 In addition to all modules shown, you can also include:

[0121] An identification module 42 is configured to identify the position of the signal light group and the position of the signal light included in each frame of image;

[0122] The binding module 44 is configured to determine the signal light group to which each signal light included in each frame of image belongs according to the position of the signal light group included in each frame of image and the position of the signal light.

[0123] Through this solution, the signal light group and the signal lights contained therein can be bound, that is, the signal lights are divided into the signal light groups to which they belong, and subsequent operations can be performed on the basis of the signal light group.

[0124] According to at least one exemplary embodiment, the position of the signal light group is represented by the coordinate information of the first target box corresponding to the signal light group, for example, box1 (xmin1, ymin1, xmax1, ymax1), which is composed of the coordinates of the upper left and lower right points of the first target box, and the position of the signal light is represented by the coordinate information of the second target box corresponding to the signal light, for example, box2 (xmin2, ymin2, xmax2, ymax2), which is composed of the coordinates of the upper left and lower right points of the second target box. In a specific implementation, the binding module 44 can be set to:

[0125] Determine a first intersection-over-union ratio between the first target frame and the second target frame based on the coordinate information of the first target frame and the coordinate information of the second target frame;

[0126] When the first intersection-over-union ratio is greater than a first threshold, it is determined that the signal light corresponding to the second target frame belongs to the signal light group corresponding to the first target frame.

[0127] Through this solution, the signal light group to which a certain signal light belongs can be quickly determined, thereby quickly implementing the binding of the signal light group and the signal lights contained therein.

[0128] Figure 5 is an exemplary structural frame of the signal light state correction device according to an embodiment of the present invention. Figure 2 ,like Figure 5 As shown, the device includes Figure 3 In addition to all the modules shown, the present invention may also include: an output module 52, which is configured to output the position of the signal light group and the position and corrected state of the signal light contained in each frame of the multiple frames of images. For example, there are N frames of images in total, and after the light groups are associated, it is known that there are M light groups. The information finally obtained is M light groups, each light group has N states. To determine whether the car runs a red light, it is necessary to determine the changes in the N states of all light groups.

[0129] Through the scheme of the embodiment of the present invention, since the signal light groups in the same position in multiple frames of images are associated, and it is determined whether there is a conflict in the states of the signal lights in the signal light groups that are mutually associated between different frames in the multiple frames of images, if there is a conflict, the states of the conflicting signal lights are corrected. Therefore, the problem of how to improve the accuracy of signal light state judgment can be solved. The scheme outputs the state according to the light group, fuses the signal light states in multiple frames of images, and uses the signal light states in the multiple frames of images to correct some erroneous recognitions, thereby improving the accuracy of signal light state judgment.

[0130] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0131] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0132] In this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0133] S1, according to the position of the signal light group contained in each frame of the multiple frames of images, associating the signal light groups at the same position in the multiple frames of images;

[0134] S2, obtaining states of signal lights in mutually related signal light groups in the multiple image frames, and determining whether there is a conflict between states of signal lights in mutually related signal light groups between different image frames in the multiple image frames;

[0135] S3: In case of a conflict, the state of the conflicting traffic light is corrected.

[0136] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0137] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0138] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0139] In an exemplary embodiment, the processor may be configured to perform the following steps via a computer program:

[0140] S1, according to the position of the signal light group contained in each frame of the multiple frames of images, associating the signal light groups at the same position in the multiple frames of images;

[0141] S2, obtaining states of signal lights in mutually related signal light groups in the multiple image frames, and determining whether there is a conflict between states of signal lights in mutually related signal light groups between different image frames in the multiple image frames;

[0142] S3: In case of a conflict, the state of the conflicting traffic light is corrected.

[0143] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0144] The following will describe in detail a processing example of the signal light state correction scheme of the embodiment of the present invention based on an actual signal light recognition scenario. The signal light state correction scheme of the present embodiment is implemented based on the above information recognized from multiple frames of images. Figure 6 FIG. 1 is a detailed flow chart of a method for correcting a signal light state according to an embodiment of the present invention. Figure 6 As shown, the scheme includes the following operations:

[0145] Step S601, based on the multiple frames of images collected by the monitoring device, the color, direction, position information of the traffic light and the position information of the light group frame in each frame of the image are identified. In this process, the information of the traffic light and the light group in the single frame of the image is obtained. The traffic light information includes the color, direction, and position information in the image; the light group information includes the position information.

[0146] Step S602, divide the traffic lights into corresponding light groups according to the position information, and all subsequent operations are performed in units of light groups. Figure 7 Schematic diagram of a signal light and a light group according to an embodiment of the present invention. The specific division process is as follows:

[0147] Light group coordinate information box1 (xmin1, ymin1, xmax1, ymax1)

[0148] Traffic light coordinate information box2 (xmin2, ymin2, xmax2, ymax2)

[0149] A signal light belongs to a light group if the following conditions are met:

[0150]

[0151] It represents the intersection-over-union ratio between the light group and the traffic light target frame, and thred represents the set threshold.

[0152] Step S603, using the position information to associate the same light group in each frame. Multiple frames are images taken at the same position in a short period of time, so the position of the same light group in different frames is basically the same. The same method as in step S602 can be used to determine whether the light groups in different frames are the same light group.

[0153] After associating the light groups in different frame images, the number of image frames is M, then each light group will have M states, and the M states contain signal light information of different frames.

[0154] Step S604, determining whether the light group belongs to a mixed light group. In real scenes, signal lights appear in light groups, which can be divided into mixed light groups and ordinary light groups according to whether the color of the same signal light in the light group changes.

[0155] Figure 8 is a schematic diagram of a common lamp group and a hybrid lamp group according to an embodiment of the present invention, such as Figure 8 As shown, a common light group is generally composed of three signal lights, which are in the same direction, red, yellow and green, and light up at different times. In a mixed light group, each signal light changes between red, yellow and green, and each signal light indicates a direction. If a light group has more than 2 lights on in the same frame, it is a mixed light group.

[0156] Step S605, processing conflicts between different frames of a single light group. In this embodiment, conflicts between ordinary light groups are processed. When using computer vision technology to detect and identify signal lights, missed detection and wrong identification may occur. Some errors can be corrected by using conflict information between multiple frames.

[0157] There are four main types of conflicts that are dealt with:

[0158] (1) Conflict 1: In different frames, the same light group has different positions with the same signal light color. Fig. 9 is a schematic diagram of conflict situation 1 according to an embodiment of the present invention. Fig. 9 As shown in , the left turn yellow is misidentified as a left turn red.

[0159] (2) Conflict 2: Traffic lights of different directions appear in the same light group. Fig.10 is a schematic diagram of conflict situation 2 according to an embodiment of the present invention, such as Fig.10 As shown, the left turn red light is mistakenly identified as a round red light.

[0160] (3) Conflict 3: The signal lights in the same group and at the same position have different colors. Fig.11 is a schematic diagram of conflict situation 3 according to an embodiment of the present invention, such as Fig.11 As shown, the circular yellow light is mistakenly identified as a circular red light.

[0161] (4) Conflict Scenario 4: The status of the signal lights at the same position in the same light group is consistent in the previous and next frames, but the intermediate frame is missed. The capture time of the image where the signal lights are located is relatively short, and it is impossible to realize a color cycle change in a few frames. It can be determined that the status (color, direction) of the signal lights at the corresponding position in the missed image is consistent with that of the signal lights in the previous and next frames.

[0162] Fig.12 is a detailed flow chart of conflict handling according to an embodiment of the present invention, such as Fig.12 As shown, the steps to handle conflicts are as follows:

[0163] (1) Solution to conflict situation 1: Change the color and status of the wrong light group based on the light position information

[0164] The non-mixed light groups are arranged in a fixed order of red, yellow and green. Since the light group frame detection is not accurate, it is impossible to achieve the ideal state where the width of the signal light is less than 1 / 3 of the light group. Considering that red and yellow signal lights are easier to distinguish errors in practice, they are only corrected to red and yellow here.

[0165] First, determine whether the light group is a vertical light group or a horizontal light group. If it is a vertical light group, and the lights at different positions in the same light group in different frames are all yellow, then the signal lights at the upper position of the correction signal lights are red; if the lights at different positions in the same light group in different frames are all red, then the signal lights at the lower position of the correction signal lights are yellow. The horizontal light group is handled in the same way.

[0166] (2) Solution to conflict situation 2: If there are no circular lights around, it is considered to be a circular light.

[0167] In actual scenarios, a single ordinary arrow light group generally does not appear alone.

[0168] If the light group has this situation in multiple frames, first take the light group as the center and check whether there is a circular light group with no shape conflict within a certain range on the left and right. If not, all conflicting light groups will be corrected to circular lights. If so, use the voting rule to determine its direction (excluding the circular light, there are already circular lights around it). If the voting fails, no processing will be done.

[0169] (3) Solution to conflict situation 3: Refer to your own light group, take the surrounding light groups as reference, and vote

[0170] If it is a vertical light group or a horizontal light group, follow the first method to handle it.

[0171] If the light group is a single-light light group, first check whether there are light groups with no color conflict in the surrounding area. If so, make corrections based on the light colors and positions in the correct light group. If not, vote and correct to the color that appears the most times. If the colors that appear the most and the second most times appear the same number of times, the problem cannot be processed. Fig.13is a schematic diagram of a single-lamp light group according to an embodiment of the present invention, such as Fig.13 As shown, the left turn red light is recognized as a left turn yellow light.

[0172] (4) Solution to conflict situation 4: The missed signal light is manually replaced, and the status is consistent with the previous and next frames (color, direction).

[0173] Step S606: After the conflict is resolved, the status of the signal lights in each light group in each frame can be obtained.

[0174] For example, there are N frames of images. After the light groups are associated, it is found that there are M light groups. The final information obtained is M light groups. Each light group has N states. To determine whether the car runs a red light, it is necessary to determine the changes in the N states of all light groups.

[0175] Through the scheme in the embodiment of the present invention, the signal lights in a single frame are filled into the corresponding light groups, the light group information in different frames are associated and fused, and the changes in different frames of each light group are obtained. In the merging process, multi-frame information can be used to correct some misdetections and missed detections.

[0176] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0177] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for correcting a signal light state, characterized in that: include: According to the position of the signal light group contained in each frame of the multiple frames of images, the signal light groups having the same position in the multiple frames of images are associated; Acquiring states of signal lights in mutually related signal light groups in the multiple image frames, and determining whether there is a conflict between states of the signal lights in the mutually related signal light groups between different image frames in the multiple image frames; In case of a conflict, modifying the state of the conflicting signal light; Wherein, determining whether there is a conflict between the states of the signal lights in the mutually associated signal light groups between different frame images in the multiple frame images comprises at least one of the following: Conflict situation 1: when the colors of the signal lights at different positions in the mutually associated signal light groups are the same between different frame images in the multiple frame images, it is determined that there is a conflict in the states of the signal lights in the mutually associated signal light groups; Conflict situation 2: when the directions of the signal lights at the same position in the mutually associated signal light groups are different between different frame images in the multiple frame images, it is determined that there is a conflict in the states of the signal lights in the mutually associated signal light groups; Conflict situation 3: when the colors of the signal lights at the same position in the mutually associated signal light groups are different between different frame images in the multiple frame images, it is determined that there is a conflict in the states of the signal lights in the mutually associated signal light groups; Conflict situation 4: In the continuous X frames of the multiple frames, the signal lights at the same position in the mutually associated signal light groups of the first frame and the last frame are lit and have the same status, but the signal lights at corresponding positions in one or more frames between the first frame and the last frame are not lit, it is determined that there is a conflict in the status of the signal lights in the mutually associated signal light groups, where X is a natural number and 3≤X≤the third threshold.

2. The method according to claim 1, characterized in that Before obtaining the states of the signal lights in the signal light groups that are mutually associated in the multiple frames of images, the method further includes: Identifying the position of the signal light group and the position of the signal light included in each frame of image; The signal light group to which each signal light contained in each frame of image belongs is determined according to the position of the signal light group contained in each frame of image and the position of the signal light.

3. The method according to claim 2, characterized in that The position of the signal light group is represented by the coordinate information of the first target frame corresponding to the signal light group, and the position of the signal light is represented by the coordinate information of the second target frame corresponding to the signal light. According to the position of the signal light group and the position of the signal light contained in each frame of image, determining the signal light group to which each signal light contained in each frame of image belongs includes: Determine a first intersection-over-union ratio between the first target frame and the second target frame based on the coordinate information of the first target frame and the coordinate information of the second target frame; When the first intersection-over-union ratio is greater than a first threshold, it is determined that the signal light corresponding to the second target frame belongs to the signal light group corresponding to the first target frame.

4. The method according to claim 1, characterized in that: The position of the signal light group is represented by the coordinate information of the first target frame corresponding to the signal light group, and according to the position of the signal light group included in each frame of the multiple frames of images, associating the signal light groups at the same position in the multiple frames of images includes: Determine, based on the coordinate information of the first target frame, a second intersection-and-union ratio between the first target frames corresponding to the signal light groups in every two frames of the multiple images; When the second intersection-over-union ratio is greater than a second threshold, it is determined that the positions of the signal light groups in the two frames of images are the same, and the signal light groups with the same positions are associated.

5. The method according to claim 1, characterized in that Determining whether there is a conflict between the states of the signal lights in the mutually associated signal light groups between different frame images in the multiple frame images includes: Determine the light group type of the mutually associated signal light groups, wherein the light group type includes: a mixed light group or a common light group, a signal light group whose number of lighted light groups in one frame of image is greater than or equal to 2 is the mixed light group, and a signal light group whose number of lighted light groups in one frame of image is less than 2 is the common light group; In a case where the light group type of the mutually associated signal light groups is the common light group, it is determined whether there is a conflict between the states of the signal lights in the mutually associated signal light groups between different frame images in the multiple frame images.

6. The method according to claim 1, characterized in that In the event of a conflict, modifying the state of the conflicting signal light includes at least one of the following: For conflict situation 1, based on the arrangement direction of the mutually related signal light groups and the position of the conflicting signal light, the color of the conflicting signal light is modified; For conflict situation 2, determine whether there is a non-conflicting pancake-shaped traffic light within a predetermined area of ​​the traffic light group to which the conflicting traffic light belongs; if not, correct the conflicting traffic light to a pancake-shaped traffic light; if so, determine the shape of the conflicting traffic light within a predetermined determination range, wherein the predetermined determination range includes directions of other traffic lights except the pancake-shaped traffic light; For conflict scenario 3, when the mutually related signal light groups belong to a vertical light group or a horizontal light group, the color of the conflicting signal light is corrected based on the arrangement direction of the mutually related signal light groups and the position of the conflicting signal light; For conflict situation 3, when the mutually associated signal light group belongs to a single light group, determine whether there is a non-conflicting signal light within the predetermined area of ​​the signal light group to which the conflicting signal light belongs, and if so, modify the color of the conflicting signal light based on the color and position of the non-conflicting signal light; if not, determine the color of the conflicting signal light; For conflict scenario 4, based on the states of the signal lights in the interrelated signal light groups of the first frame image and the last frame image, the states of the unlit signal lights in one or more frames between the first frame image and the last frame image are corrected.

7. The method according to claim 6, characterized in that In the conflict situation 1 and the conflict situation 3, based on the arrangement direction of the mutually related signal light groups and the position of the conflicting signal light, correcting the color of the conflicting signal light includes: Based on the position of the conflicting signal light, determining the position of the conflicting signal light in the signal light group to which it belongs; Deducing a correct color of the conflicting signal light based on an arrangement direction of the signal light group and a position of the conflicting signal light in the signal light group; The color of the conflicting signal light is corrected based on the derived correct color.

8. The method according to claim 1, characterized in that In the case of a conflict, after the state of the conflicting signal light is corrected, the method further includes: The position of the signal light group and the position and corrected state of the signal light included in each frame of the multiple frames of images are output.

9. A signal light state correction device, characterized in that: include: An associating module, configured to associate the signal light groups at the same position in the multiple frames of images according to the position of the signal light groups contained in each frame of the multiple frames of images; a conflict determination module, configured to obtain states of signal lights in mutually related signal light groups in the multiple image frames, and determine whether there is a conflict between the states of the signal lights in the mutually related signal light groups between different image frames in the multiple image frames; A state correction module, configured to correct the state of the conflicting signal light in the event of a conflict; Wherein, determining whether there is a conflict between the states of the signal lights in the mutually associated signal light groups between different frame images in the multiple frame images comprises at least one of the following: Conflict situation 1: when the colors of the signal lights at different positions in the mutually associated signal light groups are the same between different frame images in the multiple frame images, it is determined that there is a conflict in the states of the signal lights in the mutually associated signal light groups; Conflict situation 2: when the directions of the signal lights at the same position in the mutually associated signal light groups are different between different frame images in the multiple frame images, it is determined that there is a conflict in the states of the signal lights in the mutually associated signal light groups; Conflict situation 3: when the colors of the signal lights at the same position in the mutually associated signal light groups are different between different frame images in the multiple frame images, it is determined that there is a conflict in the states of the signal lights in the mutually associated signal light groups; Conflict situation 4: In the continuous X frames of the multiple frames, the signal lights at the same position in the mutually associated signal light groups of the first frame and the last frame are lit and have the same status, but the signal lights at corresponding positions in one or more frames between the first frame and the last frame are not lit, it is determined that there is a conflict in the status of the signal lights in the mutually associated signal light groups, where X is a natural number and 3≤X≤the third threshold.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 8 when executed.

Images