A signal lamp recognition method and device, electronic equipment and storage medium

Abstract

The application provides a signal lamp recognition method and device, electronic equipment and storage medium, which are used for solving the technical problem that single signal lamp or single lamp group recognition misjudgment influences signal lamp state recognition accuracy, and the method comprises the following steps: acquiring a first image, wherein the first image comprises a signal lamp group of a same intersection; performing feature extraction on the first image to obtain a plurality of feature values of the first image; determining significant information of the first image according to first reference values corresponding to the feature values of the first image; wherein the first reference value represents the importance of the corresponding feature value relative to other feature values, and the other feature values comprise feature values other than the corresponding feature value among the feature values; and determining first signal lamp state indication information of the same intersection in the first image according to the significant information of the first image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a signal light recognition method and device, electronic equipment and storage medium. BACKGROUND

[0002] The signal light is an important traffic facility in the traffic scene, which gives the vehicle and the pedestrian the instruction of advancing or stopping, and guarantees the smoothness and safety of the intersection. The signal light recognition has a very important role in the intelligent traffic and the automatic driving field, such as judging the traffic violation or automatically controlling the vehicle to advance or stop.

[0003] In the related art, there are mainly two methods to recognize the signal light:

[0004] Method one: recognizing in the image with a signal light as a unit, calculating the position of the signal light group and the average position of each signal light according to the relative position relationship between the signal lights, and then judging the color of the signal light in the corresponding position. The color misjudgment is prone to occur due to the color cast and the imaging quality, and the light group misjudgment is prone to occur due to the close distance between the signal light groups. Method two: recognizing in the image with a signal light group as a unit, judging the direction of the signal light group at the same time, rotating the non-horizontal arranged signal light group, and then recognizing the signal light group which is changed into the horizontal arrangement. This method involves the direction judgment and the rotation transformation of the signal light group, and the subsequent signal light recognition will also be inaccurate due to the inaccurate direction judgment or rotation transformation of the signal light group.

[0005] Therefore, how to accurately recognize the signal light becomes a problem to be solved. SUMMARY

[0006] The embodiments of the present application provide a signal light recognition method, device, electronic equipment and storage medium, which can improve the accuracy of signal light recognition.

[0007] In a first aspect, the embodiments of the present application provide a signal light recognition method, comprising:

[0008] obtaining a first image, wherein the first image comprises a signal light group of the same intersection;

[0009] performing feature extraction on the first image to obtain a plurality of feature values of the first image;

[0010] determining the salient information of the first image according to a first reference value corresponding to each feature value in the first image, wherein the first reference value represents the importance of the corresponding feature value relative to other feature values, the other feature values include the feature values other than the corresponding feature value in the each feature value, and the salient information of the first image corresponds to the feature value with high importance.

[0011] According to the salient information of the first image, first signal light state indication information of the same intersection in the first image is determined.

[0012] In the embodiments of the present application, a larger reference value is configured for the characteristic value of the area where the light group of the intersection is located by using the distribution relationship between the lights and the light groups, and the salient information is obtained, so that the overall recognition of the light group of the intersection is realized. Compared with the recognition of a single light or a single group, the misjudgment rate can be reduced, and the accuracy of signal light recognition can be improved.

[0013] In an optional implementation, after the first signal light state indication information of the same intersection in the first image is determined according to the salient information of the first image, the method further includes:

[0014] According to at least one second signal light state indication information associated with the same intersection, the first signal light state indication information is verified, and the second signal light state indication information is used to indicate a signal light state of a signal light group of the same intersection.

[0015] In an optional implementation, the method further includes: according to the verification result, target signal light state indication information is determined from the at least one second signal light state indication information, and the target signal light state indication information is used to determine the illegal behavior.

[0016] In the embodiments of the present application, the recognized signal light state indication information is verified, and the basis for determining the illegal behavior is determined based on the verification result, which can further improve the accuracy of signal light state recognition and the accuracy of illegal behavior judgment.

[0017] In an optional implementation, when the verification result indicates that the at least one second signal light state indication information includes the first signal light state indication information, the target signal light state indication information is the first signal light indication information; or,

[0018] When the verification result indicates that the at least one second signal light state indication information does not include the first signal light state indication information, the difference between the target signal light state indication information and the first signal light state indication information is less than a set threshold.

[0019] In an optional implementation, the method further includes:

[0020] determine the first signal light state indication information according to the trained model and the first image; wherein, the input of the model comprises the first image, and the output of the model comprises the first signal light state indication information; the model is obtained by training based on a plurality of training images, and at least two training images of the plurality of training images contain different intersections corresponding to a signal light group.

[0021] In the embodiments of the present application, the trained model is used to identify the signal light state information corresponding to the whole light group at the same intersection, which can improve the accuracy of signal light state identification compared with the identification mode of a single signal light or a single signal light group.

[0022] In an optional implementation, the model comprises a feature extraction sub-model, an encoder, and a decoder; and the determination of the first signal light state indication information according to the trained model and the first image comprises:

[0023] feature extraction of the first image is performed by using the feature extraction sub-model to obtain a plurality of feature values of the first image;

[0024] each feature value in the first image is encoded by using the encoder to obtain an encoded feature corresponding to the feature value;

[0025] each encoded feature is processed by using the decoder based on a second reference value corresponding to the encoded feature to obtain the first signal light state indication information; wherein, the second reference value represents the importance of the corresponding encoded feature relative to other encoded features, and the other encoded features comprise encoded features other than the corresponding encoded feature in the plurality of encoded features.

[0026] In the embodiments of the present application, the trained model can be applied to the identification of signal light state, for example, the consideration of the intersection environment is introduced in combination with the attention mechanism, different reference values are assigned to different encoded features, which can improve the accuracy of signal light state identification.

[0027] In an optional implementation, the first image is marked with a first marked area, and the signal light group at the same intersection is located in the first marked area of the first image. Such design is helpful for fast feature extraction of the first image.

[0028] In a second aspect, the embodiments of the present application provide a signal light identification device, comprising:

[0029] an acquisition module, configured to acquire a first image, wherein the first image comprises a signal light group at the same intersection;

[0030] a processing module configured to perform feature extraction on the first image to obtain a plurality of feature values of the first image;

[0031] The processing module is further configured to determine significant information of the first image according to a first reference value corresponding to each feature value in the first image; wherein the first reference value represents an importance degree of the corresponding feature value relative to other feature values, and the other feature values include feature values other than the corresponding feature value in the each feature value;

[0032] The processing module is further configured to determine the first signal lamp state indication information of the same intersection in the first image according to the significant information of the first image.

[0033] In an optional implementation, the processing module is further configured to verify the first signal lamp state indication information according to at least one second signal lamp state indication information associated with the same intersection; wherein the second signal lamp state indication information is used to indicate a signal lamp state of a signal lamp group of the same intersection.

[0034] In an optional implementation, the processing module is further configured to determine target signal lamp state indication information from the at least one second signal lamp state indication information according to a verification result, and the target signal lamp state indication information is used to determine a violation behavior.

[0035] In an optional implementation, when the verification result indicates that the at least one second signal lamp state indication information includes the first signal lamp state indication information, the target signal lamp state indication information is the first signal lamp indication information; or,

[0036] When the verification result indicates that the at least one second signal lamp state indication information does not include the first signal lamp state indication information, a difference between the target signal lamp state indication information and the first signal lamp state indication information is less than a set threshold.

[0037] In an optional implementation, the processing module is further configured to determine the first signal lamp state indication information according to a trained model and the first image; wherein an input of the model includes the first image, and an output of the model includes the first signal lamp state indication information; the model is obtained by training a plurality of training images, and at least two training images in the plurality of training images contain different intersections corresponding to a signal lamp group.

[0038] In an optional implementation, the model includes a feature extraction sub-model, an encoder, and a decoder; and the processing module is further configured to:

[0039] extracting features of the first image by using the feature extraction sub-model, to obtain a plurality of feature values of the first image;

[0040] encoding each feature value in the first image by using the encoder, to obtain an encoded feature corresponding to the each feature value;

[0041] processing the each encoded feature by using the decoder based on a second reference value corresponding to the each encoded feature, to obtain the first signal light state indication information; the second reference value represents an importance degree of the corresponding encoded feature relative to other encoded features, and the other encoded features include encoded features other than the corresponding encoded feature in the each encoded feature.

[0042] In a third aspect, an electronic device is provided, and the electronic device includes at least a processor and a memory. The processor is configured to execute steps included in any of the optional implementation manners of the first aspect when executing a computer program stored in the memory.

[0043] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores computer executable instructions. The computer executable instructions are configured to cause a computer to execute steps included in any of the implementation manners of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application.

[0045] Figure 1 One of the flowcharts of the signal light recognition method provided by the embodiments of the present application;

[0046] Figure 2 The structural diagram of a model provided by the embodiments of the present application;

[0047] Figure 3 One of the flowcharts of the signal light recognition method provided by the embodiments of the present application;

[0048] Figure 4 One of the flowcharts of the signal light recognition method provided by the embodiments of the present application;

[0049] Figure 5 One of the flowcharts of the signal light recognition method provided by the embodiments of the present application;

[0050] Figure 6 The structural diagram of a signal light recognition device provided by the embodiments of the present application;

[0051] Figure 7 FIG. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments described in the present application document, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of the present application technical solutions.

[0053] The terms "first" and "second" in the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprises" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units that are not listed, or can optionally include other steps or units inherent to the process, method, product or device.

[0054] In the embodiments of the present application, "at least one" can mean at least two, for example, two, three or more, which is not limited by the embodiments of the present application.

[0055] In addition, in the embodiments of the present application, the term "and / or" is only used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents an "or" relationship between the associated objects without special explanation.

[0056] At present, the judgment of the state of the signal lamp is usually carried out by identifying a single signal lamp or a single signal lamp group. The color judgment of a single signal lamp is easily affected by color deviation and imaging quality, and the lamp group misjudgment also occurs between the signal lamp groups with a short distance. The direction judgment and rotation transformation of the signal lamp group are affected by the inaccurate direction judgment or rotation transformation of the signal lamp group, and the subsequent signal lamp recognition is also inaccurate.

[0057] Based on this, the embodiments of the present application provide a signal lamp recognition method, device, electronic device and storage medium, which can recognize the whole lamp group at the same intersection, avoid the misjudgment caused by the recognition of a single signal lamp or a single lamp group, and improve the accuracy of signal lamp recognition.

[0058] For better understanding, the technical scheme of the present application is described in detail below by means of the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical scheme of the present application, but not limitations of the technical scheme of the present application. In the case of no conflict, the technical features in the embodiments of the present application and the embodiments can be combined with each other.

[0059] Embodiment 1

[0060] Referring to Figure 1 An identification method of a signal lamp is shown, which comprises the following steps:

[0061] S101, a first image is acquired, the first image comprising a signal lamp group of a same intersection.

[0062] Optionally, the first image can be obtained by a device with a shooting function, such as an image obtained by shooting the same intersection at a certain moment.

[0063] Optionally, the first image can be marked with a first marking area, and the signal lamp group of the same intersection is located in the first marking area of the first image. Specifically, the first marking area can be marked in the first image by manual calibration, or the first marking area in the first image can be detected and positioned by a detection network, and then marked. The first marking area can also be understood as the position range of the signal lamp group of the same intersection.

[0064] S102, feature extraction is performed on the first image to obtain a plurality of feature values of the first image.

[0065] The feature extraction on the first image can include one or more of the following operations: edge detection, corner feature detection, and the feature values of the first image can include, for example, corners and / or edges. In the embodiments of the present application, the corner can be a corner of two edges, corresponding to the corner of an object in the real world, such as a crossroad, a T-shaped intersection, etc.

[0066] Specifically, based on the region division in the first image, one or more feature values in the target region of the first image can also be determined. For example, based on the description of the first image in S101, the first image comprises a signal lamp group of the same intersection, and it can be understood that the target region contains the signal lamp group of the same intersection, and the feature values of the target region can be used to represent the signal lamp group of the same intersection.

[0067] Optionally, the feature extraction on the first image can be implemented based on a convolutional neural network.

[0068] S103, according to the first reference value corresponding to each feature value in the first image, the significant information of the first image is obtained.

[0069] The first reference value represents an importance degree of the corresponding feature value relative to other feature values, and the other feature values include feature values other than the corresponding feature value among the feature values. The salient information of the first image corresponds to the feature value with a high importance degree. The high importance degree of the feature value can be embodied in that the corresponding first reference value is greater than the first reference values corresponding to the other feature values, or the corresponding first reference value is higher than a set threshold, which is not limited in the embodiments of the present application.

[0070] Specifically, as described in S102, the first reference value corresponding to the feature value can be configured in a sub-region. For example, it can be set that the first reference value corresponding to the feature value in a target region containing a signal lamp group of the same intersection is higher than the first reference value corresponding to the feature value in other regions not containing the signal lamp group of the same intersection. The salient information of the first image can correspond to the feature value in the target region, for representing the target region, that is, the signal lamp group of the same intersection. For another example, it can be set that the first reference value corresponding to the feature value in a region containing a signal lamp is higher than the first reference value corresponding to the feature value in a region not containing the signal lamp. The salient information of the first image can correspond to the feature value in the region containing the signal lamp, for representing the signal lamp in the region.

[0071] Optionally, the attention mechanism can be used to configure a corresponding attention mechanism weight for each feature value, and the attention mechanism weight corresponds to the aforementioned first reference value.

[0072] S104, obtaining first signal lamp state indication information of the same intersection in the first image according to the salient information of the first image.

[0073] Specifically, the aforementioned first signal lamp state indication information of the same intersection is used to indicate the state of all signal lamps of the same intersection identified from the first image. The state of each signal lamp can be one of light off, red light on, yellow light on, and green light on. The type of the signal lamp can be a digital lamp, such as a lamp containing a current state countdown; or the type of the signal lamp can also be a non-countdown traffic light; or the type of the signal lamp can also be a traffic light containing a lane direction (left turn, right turn, or straight) indication.

[0074] Further, based on the setting of the aforementioned first reference value or attention mechanism weight, the first signal lamp state indication information can distinguish different signal lamp groups of the same intersection. Optionally, the first signal lamp state indication information can include signal lamp state indication information of all signal lamp groups of the same intersection, and different signal lamp groups correspond to different lane directions. The signal lamp state indication information of one signal lamp group can represent the state of each signal lamp in the lamp group.

[0075] An optional representation, the first signal lamp state indication information can include the signal lamp states of all signal lamp groups of the same intersection. For example, assuming that it is identified that the same intersection includes a signal lamp group corresponding to a left-turn lane and a signal lamp group corresponding to a straight lane, the first signal state indication information can be: the signal lamp state indication information of the signal lamp group corresponding to the left-turn lane is red light on, yellow light off, and green light off, and the signal lamp state indication information of the signal lamp group corresponding to the straight lane is red light off, yellow light off, and green light on.

[0076] Another optional representation, the first signal lamp state indication information can be a sequence for indicating the signal lamp states of all signal lamp groups of the same intersection. The sequence can be represented as S={s1, s2, …, s n}, where n=C(l)+C(g)-1, C(l) is the number of all signal lamps of the same intersection, and C(g) is the number of groups of the same intersection. Optionally, the signal lamps in a signal lamp group are arranged from left to right and from top to bottom, and the signal lamp groups are arranged from left to right. The sequence can include elements for indicating the signal lamp states and separators for distinguishing the signal lamp groups. Alternatively, it can be understood that any element in the sequence is used to indicate the signal lamp state or the separator. Specifically, different true values, natural numbers, or English letters can be used to represent different states of the lamps. For example, the true value "0" represents that the signal lamp is off, the true value "1" represents that the red light is on, the true value "2" represents that the yellow light is on, the true value "3" represents that the green light is on, the true value "4" represents a red digital lamp, the true value "5" represents a yellow digital lamp, the true value "6" represents a green digital lamp, and "-" represents the separator between the groups.

[0077] For example, assuming that it is identified that the same intersection includes a signal lamp group corresponding to a left-turn lane and a signal lamp group corresponding to a straight lane, the signal lamp state indication information of the signal lamp group corresponding to the left-turn lane is red light on, yellow light off, and green light off, and the signal lamp state indication information of the signal lamp group corresponding to the straight lane is red light off, yellow light off, and green light on. According to the bright and dark states of the signal lamps of the same intersection and the grouping, the first signal lamp indication information corresponding to the same intersection can be obtained, that is, a sequence {1, 0, 0, -, 0, 0, 3}.

[0078] In the embodiments of the present application, by using the distribution relationship between the lamps and the groups of lamps, a larger reference value is configured for the characteristic value of the area where the group of lamps of the intersection is located, significant information is obtained, and thus the overall recognition of the group of lamps of the intersection is realized. Compared with the recognition of a single lamp or a single group, the overall recognition of the group of lamps of the intersection can reduce the misjudgment rate, and thus the accuracy of the signal lamp recognition is improved.

[0079] Embodiment 2

[0080] In this embodiment, to further improve the accuracy of signal light recognition, on the basis of embodiment 1, corresponding to steps S102 and S103, a model is trained to determine the signal light group at the same intersection. That is, the first image is input into the model, and through the processing of the model, the first signal light state indication information corresponding to the first image can be output.

[0081] The training and building of the model are described in detail below.

[0082] The training data (or training data set) of the model can include multiple training images. Among them, at least two training images in the multiple training images contain different intersections corresponding to the signal light group. For the same intersection, sample images of red, yellow and green light on and off at the intersection within a certain time range can be collected. The training image includes a light group, such as a light group covering various distributed light groups such as horizontal and vertical light groups, such as a light group corresponding to a lane direction including left turn, right turn or straight. The position or region of the light group on the training image can also be marked. Optionally, the region where the light group is located in the training image can be located and marked based on the detection network. In addition, the training data of the model can also include label information corresponding to each training image. The label information is the standard (correct) signal light state indication information corresponding to the training image. According to the example described in embodiment 1 that the first signal light state indication information is a sequence, the label information corresponding to a training image can be a standard (correct) sequence indicating the signal light and light group in the training image.

[0083] Referring to Figure 2 An example of the structure of a model is shown. The aforementioned model can include a feature extraction sub-model, an encoder and a decoder. Optionally, the feature extraction sub-model in the model can be implemented based on a recurrent convolutional neural network, the encoder can be implemented based on a recurrent neural network, and the decoder can be implemented based on an attention mechanism and a recurrent neural network.

[0084] During the training of the model, the input of the feature extraction sub-model can include the input of the model, i.e., the aforementioned training image, the feature extraction sub-model can perform feature extraction on the training image, and output one or more feature values of the training image; the input of the encoder can include the output of the feature extraction sub-model, the one or more feature values of the training image are encoded, and the encoded features corresponding to the feature values are output; the input of the decoder can include the output of the encoder, the second reference value corresponding to each encoded feature is assigned, and the second signal lamp state indication information corresponding to the training image is output. Further, according to the second signal state indication information corresponding to the training image and the corresponding label information, the neural network parameters involved in the model are adjusted. In this way, based on a plurality of training images, the training of the model is completed until the set model convergence condition is met. The model convergence condition can be that the difference between the second signal state indication information corresponding to the training image and the corresponding label information is less than a set threshold.

[0085] Specifically, the encoded features output by the encoder and the signal lamp state indication information output by the decoder can comply with the same information definition rule. Taking the signal lamp state indication information as an example, the encoder can encode the signal lamp state contained in the feature value according to the definition of the elements in the sequence, for example, according to the content described in S104 in Embodiment 1, for the feature value containing one bright yellow signal lamp, the corresponding encoded feature can be a true value "2" indicating that the yellow lamp is bright. The decoder can determine the division of the lamp group according to the second reference value corresponding to each encoded feature, and output a sequence that can represent the overall signal lamp state of the lamp group in the training image.

[0086] In addition, it can be understood that for the model combining the attention mechanism to realize the output sequence of the input image, it can also be understood as a sequence recognition model based on the attention mechanism. The model can be applied to the recognition of signal lamp states, for example, by combining the attention mechanism to introduce consideration of the intersection environment, different reference values are assigned to different encoded features, which can improve the accuracy of signal lamp state recognition.

[0087] The following will be described with reference to Figure 3 An example of a signal lamp recognition method is provided, which can be understood as an application process of the above-mentioned model, and the method mainly includes the following steps.

[0088] S301, obtaining a first image, the first image including a signal lamp group at the same intersection.

[0089] This step can be implemented with reference to S101, and the embodiments of the present application will not be described again.

[0090] S302, determining first signal lamp state indication information according to the trained model and the first image.

[0091] The input of the model comprises the first image, and the output of the model comprises the first signal lamp state indication information.

[0092] Specifically, with reference to the scheme of the training process of the aforementioned model, the trained model specifically comprises a feature extraction submodel, an encoder, and a decoder. Based on this, the first signal lamp state indication information is determined according to the trained model and the first image, specifically referring to the training process of the model, and is implemented by the following steps S21-S23.

[0093] S21, performing feature extraction on the first image by using the feature extraction submodel to obtain a plurality of feature values of the first image.

[0094] Specifically, this step can be implemented with reference to S102, and the embodiments of the present application will not be described again.

[0095] S22, performing encoding processing on each feature value in the first image by using the encoder to obtain an encoded feature corresponding to each feature value.

[0096] S23, processing each encoded feature based on a second reference value corresponding to the obtained encoded feature by using the decoder to obtain the first signal lamp state indication information.

[0097] The second reference value represents the importance of the corresponding encoded feature relative to other encoded features, and the other encoded features include the encoded features other than the corresponding encoded feature.

[0098] Specifically, processing each encoded feature to obtain the first signal lamp state indication information can also be described as: generating salient information of the first image according to the second reference value corresponding to each encoded feature value, and outputting the first signal lamp state indication information according to the salient information of the first image. This step S23 can be implemented with reference to S104, and the embodiments of the present application will not be described again.

[0099] Specifically, the encoded features output by the encoder in S22 and the first signal lamp state indication information output by the decoder in S23 can comply with the same information definition rule. Taking the first signal lamp state indication information as an example, the encoder can encode the feature value of the region in the first image corresponding to the signal lamp according to the definition of the elements in the sequence. For example, according to the content described in S104 in Embodiment 1, if the first image has a region containing a bright yellow signal lamp, the encoded feature of the region can be a true value "2" indicating that the yellow lamp is bright. The decoder can determine the division of the signal lamp group according to the second reference value corresponding to each encoded feature of the first image, thereby outputting a sequence that can represent the signal lamp state of the whole lamp group in the first image.

[0100] In the embodiments of the present application, the model trained is used to identify the signal light state information corresponding to the whole light group at the same intersection. Compared with the identification mode of a single signal light or a single signal light group, the accuracy of signal light state identification can be improved.

[0101] Embodiment 3

[0102] On the basis of the foregoing embodiment 1 or embodiment 2, after determining the first signal light state information corresponding to the same intersection, the following verification process can be performed to verify the accuracy of the identified first signal light state information.

[0103] For example, the verification process can be implemented in the following manner:

[0104] The first signal light state indication information is verified according to at least one second signal light state indication information associated with the same intersection. The second signal light state indication information is used to indicate a signal light state of a signal light group at the same intersection, or it can be understood that the second signal light state indication information is a standard (correct) signal light state that can occur at the same intersection.

[0105] The verification mode of the first signal light state indication information mainly involves judging whether the at least one second signal light state indication information associated with the same intersection includes the first signal light state indication information. Alternatively, it can be understood that the first signal light state indication information matches one second signal light state indication information associated with the same intersection.

[0106] Optionally, for the scheme that the signal lamp indication information is in sequence, the at least one second signal lamp state indication information associated with the same intersection can also be understood as the sequence rule associated with the same intersection. For example, there are two signal lamp groups at the same intersection, from left to right, they are the signal lamp group corresponding to the left-turn lane and the signal lamp group corresponding to the straight lane. Each signal lamp group contains three signal lamps, from top to bottom, they correspond to red, yellow and green respectively. Then the standard signal lamp states that can appear at the same intersection can include: left-turn red, left-turn yellow, left-turn green, straight red, straight yellow, straight green. Correspondingly, the sequence rule associated with the same intersection can include the following standard sequences: {1, 0, 0, -, 1, 0, 0}, {1, 0, 0, -, 0, 2, 0}, {1, 0, 0, -, 0, 0, 3}, {0, 2, 0, -, 0, 2, 0}, {0, 0, 3, -, 1, 0, 0}, {0, 2, 0, -, 1, 0, 0}. Based on this, the first signal lamp state indication is referred to as the first sequence, the at least one second signal lamp state indication information associated with the same intersection is referred to as the second sequence, and the sequence rule associated with the same intersection includes at least one second sequence. The foregoing verification process can be understood as judging whether the at least one second sequence associated with the same intersection includes the first sequence. Alternatively, it can be understood as whether the first sequence matches one of the second sequences associated with the same intersection.

[0107] Further optionally, it can also be determined whether the first signal lamp state indication information needs to be corrected according to the verification result. That is, after the foregoing verification process, the following steps can also be performed:

[0108] According to the verification result, the target signal lamp state indication information is determined from the at least one second signal lamp state indication information, and the target signal lamp state indication information is used for determining the illegal behavior. The illegal behavior refers to the behavior of a vehicle or a pedestrian that does not proceed or stop proceeding according to the traffic rules corresponding to the signal lamp state.

[0109] Specifically, when the verification result indicates that the at least one second signal lamp state indication information includes the first signal lamp state indication information, the first signal lamp indication information can be determined as the target signal lamp state indication information; or,

[0110] When the verification result indicates that the at least one second signal lamp state indication information does not include the first signal lamp state indication information, the target signal lamp state indication information can be determined from the at least one second signal lamp state indication information according to the set condition. The set condition is that the difference between the target signal lamp state indication information and the first signal lamp state indication information is less than a set threshold. Alternatively, the difference between the target signal lamp state indication information and the first signal lamp state indication information is the smallest.

[0111] Optionally, the edit distance can be used to represent the difference between the target signal light state indication information and the first signal light state indication information. Taking the signal light state indication information as a sequence for example, corresponding to the definition of the first sequence and the second sequence, at least one second sequence is associated with the same intersection, and the first sequence corresponding to the same intersection can be obtained in the manner identified in Embodiment 1 or Embodiment 2. The edit distance between the first sequence and any one second sequence can indicate the number of different elements between the first sequence and the second sequence. Corresponding to the aforementioned target signal light state indication information, the target sequence determined in the at least one second sequence needs to satisfy that the edit distance between the first sequence and the target sequence is less than a set threshold, such as 2, corresponding to 2 different elements. Alternatively, the edit distance between the first sequence and the target sequence is the minimum.

[0112] In the embodiments of the present application, the identified signal light state indication information is verified, and the result of the verification is used as the basis for determining the violation behavior judgment, which can further improve the accuracy of signal light state recognition and the accuracy of violation behavior judgment.

[0113] Referring to Figure 4 , the embodiments of the present application in Figure 4 illustrate a signal light recognition method combining Embodiment 1 and Embodiment 3, which includes the following steps.

[0114] S101, obtaining a first image, the first image including a signal light group of the same intersection.

[0115] S102, performing feature extraction on the first image to obtain a plurality of feature values of the first image.

[0116] S103, determining the salient information of the first image according to the first reference value corresponding to each feature value in the first image.

[0117] The first reference value represents the importance of the corresponding feature value relative to other feature values, and the other feature values include feature values other than the corresponding feature value in the plurality of feature values. The salient information of the first image corresponds to the feature value with high importance.

[0118] S104, obtaining the first signal light state indication information of the same intersection in the first image according to the salient information of the first image.

[0119] S105, verifying the first signal light state indication information according to at least one second signal light state indication information associated with the same intersection.

[0120] Specifically, the foregoing verification process can be implemented, and the embodiments of the present application will not be described again.

[0121] S106, determining target signal lamp state indication information from the at least one second signal lamp state indication information according to the verification result, the target signal lamp state indication information being used for determining the illegal behavior.

[0122] Specifically, the steps after the foregoing verification process can be implemented with reference to the foregoing verification process, and the embodiments of the present application will not be described again.

[0123] Referring to Figure 5 , the embodiments of the present application illustrate a signal lamp recognition method combined with the embodiments 2 and 3, which comprises the following steps. Figure 5

[0124] S301, acquiring a first image, the first image comprising a signal lamp group of a same intersection.

[0125] S302, determining first signal lamp state indication information according to the trained model and the first image.

[0126] S303, verifying the first signal lamp state indication information according to at least one second signal lamp state indication information associated with the same intersection.

[0127] Specifically, the steps after the foregoing verification process can be implemented with reference to the foregoing verification process, and the embodiments of the present application will not be described again.

[0128] S304, determining target signal lamp state indication information from the at least one second signal lamp state indication information according to the verification result, the target signal lamp state indication information being used for determining the illegal behavior.

[0129] Specifically, the steps after the foregoing verification process can be implemented with reference to the foregoing verification process, and the embodiments of the present application will not be described again.

[0130] Based on the same concept, the embodiments of the present application provide a signal lamp recognition device, which can realize the functions corresponding to the foregoing signal lamp recognition method. The signal lamp recognition device can be a hardware structure, a software module, or a hardware structure plus a software module. The signal lamp recognition device can be realized by a chip system, which can be composed of a chip or can contain a chip and other discrete devices. Please refer to Figure 6 , a structural schematic diagram of a signal lamp recognition device provided by the embodiments of the present application, which comprises an acquisition module 601 and a processing module 602.

[0131] The acquisition module 601 is used for acquiring a first image, the first image comprising a signal lamp group of a same intersection.

[0132] The processing module 602 is used for performing feature extraction on the first image to obtain a plurality of feature values of the first image. ​

[0133] The processing module is further configured to determine the significant information of the first image according to a first reference value corresponding to each feature value in the first image; the first reference value represents an importance degree of the corresponding feature value relative to other feature values, and the other feature values include feature values other than the corresponding feature value in the each feature value.

[0134] The processing module is further configured to determine the first signal lamp state indication information of the same intersection in the first image according to the significant information of the first image.

[0135] In an optional implementation, the processing module 602 is further configured to verify the first signal lamp state indication information according to at least one second signal lamp state indication information associated with the same intersection; the second signal lamp state indication information is used to indicate a signal lamp state of a signal lamp group of the same intersection.

[0136] In an optional implementation, the processing module 602 is further configured to determine target signal lamp state indication information from the at least one second signal lamp state indication information according to a verification result, and the target signal lamp state indication information is used to determine the illegal behavior.

[0137] In an optional implementation, when the verification result indicates that the at least one second signal lamp state indication information includes the first signal lamp state indication information, the target signal lamp state indication information is the first signal lamp indication information; or,

[0138] When the verification result indicates that the at least one second signal lamp state indication information does not include the first signal lamp state indication information, a difference between the target signal lamp state indication information and the first signal lamp state indication information is less than a set threshold.

[0139] In an optional implementation, the processing module 602 is further configured to determine the first signal lamp state indication information according to the trained model and the first image; an input of the model includes the first image, and an output of the model includes the first signal lamp state indication information; the model is obtained by training a plurality of training images, and at least two training images in the plurality of training images contain different intersections corresponding to a signal lamp group.

[0140] In an optional implementation, the model includes a feature extraction sub-model, an encoder, and a decoder; the processing module 602 is further configured to:

[0141] extract features of the first image by using the feature extraction sub-model to obtain a plurality of feature values of the first image;

[0142] The encoder is used to encode each feature value in the first image to obtain an encoded feature corresponding to the each feature value;

[0143] The decoder is used to process the each encoded feature based on a second reference value corresponding to the each encoded feature to obtain the first signal light state indication information, wherein the second reference value represents an importance of the corresponding encoded feature relative to other encoded features, and the other encoded features include encoded features other than the corresponding encoded feature in the each encoded feature.

[0144] In an optional implementation, a first marking area is marked on the first image, and the signal light group of the same intersection is located in the first marking area of the first image.

[0145] Based on the same inventive concept, the embodiments of the present application also provide an electronic device, please refer to Figure 7 The electronic device provided by the embodiments of the present application includes at least one processor 702, and a memory 701 connected or coupled with the at least one processor 702. In addition, the electronic device can also include a communication interface 703. The electronic device can interact with other devices through the communication interface 703. For example, the communication interface 703 can be a transceiver, a circuit, a bus, a module, a pin or other types of communication interfaces. When the electronic device is a chip device or a circuit, the communication interface 703 in the electronic device can also be an input and output circuit, which can input information (or receive information) and output information (or send information). The processor is an integrated processor or a microprocessor or an integrated circuit or a logic circuit. The processor can determine the output information according to the input information.

[0146] The coupling in the embodiments of the present application is an indirect coupling or communication connection between devices, units or modules, which can be electrical, mechanical or other forms, and is used for information interaction between devices, units or modules. The processor 702 can operate cooperatively with the memory 701 and the communication interface 703. The specific connection medium between the processor 702, the memory 701 and the communication interface 703 is not limited in the present disclosure.

[0147] Optionally, referring to Figure 7The processor 702, the memory 701 and the communication interface 703 are connected with each other through a bus 740. The bus 700 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 7 Only one thick line is used to represent the bus in the figure, but it does not mean that there is only one bus or only one type of bus.

[0148] In the embodiment of the present application, the memory 701 is a non-volatile computer readable storage medium, which can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 701 can include at least one type of storage medium, for example, can include flash memory, hard disk, multimedia card, card type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. The memory 701 is any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but is not limited to this. The memory 701 in the embodiment of the present application can also be a circuit or other any device capable of realizing a storage function, used to store instructions, computer programs and / or data.

[0149] In the embodiment of the present application, the processor 702 can be a general-purpose processor, for example, a central processing unit (CPU), a digital signal processor, an application specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, which can realize or execute the methods, steps and logic block diagrams disclosed in the embodiment of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the signal lamp recognition method disclosed in combination with the embodiment of the present application can be directly embodied as hardware processor execution completion, or executed by a combination of hardware and software modules in the processor.

[0150] The signal lamp recognition method introduced in the foregoing embodiments can be implemented by programming the processor 702. The code corresponding to the signal lamp recognition method can be fixed in the chip, so that the chip can execute the steps of the signal lamp recognition method when running. How to program the processor 702 is known to those skilled in the art, and will not be described here.

[0151] Specifically, in the embodiments of the present application, the memory 701 stores instructions executable by the at least one processor 702. The at least one processor 702 can execute the steps included in the foregoing signal lamp recognition method by calling the instructions or computer programs stored in the memory 701. For example, the processor 702 can acquire a first image including a signal lamp group of the same intersection through the communication interface 703. The processor 702 can also call the computer programs stored in the memory 701 to perform feature extraction on the first image to obtain a plurality of feature values of the first image. The processor 702 can further process the plurality of feature values based on the first reference values corresponding to the plurality of feature values to obtain first signal lamp state indication information of the same intersection in the first image, and so on.

[0152] Based on the same inventive concept, the embodiments of the present application also provide a computer readable storage medium storing a computer program. When the computer program runs on a computer, the computer program causes the computer to execute the steps of the signal lamp recognition method as described above.

[0153] In some possible implementation manners, various aspects of the signal lamp recognition method provided by the present application can also be implemented in the form of a computer program product, which includes program codes. When the computer program product runs on an electronic device, the program codes are used to cause the electronic device to execute the steps of the signal lamp recognition method according to various exemplary embodiments of the present application described above.

[0154] Those skilled in the art should understand that the embodiments of the present application can be provided in the form of a method, a system, or a computer program product. Therefore, the embodiments of the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0155] The embodiments of the present application are described with reference to the flowchart illustrations and / or block diagrams of the methods, apparatus (systems) and computer program products according to the embodiments of the present application. It is understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0156] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0158] Obviously, persons having ordinary skill in the art can make various modifications and variations without departing from the spirit and scope of the embodiments of the present application. Thus, it is intended that the embodiments of the present application cover such modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.

Claims

1. A traffic light recognition method, characterized in that, include: Acquire a first image, which includes the traffic light group at the same intersection; Feature extraction is performed on the first image to obtain multiple feature values ​​of the first image; Based on the first reference value corresponding to each feature value in the first image, the salient information of the first image is determined; wherein, the first reference value represents the importance of the corresponding feature value relative to other feature values, and the other feature values ​​include feature values ​​other than the corresponding feature value among the various feature values; Based on the significant information in the first image, determine the status indication information of the first traffic light at the same intersection in the first image; After determining the status indication information of the first traffic light at the same intersection in the first image based on the salient information of the first image, the method further includes: The first traffic light status indication information is verified based on at least one second traffic light status indication information associated with the same intersection; wherein, the second traffic light status indication information is used to indicate a traffic light status of the traffic light group at the same intersection.

2. The method according to claim 1, characterized in that, Also includes: Based on the verification results, the target traffic light status indication information is determined from the at least one second traffic light status indication information, and the target traffic light status indication information is used to determine the violation.

3. The method according to claim 2, characterized in that, When the result of the verification indicates that the at least one second traffic light status indication information includes the first traffic light status indication information, the target traffic light status indication information is the first traffic light indication information. or, When the result of the verification indicates that the at least one second traffic light status indication information does not include the first traffic light status indication information, the difference between the target traffic light status indication information and the first traffic light status indication information is less than a set threshold.

4. The method according to any one of claims 1-3, characterized in that, Also includes: Based on the trained model and the first image, the status indication information of the first traffic light is determined; wherein, the input of the model includes the first image, and the output of the model includes the status indication information of the first traffic light; the model is obtained by training on multiple training images, and at least two of the multiple training images contain different intersections corresponding to the traffic light groups.

5. The method according to claim 4, characterized in that, The model includes a feature extraction sub-model, an encoder, and a decoder; determining the state indication information of the first traffic light based on the trained model and the first image includes: The feature extraction sub-model is used to extract features from the first image to obtain multiple feature values ​​of the first image; The encoder is used to encode each feature value in the first image to obtain the encoded features corresponding to each feature value. The decoder processes each encoded feature based on the second reference value corresponding to each encoded feature to obtain the first signal light status indication information; wherein, the second reference value represents the importance of the corresponding encoded feature relative to other encoded features, and the other encoded features include encoded features other than the corresponding encoded feature among the various encoded features.

6. The method according to any one of claims 1-3, characterized in that, The first image is marked with a first marked area, and the traffic light group at the same intersection is located in the first marked area of ​​the first image.

7. A traffic light recognition device, characterized in that, The device includes: The acquisition module is used to acquire a first image, which includes a group of traffic lights at the same intersection. The processing module is used to extract features from the first image to obtain multiple feature values ​​of the first image; The processing module is further configured to determine the salient information of the first image based on the first reference value corresponding to each feature value in the first image; wherein, the first reference value represents the importance of the corresponding feature value relative to other feature values, and the other feature values ​​include feature values ​​other than the corresponding feature value among the various feature values; The processing module is further configured to determine the status indication information of the first traffic light at the same intersection in the first image based on the salient information of the first image; The processing module is further configured to verify the first traffic light status indication information based on at least one second traffic light status indication information associated with the same intersection; wherein the second traffic light status indication information is used to indicate a traffic light status of the traffic light group at the same intersection.

8. An electronic device, characterized in that, The electronic device includes at least a processor and a memory, wherein the processor is used to execute the steps of the traffic light recognition method according to any one of claims 1-6 when executing a computer program stored in the memory.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the steps of the method according to any one of claims 1-6.

Images