Method and device for identifying indication information of indicator light, electronic device and storage medium

Abstract

The present disclosure relates to a method and device for identifying indication information of an indicator light, an electronic device and a storage medium, the method comprising: acquiring an input image; determining a detection result of a target object based on the input image, the target object comprising at least one of an indicator light base and an indicator light in a lit state, the detection result comprising the type of the target object and the position of a target area where the target object is located in the input image; based on the detection result of the target object, identifying the target area where the target object is located in the input image to obtain indication information of the target object.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a method and device for identifying indication information of an indicator light, an electronic device, and a storage medium. Background Art

[0002] Traffic lights are devices installed on the road to provide guidance signals for vehicles and pedestrians. In practical applications, traffic lights can be used in different scenarios and have different shapes and types, which have complex relationships. There are certain technical difficulties in detecting and identifying different types of traffic lights in different situations. Summary of the invention

[0003] The present disclosure proposes a technical solution for identifying indication information of an indicator light.

[0004] According to one aspect of the present disclosure, a method for identifying indication information of an indicator light is provided, comprising:

[0005] Get the input image;

[0006] Based on the input image, determining a detection result of a target object, the target object comprising at least one of an indicator light base and an indicator light in a lit state, the detection result comprising a type of the target object and a position of a target area where the target object is located in the input image;

[0007] Based on the detection result of the target object, a target area where the target object is located in the input image is identified to obtain indication information of the target object.

[0008] In some possible implementations, determining a detection result of the target object based on the input image includes:

[0009] Extracting image features of the input image;

[0010] Determining a first position of each candidate region of at least one candidate region of the target object based on an image feature of the input image;

[0011] Based on the image features at the first position corresponding to each candidate area in the input image, an intermediate detection result of each candidate area is determined, wherein the intermediate detection result includes a predicted type of the target object and a predicted probability that the target object is the predicted type; the predicted type is an indicator light base and any one of N indicator lights in lighting states, where N is a positive integer;

[0012] A detection result of the target object is determined based on an intermediate detection result of each candidate area in the at least one candidate area and a first position of each candidate area.

[0013] In some possible implementations, determining an intermediate detection result of each candidate region based on an image feature at a first position corresponding to each candidate region in the input image includes:

[0014] For each candidate area, based on the image feature at the first position corresponding to the candidate area, the target object in the candidate area is classified to obtain a predicted probability that the target object is each of the at least one preset type; wherein the preset type includes at least one of an indicator light base and indicator lights in N lighting states, where N is a positive integer;

[0015] The preset type with the highest prediction probability among the at least one preset type is used as the prediction type of the target object in the candidate area, and the prediction probability of the prediction type is obtained.

[0016] In some possible implementations, before determining the detection result of the target object based on the intermediate detection result of each candidate area in at least one candidate area and the first position of each candidate area, the method further includes:

[0017] Determining a position deviation of a first position of each candidate region based on an image feature of the input image;

[0018] The first position of each candidate region is adjusted using the position deviation corresponding to each candidate region.

[0019] In some possible implementations, determining the detection result of the target object based on the intermediate detection result of each candidate area in the at least one candidate area and the first position of each candidate area includes:

[0020] In response to there being at least two candidate regions for the target object, based on an intermediate detection result of each of the at least two candidate regions, or based on the intermediate detection result of each candidate region and a first position of each candidate region, screening out the target region from the at least two candidate regions;

[0021] The predicted type of the target object in the target area is used as the type of the target object, and the first position of the target area is used as the position of the target area where the target object is located, so as to obtain a detection result of the target object.

[0022] In some possible implementations, after determining the detection result of the target object based on the input image, the method further includes at least one of the following:

[0023] In response to the detection results of the target object only including the detection results corresponding to the indicator light base, determining that the indicator light is in a fault state;

[0024] In response to the detection results of the target object only including detection results corresponding to indicator lights in a lit state, it is determined that the scene state of collecting the input image is a dark state.

[0025] In some possible implementations, based on the detection result of the target object, identifying the target area where the target object is located in the input image to obtain the indication information of the target object includes:

[0026] Determining a classifier matching the target object based on the type of the target object in the detection result of the target object;

[0027] The image features of the target area in the input image are identified by using a matching classifier to obtain indication information of the target object.

[0028] In some possible implementations, based on the detection result of the target object, identifying the target area where the target object is located in the input image to obtain the indication information of the target object includes:

[0029] In response to the type of the target object being an indicator light base, determining that the matching classifier includes a first classifier for identifying an arrangement of indicator lights in the indicator light base, using the first classifier to identify image features of a target area where the target object is located, and determining an arrangement of the indicator lights in the indicator light base; and / or,

[0030] The matched classifier includes a second classifier for identifying the scene where the indicator light is located. The second classifier is used to identify image features of the target area where the target object is located to determine scene information where the indicator light is located.

[0031] In some possible implementations, based on the detection result of the target object, identifying the target area where the target object is located in the input image to obtain the indication information of the target object includes:

[0032] In response to the type of the target object being a circular spot light or a pedestrian light, determining that the matching classifier includes a third classifier for identifying a color attribute of the circular spot light;

[0033] The third classifier is used to identify the image features of the target area where the target object is located, and determine the color attribute of the circular spot light or the pedestrian light.

[0034] In some possible implementations, based on the detection result of the target object, identifying the target area where the target object is located in the input image to obtain the indication information of the target object includes:

[0035] In response to the type of the target object being an arrow light, determining that the matching classifier includes a fourth classifier for identifying a color attribute of the arrow light and a fifth classifier for identifying a direction attribute;

[0036] The fourth classifier and the fifth classifier are used to identify the image features of the target area where the target object is located, and the color attribute and the direction attribute of the arrow light are determined respectively.

[0037] In some possible implementations, based on the detection result of the target object, identifying the target area where the target object is located in the input image to obtain the indication information of the target object includes:

[0038] In response to the type of the target object being a digital light, determining that the matching classifiers include a sixth classifier for identifying a color attribute of the digital light and a seventh classifier for identifying a numerical attribute;

[0039] Based on the sixth classifier and the seventh classifier, the image features of the target area where the target object is located are identified, and the color attribute and the numerical attribute of the digital light are determined respectively.

[0040] In some possible implementations, in response to the input image including at least two indicator light bases, the method further includes:

[0041] For a first indicator light base, determining an indicator light of a lighting state matching the first indicator light base; the first indicator light base is one of the at least two indicator light bases;

[0042] The indication information of the first indicator light base and the indication information of the indicator light in the lighting state that matches the first indicator light base are combined to obtain the combined indication information.

[0043] In some possible implementations, determining the indicator light of the lighting state matching the first indicator light base includes:

[0044] Based on the position of the target area where the target object is located in the detection result of the target object, determine a first area of ​​intersection between the target area where the at least one indicator light in the lit state is located and the target area where the first indicator light base is located, and determine a second area of ​​the target area where the at least one indicator light in the lit state is located;

[0045] In response to a ratio between the first area between the first indicator light in the illuminated state and the first indicator light base and the second area of ​​the first indicator light in the illuminated state being greater than a set area threshold, determining that the first indicator light in the illuminated state matches the first indicator light base;

[0046] The first indicator light in the lighting state is one of the at least one indicator light in the lighting state.

[0047] According to a second aspect of the present disclosure, there is provided a driving control method, comprising:

[0048] Using an image acquisition device in an intelligent driving device to collect driving images;

[0049] Executing the indicator light indication information recognition method described in the first aspect on the driving image to obtain indication information for the driving image;

[0050] The instruction information is used to generate a control instruction for the intelligent driving device.

[0051] According to a third aspect of the present disclosure, a device for identifying indication information of an indicator light is provided, comprising:

[0052] An acquisition module, which is used to acquire an input image;

[0053] a determination module, configured to determine a detection result of a target object based on the input image, wherein the target object includes at least one of an indicator light base and an indicator light in a lit state, and the detection result includes a type of the target object and a position of a target area where the target object is located in the input image;

[0054] The recognition module is used to recognize the target area where the target object is located in the input image based on the detection result of the target object, so as to obtain the indication information of the target object.

[0055] In some possible implementations, the determining module is further configured to:

[0056] Extracting image features of the input image;

[0057] Determining a first position of each candidate region of at least one candidate region of the target object based on an image feature of the input image;

[0058] Based on the image features at the first position corresponding to each candidate area in the input image, an intermediate detection result of each candidate area is determined, wherein the intermediate detection result includes a predicted type of the target object and a predicted probability that the target object is the predicted type; the predicted type is an indicator light base and any one of N indicator lights in lighting states, where N is a positive integer;

[0059] A detection result of the target object is determined based on an intermediate detection result of each candidate area in the at least one candidate area and a first position of each candidate area.

[0060] In some possible implementations, the determination module is further used to: for each candidate area, based on the image feature at the first position corresponding to the candidate area, classify the target object in the candidate area to obtain a predicted probability that the target object is each of the at least one preset type; wherein the preset type includes at least one of an indicator light base and indicator lights in N lighting states, where N is a positive integer;

[0061] The preset type with the highest prediction probability among the at least one preset type is used as the prediction type of the target object in the candidate area, and the prediction probability of the prediction type is obtained.

[0062] In some possible implementations, the determination module is further configured to: determine a position deviation of the first position of each candidate region based on an image feature of the input image before determining the detection result of the target object based on the intermediate detection result of each candidate region in the at least one candidate region and the first position of each candidate region;

[0063] The first position of each candidate region is adjusted using the position deviation corresponding to each candidate region.

[0064] In some possible implementations, the determination module is further configured to, when there are at least two candidate regions for the target object, screen out the target region from the at least two candidate regions based on an intermediate detection result of each of the at least two candidate regions, or based on the intermediate detection result of each candidate region and a first position of each candidate region;

[0065] The predicted type of the target object in the target area is used as the type of the target object, and the first position of the target area is used as the position of the target area where the target object is located, so as to obtain a detection result of the target object.

[0066] In some possible implementations, the determination module is further configured to determine that the indicator light is in a fault state when the detection result of the target object only includes the detection result corresponding to the indicator light base;

[0067] In a case where the detection results of the target object only include detection results corresponding to indicator lights in a lit state, it is determined that the scene state of collecting the input image is a dark state.

[0068] In some possible implementations, the recognition module is further used to determine a classifier matching the target object based on the type of the target object in the detection result of the target object;

[0069] The image features of the target area in the input image are identified by using a matching classifier to obtain indication information of the target object.

[0070] In some possible implementations, the recognition module is further used to, when the type of the target object is an indicator light base, determine that the matching classifier includes a first classifier for identifying an arrangement of the indicator lights in the indicator light base, and use the first classifier to identify image features of a target area where the target object is located to determine the arrangement of the indicator lights in the indicator light base; and / or,

[0071] The matched classifier includes a second classifier for identifying the scene where the indicator light is located. The second classifier is used to identify image features of the target area where the target object is located to determine scene information where the indicator light is located.

[0072] In some possible implementations, the recognition module is further configured to, when the type of the target object is a circular spot light or a pedestrian light, determine that the matching classifier includes a third classifier for identifying a color attribute of the circular spot light;

[0073] The third classifier is used to identify the image features of the target area where the target object is located, and determine the color attribute of the circular spot light or the pedestrian light.

[0074] In some possible implementations, the recognition module is further configured to, when the type of the target object is an arrow light, determine that the matching classifier includes a fourth classifier for identifying a color attribute of the arrow light and a fifth classifier for identifying a direction attribute;

[0075] The fourth classifier and the fifth classifier are used to identify the image features of the target area where the target object is located, and the color attribute and the direction attribute of the arrow light are determined respectively.

[0076] In some possible implementations, the recognition module is further used to, when the type of the target object is a digital light, determine that the matching classifier includes a sixth classifier for identifying a color attribute of the digital light and a seventh classifier for identifying a numerical attribute;

[0077] Based on the sixth classifier and the seventh classifier, the image features of the target area where the target object is located are identified, and the color attribute and the numerical attribute of the digital light are determined respectively.

[0078] In some possible implementations, the device further includes a matching module, which is used to determine, for a first indicator light base, an indicator light in a lighting state that matches the first indicator light base when the input image includes at least two indicator light bases; the first indicator light base is one of the at least two indicator light bases;

[0079] The indication information of the first indicator light base and the indication information of the indicator light in the lighting state that matches the first indicator light base are combined to obtain the combined indication information.

[0080] In some possible implementations, the matching module is further used for:

[0081] Based on the position of the target area where the target object is located in the detection result of the target object, determine a first area of ​​intersection between the target area where the at least one indicator light in the lit state is located and the target area where the first indicator light base is located, and determine a second area of ​​the target area where the at least one indicator light in the lit state is located;

[0082] In the case where a ratio of the first area between the first indicator light in the lit state and the first indicator light base to the second area of ​​the first indicator light in the lit state is greater than a set area threshold, determining that the first indicator light in the lit state matches the first indicator light base;

[0083] The first indicator light in the lighting state is one of the at least one indicator light in the lighting state.

[0084] According to a fourth aspect of the present disclosure, there is provided a driving control device, comprising:

[0085] An image acquisition module, which is provided in the intelligent driving device and is used to acquire a driving image of the intelligent driving device;

[0086] An image processing module, configured to execute the indicator light indication information recognition method as described in any one of the first aspects on the driving image to obtain indication information for the driving image;

[0087] A control module is used to generate control instructions for the intelligent driving device using the indication information.

[0088] According to a fifth aspect of the present disclosure, there is provided an electronic device, comprising:

[0089] processor;

[0090] a memory for storing processor-executable instructions;

[0091] The processor is configured to call instructions stored in the memory to execute the method described in any one of the first aspect or the second aspect.

[0092] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method described in any one of the first aspect or the second aspect is implemented.

[0093] The disclosed embodiment can first perform target detection processing on the input image to obtain the target object detection result, wherein the target object detection result can include information such as the location and type of the target object, and then further perform the recognition of the target object's indication information based on the target object detection result. The disclosed embodiment divides the target object detection process into two detection processes, namely, the indicator light base and the indicator light in the lit state, so as to achieve the first distinction of the target object in the detection process. When further identification is performed based on the target object detection result, it is helpful to reduce the recognition complexity and difficulty in the process of identifying the target object's indication information, and can simply and conveniently realize the detection and identification of various types of indicator lights in different situations.

[0094] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

[0095] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0097] Figure 1 A flow chart showing a method for identifying indication information of an indicator light according to an embodiment of the present disclosure;

[0098] FIG2( a ) shows different display states of a traffic light;

[0099] Figure 2(b) shows different arrangements of traffic light bases;

[0100] Figure 2(c) shows different application scenarios of traffic lights;

[0101] Figure 2(d) shows various types of traffic lights;

[0102] FIG2( e ) is a schematic diagram showing a combined traffic light in different situations;

[0103] Figure 3A flowchart showing step S20 in a method for identifying indication information of an indicator light according to an embodiment of the present disclosure is shown;

[0104] Figure 4 A schematic diagram showing the execution of target detection through a region candidate network according to an embodiment of the present disclosure;

[0105] Figure 5 A flowchart showing step S30 in a method for identifying indication information of an indicator light according to an embodiment of the present disclosure is shown;

[0106] Figure 6 A schematic diagram showing classification detection of different target objects according to an embodiment of the present disclosure;

[0107] Figure 7 A schematic diagram of a traffic light structure showing multiple bases;

[0108] Figure 8 Another flow chart showing a method for identifying indication information of an indicator light according to an embodiment of the present disclosure;

[0109] Fig. 9 A flow chart showing a driving control method according to an embodiment of the present disclosure;

[0110] Fig.10 A block diagram showing an indication information recognition device of an indicator light according to an embodiment of the present disclosure is shown;

[0111] Fig.11 A block diagram showing a driving control device according to an embodiment of the present disclosure;

[0112] Fig.12 A block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0113] Fig.13 Another block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0114] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0115] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0116] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0117] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0118] The indicator light indication information recognition method provided in the embodiment of the present disclosure can be used to perform the indication information detection of different types of indicator lights, wherein the indicator light indication information recognition method can be performed by any electronic device with image processing function, for example, by a terminal device or a server or other processing device, wherein the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. Alternatively, in some possible implementations, the indicator light indication information recognition method can also be applied to intelligent driving devices, such as intelligent flight equipment, intelligent vehicles, and guide devices, for intelligent control of intelligent driving devices. In addition, in some possible implementations, the image processing method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0119] Figure 1 A flow chart showing a method for identifying indication information of an indicator light according to an embodiment of the present disclosure is shown. Figure 1 As shown, the method for identifying the indication information of the indicator light includes:

[0120] S10: Obtain an input image.

[0121] In some possible implementations, the input image may be an image of an indicator light, which may include at least one of a traffic indicator light (e.g., a traffic light), an emergency indicator light (e.g., a flashing indicator light), and a direction indicator light. In other embodiments, it may also be other types of indicator lights.

[0122] The present disclosure can realize the recognition of the indication information of the indicator light in the input image. The input image can be an image captured by an image acquisition device, for example, a road driving image captured by an image acquisition device set in a vehicle, or an image captured by a deployed camera, or in other embodiments, an image captured by a handheld terminal device or other device, or the input image can also be an image frame selected from an acquired video stream, which is not specifically limited by the present disclosure.

[0123] S20: Based on the input image, determine the detection result of the target object, the target object includes at least one of an indicator light base and an indicator light in a lit state, and the detection result includes the type of the target object and the position of the target area where the target object is located in the input image.

[0124] In some possible implementations, when an input image is obtained, the target object in the input image can be detected and identified to obtain a detection result about the target object. The detection result may include the type and position information of the target object. In the disclosed embodiment, target detection of the target object in the input image can be achieved through a neural network to obtain a detection result. The neural network can be used to detect at least one of the information including the base type of the indicator light, the type of the indicator light in the lit state, and the base position and the position of the lit indicator light in the input image. The detection result of the input image can be obtained by any neural network that can detect the target object and can classify it. The neural network can be a convolutional neural network.

[0125] In practical applications, the indicator lights included in the collected input images may be in various forms. Taking traffic indicator lights (hereinafter referred to as traffic lights) as an example, the forms of traffic lights may be various. In the case where the traffic lights are round spot lights, Figure 2(a)-Figure 2(e) Schematic diagrams of various display states of a traffic light are shown respectively, wherein FIG2(a) shows different display states of a traffic light. The shape of the base of the traffic light is not limited in the present disclosure.

[0126] In real life, the indicator light base may include indicator lights of multiple color states, so the corresponding indicator light display states may also be multiple. Take the traffic lights in Figure 2(a) as an example, where the first group of traffic lights is used as an example, where L represents the traffic light and D represents the traffic light base. It can be seen from Figure 2(a) that the three red, yellow and green lights in the first group of traffic lights are all off, which may be a fault state. The second group of traffic lights is in the state where the red light is on, the third traffic light is in the state where the yellow light is on, and the fourth group of traffic lights is in the state where the green light is on. In the process of identifying the target object, it is necessary to identify whether the indicator light is in the lit state and the color of the indicator light in the lit state. Among them, the red, yellow and green texts are only schematic representations of the corresponding color traffic lights being in the lit state.

[0127] FIG2(b) shows different arrangements of the traffic light base. Generally, traffic lights or other types of indicator lights can be installed on the indicator light base, as shown in FIG2(b), wherein the arrangement of the traffic lights on the base can include horizontal arrangement, vertical arrangement or a single light. Therefore, in the process of identifying the target object, the arrangement of the indicator lights can also be identified. The above is only an exemplary description of the arrangement of the traffic lights on the base. In other embodiments, the arrangement of the indicator lights on the base can also include other types of arrangements.

[0128] FIG2(c) shows different application scenarios of traffic lights. In actual applications, traffic lights and other indicator lights can be set at road intersections, highway intersections, sharp turns, safety warning locations, or travel channels. Therefore, the identification of indicator lights can also be used to determine and identify the application scenarios of indicator lights. For example, the actual application scenarios in FIG2(c) are highway intersections marked with "Electronic Toll Collection (ETC)" signs, sharp turns marked with warning signs such as "Warning Signal" or other dangerous scenes and general scenes. The above scenarios are exemplary descriptions, and the present disclosure does not specifically limit them.

[0129] Figure 2(d) shows various types of traffic lights. Generally, the shapes of traffic lights or other indicator lights vary according to the needs or scenarios. For example, Figure 2(d) shows arrow lights with arrow shapes, circular spot lights with circular spot shapes, pedestrian lights with pedestrian signs, or digital lights with digital values. At the same time, each type of light can also have different colors, which is not limited in the present application.

[0130] FIG2(e) shows schematic diagrams of combined traffic lights in different situations, such as a combination of arrow lights with different arrow directions, a combination of digital lights and pedestrian lights, and indication information such as color. As described above, in actual applications, there are various types of indicator lights, and the present disclosure can realize the recognition of indication information of various indicator lights.

[0131] Precisely because of the complexity of the above situation, the disclosed embodiment can first determine the detection result of the target object in the input image by detecting the target object in the input image, and then further obtain the indication information of the target object based on the detection result. For example, by performing target detection on the input image, the type and position of the target object in the input image can be detected, or the detection result can also include the probability that the target object is the detected type. When the above detection result is obtained, classification detection is further performed according to the type of the detected target object to obtain the indication information in the target object, such as the light color, value, direction, scene and other information.

[0132] The disclosed embodiment can divide the types of detection targets (i.e., target objects) into two parts: indicator light base and indicator light in a lit state, wherein the indicator light in a lit state can include N types, for example, the indicator light type can include at least one of the above-mentioned digital lights, pedestrian lights, arrow lights, and circular spot lights. Therefore, when performing target object detection, it can be determined that each target object included in the input image is any one of the N+1 types (base and N types of lit indicator lights). Alternatively, other types of indicator lights may also be included in other embodiments, and the disclosed embodiment does not specifically limit this.

[0133] For example, in the present disclosure, detection may not be performed for an indicator light in an off state. If the indicator light base and the indicator light in a lit state are not detected, it may be considered that there is no indicator light in the surrounding area, so there is no need to perform the process of further identifying the indication information of the target object in S30. In addition, if the indicator light base is detected but the indicator light in a lit state is not detected, it may also be considered that there is an indicator light in an off state, and in this case, there is no need to identify the indication information of the target object.

[0134] S30: Based on the detection result of the target object, the target area where the target object is located in the input image is identified to obtain indication information of the target object.

[0135] In some possible implementations, when the detection result of the target object is obtained, the indication information of the target object can be further detected, wherein the indication information is used to describe the relevant attributes of the target object. In the field of intelligent driving, the indication information of the target object can be used to instruct the intelligent driving device to generate a control instruction based on the indication information. For example, for a target object of the type of a base, the arrangement of the indicator lights and at least one of the application scenarios can be identified. For a target object of the type of an indicator light in a lit state, at least one of the information such as the light color of the indicator light, the direction indicated by the arrow, and the numerical value of the number can be identified.

[0136] Based on the embodiments of the present disclosure, the base and the indicator light in the lighting state can be detected first, and the indication information of the target object can be further classified and identified based on the obtained detection results. That is, there is no need to directly classify and identify the type, position, and various types of indication information of the target object through a classifier. Instead, the classification and identification of the indication information is performed according to the detection results such as the type of the target object. This is beneficial to reducing the recognition complexity and difficulty in the process of identifying the indication information of the target object, and at the same time, it is also possible to simply and conveniently realize the detection and identification of various types of indicator lights in different situations.

[0137] The specific processes of the embodiments of the present disclosure are described below in conjunction with the accompanying drawings. Figure 3 A flowchart of step S20 in a method for identifying indication information of an indicator light according to an embodiment of the present disclosure is shown. Determining the detection result of the target object based on the input image (step S20) may include:

[0138] S21: extracting image features of the input image;

[0139] In some possible implementations, when an input image is obtained, feature extraction processing may be performed on the input image to obtain image features of the input image. The image features in the input image may be obtained by a feature extraction algorithm, or by extracting image features through a neural network that has been trained to achieve feature extraction. For example, the embodiment of the present disclosure may use a convolutional neural network to obtain image features of the input image, and the corresponding image features may be obtained by performing at least one layer of convolution processing on the input image, wherein the convolutional neural network may include at least one of a visual geometry group (Visual Geometry Group, VGG) network, a residual network, and a pyramid feature network, but this is not a specific limitation of the present disclosure, and image features may also be obtained by other means.

[0140] S22: determining a first position of each candidate region of at least one candidate region of the target object based on an image feature of the input image;

[0141] In some possible implementations, the location area where the target object is located in the input image can be detected based on the image features of the input image, that is, the first position of the candidate area of ​​each target object can be obtained. Among them, for each target object, at least one candidate area can be obtained, and the first position of each candidate area can be obtained accordingly. The first position of the embodiment of the present disclosure can be represented by the coordinates of the diagonal vertex position of the candidate area, but the present disclosure does not specifically limit this.

[0142] Figure 4 A schematic diagram of performing target detection according to an embodiment of the present disclosure is shown. Among them, the target detection network used to perform target detection may include a base network module, a region proposal network (RPN) module, and a classification module, wherein the base network module is used to perform feature extraction processing of the input image (image) to obtain image features of the input image. The region candidate network module is used to detect the candidate region (Region of Interest, ROI) of the target object in the input image based on the image features of the input image, and the classification module is used to judge the type of the target object in the candidate region based on the image features of the candidate region, and obtain the detection result of the target object in the target region (Box) in the input image. Exemplarily, the detection result of the target object includes the type of the target object and the position of the target region. The type of the target object is, for example, any one of a base, an indicator light in a lit state (such as a round spot light, an arrow light, a pedestrian light, a digital light), and a background. Among them, the background can be understood as the image area in the input image except the base and the area where the indicator light in the lit state is located.

[0143] In some possible implementations, the region candidate network may obtain at least one ROI for each target object in the input image, and may select the ROI with the highest accuracy therefrom through subsequent post-processing.

[0144] S23: Based on the image features at the first position corresponding to each candidate area in the input image, determine the intermediate detection result of each candidate area, the intermediate detection result includes the predicted type of the target object and the predicted probability that the target object is the predicted type; the predicted type is the indicator light base, any one of N types of indicator lights in the lighting state, where N is a positive integer.

[0145] When at least one candidate region (such as the first candidate region or the second candidate region) of each target object is obtained, the type information of the target object in the candidate region can be further classified and identified, that is, the predicted type of the target object in the candidate region and the predicted probability for the predicted type can be obtained. The predicted type can be one of the above-mentioned N+1 types, for example, it can be any one of a base, a circular spot light, an arrow light, a pedestrian light, and a digital light. In other words, it can be predicted whether the type of the target object in the candidate region is a base or one of the N types of indicator lights in the lighting state.

[0146] Among them, step S23 may include: for each candidate area, based on the image features at the first position corresponding to the candidate area, classifying the target object in the candidate area, and obtaining the predicted probability that the target object is each preset type in at least one preset type; wherein the preset types include an indicator light base, and at least one of N types of indicator lights in lighting states, and N is a positive integer; taking the preset type with the highest predicted probability among at least one preset type as the predicted type of the target object in the candidate area, and obtaining the predicted probability of the predicted type.

[0147] In some possible implementations, when at least one candidate region of each target object is obtained, an image feature corresponding to the first position in the image feature of the input image can be obtained according to the first position of the candidate region, and the obtained image feature is determined as the image feature of the candidate region. Furthermore, the prediction probability of the target object in each candidate region being each preset type can be predicted according to the image feature of each candidate region.

[0148] For each candidate area, classification recognition can be performed on the image features in the candidate area, and correspondingly, the predicted probability of each candidate area for each preset type can be obtained, wherein the preset type is the above-mentioned N+1 types, such as the base and N types of indicator light types. Alternatively, in other embodiments, the preset type can also be N+2 types, which further includes a background type relative to the N+1 type, but the present disclosure does not specifically limit this.

[0149] When the prediction probabilities of the target object in the candidate area being of each preset type are obtained, the preset type with the highest prediction probability can be determined as the prediction type of the target object in the candidate area, and the corresponding highest prediction probability is the prediction probability of the corresponding prediction type.

[0150] In some possible implementations, before performing type classification detection on the target object in the candidate area, the image features of each candidate area may be pooled so that the scale of the image features of each candidate area is the same. For example, for each ROI, the size of the image features may be scaled to 7*7, but the present disclosure does not specifically limit this. After the pooling process, the pooled image features may be classified to obtain an intermediate detection result corresponding to each candidate box of each target object.

[0151] In some possible implementations, the classification process of the image features of each candidate region in step S23 may be implemented using one classifier or multiple classifiers. For example, the predicted probability of the candidate region for each preset type may be obtained according to one classifier, or N+1 or N+2 classifiers may be used to detect the predicted probability of the candidate region for each type, respectively. The N+1 or N+2 classifiers correspond to the preset types one by one, that is, each classifier may be used to obtain the preset result of the corresponding preset type.

[0152] In some possible implementations, when performing classification processing on the candidate area, the image features (or pooled image features) of the candidate area can also be input into the first convolutional layer using a convolution layer to perform convolution processing, and a first feature map with a dimension of a×b×c is obtained, where b and c represent the length and width of the first feature map, respectively, a represents the number of channels of the first feature map, and the value of a is the total number of preset types (such as N+1). Then, a global pooling process is performed on the first feature map to obtain a second feature map corresponding to the first feature map, and the dimension of the second feature map is a×d. The second feature map is input into the softmax function, and a third feature map with a dimension of a×d can also be obtained, where d is an integer greater than or equal to 1. In one example, d represents the number of columns of the third feature map, such as 1. The corresponding elements in the third feature map represent the predicted probability that the target object in the candidate area is of each preset type. The numerical value corresponding to each element can be the probability value of the predicted probability. The order of the probability values ​​corresponds to the order of the set preset types, or each element in the third feature map can be composed of the preset type identifier and the corresponding predicted probability, so as to conveniently determine the correspondence between the preset type and the predicted probability.

[0153] In another example, d may also be another integer value greater than 1, and the prediction probability corresponding to the preset type may be obtained according to the elements of the first preset number of columns in the third feature map. The first preset number of columns may be a preset value, such as 1, but it is not a specific limitation of the present disclosure.

[0154] Through the above configuration, an intermediate detection result of each candidate region of each target object can be obtained, and further, a detection result of each target object can be obtained by using the intermediate detection result.

[0155] S24: Determine a detection result of the target object based on the intermediate detection result of each candidate area in the at least one candidate area and the first position of each candidate area.

[0156] As described in the above embodiments, intermediate detection results corresponding to all candidate areas of each target object (such as the first position of the candidate area, the predicted type and predicted probability of the target object within the candidate area) can be obtained. Furthermore, the final detection result of the target object, that is, the position, type and other information of the candidate area of ​​the target object, can be determined based on the intermediate detection result of each candidate area of ​​the target object.

[0157] It should be noted here that the embodiment of the present disclosure can use the first position of the candidate area of ​​each target object as the position of the candidate area, or can optimize the first position to obtain a more accurate first position. Among them, the embodiment of the present disclosure can also obtain the position deviation of the corresponding candidate area through the image features of each candidate area, and adjust the first position of the candidate area according to the position deviation. Among them, the image features of the candidate area of ​​each target object can be input into the second convolution layer to obtain a fourth feature map, and the dimension of the fourth feature map is e×b×c, wherein b and c represent the length and width of the fourth feature map and the third feature map, respectively, and b and c can also be the length and width of the image features of the candidate area, and e represents the number of channels of the fourth feature map, and e can be an integer greater than or equal to 1, for example, e can be 4. Further, by performing global pooling processing on the fourth feature map, a fifth feature map can be obtained, and the fifth feature map can be a feature vector with a length of e, such as e equals 4, at this time, the elements in the fifth feature map are the position deviations corresponding to the corresponding candidate areas. Alternatively, in other embodiments, the dimension of the fifth feature map can be e×f, and f is a value greater than or equal to 1, indicating the number of columns of the fifth feature map. At this time, the position deviation of the candidate area can be obtained according to the elements in the preset position area in the third feature map, wherein the preset position area can be a pre-set position area, such as the elements in the 1st column of the 1st row to the 4th row, but it is not specifically limited to the present disclosure.

[0158] Exemplarily, the first position of the candidate area can be represented as the horizontal and vertical coordinate values ​​of the two diagonal vertex positions, and the elements in the fifth feature map can be the position offsets of the horizontal and vertical coordinate values ​​of the two vertices. After obtaining the fifth feature map, the first position of the candidate area can be adjusted according to the corresponding position deviation in the fifth feature map to obtain a first position with higher accuracy. Among them, the first convolution layer and the second convolution layer are two different convolution layers.

[0159] Since at least one candidate region can be detected for each target object in the input image during the detection of the target object, the embodiment of the present disclosure can filter out the target region of the target object from the at least one candidate region.

[0160] In the case that only one candidate region is detected for any target object of the input image, it can be determined whether the prediction probability of the prediction type of the target object determined based on the candidate region is greater than the probability threshold. If it is greater than the probability threshold, the candidate region can be determined as the target region of the target object, and the prediction type corresponding to the candidate region is determined as the type of the target object. If the prediction probability of the prediction type of the target object determined based on the candidate region is less than the probability threshold, the candidate region is discarded, and it is determined that the object in the candidate region does not contain any target object to be detected.

[0161] Alternatively, when multiple candidate areas are detected for one or more target objects in the input image, the target area can be screened out from the multiple candidate areas based on the intermediate detection result of each candidate area, or based on the first position of each candidate area based on the intermediate detection result of each candidate area, and the predicted type of the target object in the target area is used as the type of the target object, and the first position of the target area is used as the position of the target area where the target object is located, to obtain the detection result of the target object.

[0162] Exemplarily, the step of filtering out the target area based on the intermediate detection results of the candidate areas may include: selecting a candidate area with the highest corresponding prediction probability from multiple candidate areas of the target object, and when the highest prediction probability is greater than a probability threshold, using the first position (or adjusted first position) of the candidate area corresponding to the highest prediction probability as the target area of ​​the target object, and determining the prediction type corresponding to the highest prediction probability as the type of the target object.

[0163] Exemplarily, the step of screening out the target area of ​​the target object based on the first position of the candidate area may include: selecting the target area of ​​the target object from multiple candidate areas using a non-maximum suppression algorithm (NMS). Among them, a candidate area with the largest prediction probability can be selected from multiple candidate areas of the target object in the input image, hereinafter referred to as the first candidate area. And according to the first position of the first candidate area and the first positions of the remaining candidate areas, the overlapping area values ​​(Intersection over Union, IOU) between the remaining candidate areas and the first candidate area are determined. If the IOU between any candidate area in the remaining candidate areas and the first candidate area is greater than the area threshold, the any candidate area is discarded. If the remaining candidate areas are discarded after comparing the IOU, the first candidate area is the target area of ​​the target object, and the prediction type of the target object obtained based on the first candidate area can be the type of the target object. If there is at least one second candidate region in the remaining candidate regions whose IOU value with the first candidate region is less than the area threshold, the candidate region with the highest prediction probability in the second candidate region can be re-used as the new first candidate region, and the remaining candidate regions in the second candidate region and the IOU of the new first candidate region are continuously obtained, and the second candidate regions whose IOU is greater than the area threshold are also discarded, until there is no candidate region whose IOU with the first candidate region (or the new candidate region) is greater than the area threshold. Each first candidate region obtained in the above manner can be determined as the target region of each target object.

[0164] Alternatively, in other possible implementations, the probability threshold can be used to screen out candidate areas with a prediction probability greater than the probability threshold from the candidate areas of each target object, and then the target area of ​​each target object is obtained through the above-mentioned NMS algorithm, and the prediction type for the target object in the target area is obtained, that is, the detection result of the target object is determined.

[0165] It should be noted that the above process of determining the detection result by executing the first position can also be performed by executing the detection result of the target object by executing the adjusted first position. The specific principles are the same and will not be repeated here.

[0166] Based on the above embodiment, the detection result of the target object present in the input image can be obtained, that is, the type of the target object and the corresponding position can be conveniently determined. Among them, through the above target detection, a detection frame (candidate area) for each target object (such as an indicator light in a lit state, an indicator light base) can be obtained. For example, for an indicator light in a lit state, the detection result may include the position of the indicator light in the input image and the type of the indicator light. For example, the detection result can be expressed as (x1, y1, x2, y2, label1, score1). Among them, (x1, y1), (x2, y2) are the position coordinates of the target area of ​​the indicator light in the lit state (the coordinates of the two diagonal points), label1 represents the type identification of the indicator light in the lit state (one of 1 to N+1, such as 2, which can be expressed as a digital light), and score1 represents the confidence level (i.e., the prediction probability) of the detection result.

[0167] For the indicator light base, the detection result is expressed as (x3, y3, x4, y4, label2, score12). Among them, (x3, y3), (x4, y4) are the position coordinates of the target area of ​​the base (the coordinates of the two diagonal points), label2 represents the type identification of the base (one of 1 to N, such as 1), and score2 represents the confidence of the detection result. The base identification can be 1, and the remaining N identifications can be N types of indicator lights in the lit state. In some possible implementations, N+2 can also be identified to represent the target area of ​​the background, and this disclosure does not specifically limit this.

[0168] Based on the above, the detection result for the target object can be obtained simply and conveniently. At the same time, since the detection result already includes the type information of the indicator light or the base, the classification pressure of the subsequent classifier can be reduced.

[0169] In some possible implementations, when the detection result of the target object in the input image is obtained, it is also possible to further determine whether the indicator light is faulty, or to collect information such as the acquisition environment of the input image based on the detection result. Among them, in the result of the target object of the input image, the type of the detected target object only includes the indicator light base, and does not include any type of indicator light in a lit state. At this time, it can be determined that the indicator light is in a faulty state. For example, in a traffic light, if no traffic light is detected to be in a lit state, it can be determined that the traffic light is a faulty light. At this time, a fault alarm operation can be performed based on the acquisition time, acquisition location and other information related to the input image. For example, fault information is sent to a server or other management device, and the fault information may include the fault condition of the indicator light not being lit, and the location information of the faulty light (determined based on the above-mentioned acquisition location).

[0170] Alternatively, in some embodiments, if the detection result of the target object detected for the input image only includes an indicator light in a lit state, but does not include a base corresponding to the indicator light in the lit state, it can be determined that the acquisition environment for acquiring the input image is a dark environment, or a dark state, where the dark state or dark environment refers to an environment where the brightness is less than a preset brightness, and the preset brightness can be set according to different locations or different weather conditions, and the present disclosure does not make specific limitations on this.

[0171] Figure 5 A flowchart of step S30 in a method for identifying indication information of an indicator light according to an embodiment of the present disclosure is shown, wherein based on the detection result of the target object, the target area where the target object is located in the input image is identified to obtain the indication information of the target object (step S30), which may include:

[0172] S31: Determine a classifier matching the target object based on the type of the target object in the detection result of the target object;

[0173] S32: using the matching classifier, identifying the image features of the target area in the input image to obtain indication information of the target object.

[0174] Exemplarily, the classifiers matched with the target object include at least one, and each classifier may correspond to one or more types of target objects.

[0175] In some possible implementations, after the detection result of the target object in the input image is obtained, the classification detection of the indication information can be performed, such as the classification and recognition of at least one of the scene information in the base, the arrangement of the indicator lights, the color, description, and indication direction of the indicator lights. In particular, the embodiment of the present disclosure can use different classifiers to perform classification and recognition of different indication information, so the classifier that performs classification and recognition can be determined first.

[0176] Figure 6 A schematic diagram showing classification detection of different target objects according to an embodiment of the present disclosure is shown.

[0177] In some possible implementations, when the type of the identified target object is a base of an indicator light, the target object of the base type may be further classified and identified for indication information to obtain an arrangement of the indicator lights and at least one indication information in a scene where the indicator lights are located. The arrangement may include horizontal arrangement, vertical arrangement, and an arrangement of a single indicator light, etc. The scene may include a highway intersection, a sharp turn, a general scene, etc. The above is only an exemplary description of the arrangement and scene, and may also include other arrangements or scenes, which are not specifically limited in the present disclosure.

[0178] In some possible implementations, when the type of the identified target object is a circular spot light in a lit state, the color of the circular spot light can be classified and identified to obtain indication information of the color of the light (such as red, green, and yellow). When the type of the identified target object is a digital indicator light in a lit state, it is necessary to classify and identify the numerical value (such as 1, 2, 3, etc.) and the color of the light to obtain indication information of the color of the light and the numerical value. When the type of the identified target object is an arrow indicator light in a lit state, it is necessary to classify and identify the direction indicated by the arrow (such as forward, left, right, etc.) and the color of the light to obtain indication information of the color of the light and the direction. When the type of the identified target object is an indicator light for pedestrian identification (pedestrian light), it is necessary to identify the color of the light to obtain indication information of the color of the light.

[0179] That is to say, the embodiment of the present disclosure can perform recognition of different indication information for different types of target objects in the target object detection results, so as to obtain the indication information of the indicator light more conveniently and accurately. When performing the recognition of the indication information, the image features corresponding to the target area where the corresponding type of target object is located can be input into the matching classifier to obtain the classification result, that is, the corresponding indication information.

[0180] Specifically, when the detection results of the target objects in the input image show that the type of at least one target object is a base, the determined matching classifier includes at least one of a first classifier and a second classifier, wherein the first classifier is used to classify and identify the arrangement of the indicator lights in the base, and the second classifier is used to classify and identify the scene where the indicator lights are located. The image features corresponding to the target area of ​​the base-type target object are input into the first classifier to obtain the arrangement of the indicator lights in the base, and the image features corresponding to the target area of ​​the base-type target object are input into the second classifier to obtain the scene of the indicator lights, for example, the scene information can be obtained by using text recognition.

[0181] In some possible implementations, when the type of the identified target object is a circular spot light or a pedestrian light in a lit state, it is determined that the matching classifier includes a third classifier for identifying the color attribute of the circular spot light or the pedestrian light. At this time, the image features of the target area corresponding to the target object of the circular spot light type or the pedestrian light type can be input into the matching third classifier to obtain the color attribute of the indicator light.

[0182] In some possible implementations, when the type of the identified target object is an arrow light in a lit state, the matching classifier is determined to include a fourth classifier for identifying the color attribute of the arrow light and a fifth classifier for identifying the direction attribute. At this time, the image features of the target area corresponding to the target object of the arrow light type can be input into the matching fourth classifier and fifth classifier, and the image features of the target area where the target object is located can be identified using the fourth classifier and the fifth classifier to obtain the color attribute of the arrow light and the direction attribute of the arrow light, respectively.

[0183] In some possible implementations, when the type of the identified target object is a digital light in a lit state, the matched classifiers include a sixth classifier for identifying the color attribute of the digital light and a seventh classifier for identifying the numerical attribute of the digital light. At this time, the image features of the target area corresponding to the target object of the digital light type can be input into the matching sixth classifier and seventh classifier, and based on the sixth classifier and the seventh classifier, the image features of the target area where the target object is located are identified to obtain the color attribute and the numerical direction attribute of the digital light respectively.

[0184] It should be noted that the third classifier, the fourth classifier and the sixth classifier for performing classification and recognition of color attributes may be the same classifier or different classifiers, and the present disclosure does not make any specific limitation on this.

[0185] In addition, in some possible implementations, the method for acquiring the image features of the target area may include determining the image features of the target area based on the image features of the input image obtained by extracting features from the input image, and the location information of the target area. That is, the features corresponding to the location information of the target area can be directly obtained from the image features of the input image as the image features of the target area. Alternatively, a sub-image corresponding to the target area in the input image may be obtained, and then feature extraction, such as convolution processing, may be performed on the sub-image to obtain the image features of the sub-image, thereby determining the image features of the target area. The above is only an exemplary description, and in other embodiments, the image features of the target area may also be obtained by other methods, and the present disclosure does not specifically limit this.

[0186] Through the above embodiment, the indication information of the target object in each target area can be obtained. Different classifiers can be used to detect different indication information, so that the classification result is more accurate. At the same time, based on the type of the target object, a matching classifier is further used for classification and identification, rather than using all classifiers for identification, which can effectively utilize the classifier resources and speed up the classification speed.

[0187] In some possible implementations, the input image may include a plurality of indicator light bases, a plurality of indicator lights in a lit state, Figure 7 A schematic diagram of a traffic light structure with multiple bases is shown. In the case where the detection results obtained include multiple indicator light bases and multiple indicator lights in a lit state, it is necessary to match the bases and the indicator lights in a lit state. Figure 7 The device includes two indicator light bases D1 and D2, and each indicator light base may include a corresponding indicator light. In the process of identifying the indication information, three lighted indicator lights, namely L1, L2 and L3, can be determined. By matching the indicator light bases with the indicator lights in the lighted state, it can be determined that the lighted indicator light L1 matches the indicator light base D1, and the indicator lights L2 and L3 match the base D2.

[0188] Figure 8 Another flow chart of a method for identifying indication information of an indicator light according to an embodiment of the present disclosure is shown, wherein the method for identifying indication information of an indicator light also includes a matching process of an indicator light base and an indicator light in a lit state, specifically:

[0189] S41: for a first indicator light base, determining an indicator light of a lighting state matching the first indicator light base; the first indicator light base is one of at least two indicator light bases;

[0190] Among them, the detection results of the target object obtained may include the first position of the target area of ​​the target object of the base type and the second position of the target area where the indicator light in the lit state is located. The embodiment of the present disclosure can determine whether the base and the indicator light in the lit state match based on the first position of each base and the second position of each indicator light.

[0191] Among them, based on the position of the target area where the target object is located in the detection result of the target object, a first area of ​​the intersection between the target area where at least one indicator light in the lit state is located and the target area where the first indicator light base is located can be determined, and a second area of ​​the target area where at least one indicator light in the lit state is located can be determined; in response to the ratio between the first area corresponding to the first indicator light in the lit state and the second area of ​​the first indicator light in the lit state being greater than a set area threshold, it is determined that the first indicator light in the lit state matches the first indicator light base; wherein the first indicator light in the lit state is one of the at least one indicator light in the lit state.

[0192] That is to say, for each first indicator light base, the first area S1 that intersects or overlaps between each base and the target area of ​​each indicator light can be determined based on the first position of the target area of ​​the first indicator light base and the second position of the target area of ​​each indicator light in the lit state. If there is a ratio (S1 / S2) between the first area S1 between the indicator light in the lit state (first indicator light) and the indicator light base and the second area S2 of the target area of ​​the indicator light in the lit state that is greater than the area threshold, it can be determined that the first indicator light matches the first indicator light base. If multiple first indicator lights that match the first indicator light base can be determined, the multiple first indicator lights can be used as indicator lights that match the first indicator light base at the same time, or the first indicator light with the largest ratio can be determined as the indicator light in the lit state that matches the first indicator light base. Alternatively, the preset number of indicator lights with the largest ratio of the above S1 / S2 between the first indicator light base and the first indicator light base can be determined as the indicator lights that match the first indicator light base. The preset number can be 2, but it is not a specific limitation of the present disclosure. In addition, the area threshold can be a preset value, such as 0.8, but it is not a specific limitation of the present disclosure.

[0193] S42: Combine the indication information of the first indicator light base and the indication information of the indicator light in the lighting state that matches the first indicator light base to obtain combined indication information.

[0194] After obtaining the indicator light in the lighting state that matches the indicator light base, the indication information obtained by the indicator light base and the matching indicator light in the lighting state can be combined to obtain the indication information of the indicator light. Figure 7 As shown, the indication information of the indicator light base D1 and the indicator light L1 in the lighting state can be combined, and the determined indication information includes that the scene is a general scene, the arrangement of the indicator lights is a horizontal arrangement, the indicator lights in the lighting state are round spot lights, and the color is red. At the same time, the indication information of the indicator light base D2 and the indicator lights L2 and L3 in the lighting state can also be combined, and the determined indication information includes that the scene is a general scene, the arrangement of the indicator lights is a horizontal arrangement, the indicator lights in the lighting state are arrow lights, and include right arrow lights and forward arrow lights, wherein the color of the right arrow light is red, and the color of the forward arrow light is green.

[0195] In addition, for an indicator light base that cannot find a corresponding indicator light in a lighting state, the base can be determined to be in an off state. That is, the indicator light corresponding to the base can be determined to be a fault light. For an indicator light in a lighting state that cannot find a matching indicator light base, the indication information corresponding to the indicator light in the lighting state is output separately. This situation is often caused by the unclear visual features of the base, such as the situation where it is difficult to detect the base at night.

[0196] In addition, in the field of intelligent driving, the input image obtained can be an image of the front or rear of the vehicle collected in real time. When the indication information corresponding to the indicator light in the input image is obtained, the control instructions of the driving parameters of the driving device can be further generated based on the obtained indication information. The driving parameters may include driving speed, driving direction, control mode, stop and other driving states.

[0197] In order to more clearly reflect the embodiment of the present disclosure, the following example illustrates the process of obtaining indication information in the embodiment of the present disclosure. The algorithm model adopted in the embodiment of the present disclosure may include two parts, one of which is a reference Figure 4 The target detection network shown in the figure performs target detection, and the other part is a classification network that performs classification and recognition of indication information. Figure 4 As shown, the target detection network may include a base network module, a region candidate network (RPN) module, and a classification module, wherein the base network module is used to perform feature extraction processing of the input image to obtain image features of the input image. The region candidate network module is used to detect the candidate region (ROI) of the target object in the input image based on the image features of the input image, and the classification module is used to judge the type of the target object in the candidate region based on the image features of the candidate region to obtain the detection result of the target object in the input image.

[0198] The input of the target detection network is the input image, and the output is the 2D detection boxes of several target objects (i.e., the target area of ​​the target object). Each detection box can be represented as (x1, y1, x2, y2, label, score). x1, y1, x2, y2 are the position coordinates of the detection box, and label is the category (the value range is 1 to N+1, the first category represents the base, and the other categories represent indicator lights of various lighting states).

[0199] The process of target detection may include: inputting the input image into the Base Network to obtain the image features of the input image. Use the Region Proposal Network (RPN) to generate the candidate frame ROI (Region of interest) of the indicator light, which includes the candidate frame of the base and the candidate frame of the indicator light in the lit state. Then the pooling layer can be used to obtain the feature map of the candidate frame of a fixed size. For example, for each ROI, the size of the feature map is scaled to 7*7, and then the classification module is used to determine the category of N+2 categories (adding the background category) to obtain the predicted type and position of the candidate frame of each target object in the input image. Then perform post-processing such as NMS and thresholding to obtain the final detection frame of the target object (the candidate frame corresponding to the target area).

[0200] The rationality of the embodiment of the present disclosure classifying the indicator lights in the detected target objects into N categories is as follows:

[0201] 1. The meanings of indicator lights in different lighting states are not the same, and it is often necessary to study the test results of each type separately. For example, pedestrian lights and vehicle circular spot lights cannot be confused.

[0202] 2. There is a serious problem of imbalance in the number of samples between indicator lights of different types. The indicator lights of different types are divided into N different categories to facilitate the adjustment of model parameters and optimize them separately.

[0203] When the detection result of each target object is obtained, the indication information of the target object can be further identified. The indication information can be classified and identified by a matching classifier. The identification of the indication information of the target object can be performed using a classification module including multiple classifiers. The classification module can include multiple types of classifiers for performing classification and identification of different indication information, or can also include a convolution layer for extracting features, which is not specifically limited in the present disclosure.

[0204] The input of the classification module may be the image features corresponding to the target area of ​​the target object obtained by detection, and the output may be the indication information corresponding to each target object in the target area.

[0205] The specific process may include: inputting a detection frame of the target area of ​​the target object, selecting a classifier that matches the type (1 to N+1) of the target object in the detection frame, and obtaining a corresponding classification result. If it is a detection frame of the indicator light base, since the indicator light base is regarded as a simple whole, all the classifiers of the indicator light base are activated, for example, all the classifiers used to identify scenes and arrangement methods are activated to identify scene attributes and arrangement method attributes; if it is a detection frame of an indicator light in a lit state, different types of indicator lights in a lit state need to select different classifiers, for example, arrow lights correspond to the "color" and "arrow direction" classifiers, circular spot lights correspond to the "color" classifier, and so on. In addition, if there is a need for other attribute determination, other classifiers may be added, and the present disclosure does not specifically limit this.

[0206] To sum up, the embodiments of the present disclosure can first perform target detection processing on the input image to obtain a target object detection result, wherein the target object detection result may include information such as the location and type of the target object, and then further perform recognition of the target object's indication information based on the target object detection result.

[0207] The present invention divides the detection process of the target object into two detection processes, namely, the detection process of the base and the indicator light in the lighting state, thereby realizing the initial distinction of the target object in the detection process. When further identification is subsequently performed based on the detection result of the target object, it is beneficial to reduce the recognition complexity in the process of identifying the indication information of the target object, reduce the recognition difficulty, and can simply and conveniently realize the detection and identification of various types of indicator lights in different situations.

[0208] In addition, the embodiment of the present disclosure only uses image information without using other sensors to achieve the detection of indicator lights and the determination of indication information. At the same time, the embodiment of the present disclosure can detect different types of indicator lights and has better applicability.

[0209] Fig. 9 A flow chart of a driving control method according to an embodiment of the present disclosure is shown, and the driving control method can be applied to devices such as smart vehicles, smart aircraft, toys, etc. that can adjust driving parameters according to control instructions. The driving control method may include:

[0210] S100: using an image acquisition device in an intelligent driving device to acquire a driving image;

[0211] During the driving process of the intelligent driving device, an image acquisition device in the intelligent driving device may be provided to acquire driving images, or an image of the driving position acquired from other devices may be received.

[0212] S200: executing the indicator light indication information recognition method on the driving image to obtain indication information for the driving image;

[0213] The indication information detection process is performed on the driving image, that is, the indicator light indication information recognition method described in the above embodiment is performed to obtain the indication information of the indicator light in the driving image.

[0214] S300: Generate a control instruction for an intelligent driving device using the instruction information.

[0215] Based on the obtained indication information, the driving parameters of the driving device can be controlled in real time, that is, a control instruction for controlling the functional driving device can be generated according to the obtained indication information, and the control instruction can be used to control the driving parameters of the intelligent driving device, and the driving parameters may include at least one of the driving speed, driving direction, driving mode or driving state. For the parameter control of the driving device or the type of control instruction, those skilled in the art can set it according to the existing technical means and requirements, and this disclosure does not specifically limit this.

[0216] Based on the embodiments of the present disclosure, intelligent control of intelligent driving equipment can be achieved. Since the process of obtaining indication information is simple, fast and highly accurate, the control efficiency and accuracy of the intelligent driving equipment can be improved.

[0217] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0218] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not go into details.

[0219] In addition, the present disclosure also provides an indicator light indication information recognition device, a driving control device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any indicator light indication information recognition method and / or driving control method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0220] Fig.10 A block diagram of a device for identifying indication information of an indicator light according to an embodiment of the present disclosure is shown. Fig.10 As shown, the indication information recognition device of the indicator light includes:

[0221] An acquisition module 10, which is used to acquire an input image;

[0222] A determination module 20, which is used to determine a detection result of a target object based on the input image, the target object includes at least one of an indicator light base and an indicator light in a lit state, and the detection result includes a type of the target object and a position of a target area where the target object is located in the input image;

[0223] The recognition module 30 is used to recognize the target area where the target object is located in the input image based on the detection result of the target object, and obtain the indication information of the target object.

[0224] In some possible implementations, the determining module is further configured to:

[0225] Extracting image features of the input image;

[0226] Determining a first position of each candidate region of at least one candidate region of the target object based on an image feature of the input image;

[0227] Based on the image features at the first position corresponding to each candidate area in the input image, an intermediate detection result of each candidate area is determined, wherein the intermediate detection result includes a predicted type of the target object and a predicted probability that the target object is the predicted type; the predicted type is an indicator light base and any one of N indicator lights in lighting states, where N is a positive integer;

[0228] A detection result of the target object is determined based on an intermediate detection result of each candidate area in the at least one candidate area and a first position of each candidate area.

[0229] In some possible implementations, the determination module is further used to: for each candidate area, based on the image feature at the first position corresponding to the candidate area, classify the target object in the candidate area to obtain a predicted probability that the target object is each of the at least one preset type; wherein the preset type includes at least one of an indicator light base and indicator lights in N lighting states, where N is a positive integer;

[0230] The preset type with the highest prediction probability among the at least one preset type is used as the prediction type of the target object in the candidate area, and the prediction probability of the prediction type is obtained.

[0231] In some possible implementations, the determination module is further configured to: determine a position deviation of the first position of each candidate region based on an image feature of the input image before determining the detection result of the target object based on the intermediate detection result of each candidate region in the at least one candidate region and the first position of each candidate region;

[0232] The first position of each candidate region is adjusted using the position deviation corresponding to each candidate region.

[0233] In some possible implementations, the determination module is further configured to, when there are at least two candidate regions for the target object, screen out the target region from the at least two candidate regions based on an intermediate detection result of each of the at least two candidate regions, or based on the intermediate detection result of each candidate region and a first position of each candidate region;

[0234] The predicted type of the target object in the target area is used as the type of the target object, and the first position of the target area is used as the position of the target area where the target object is located, so as to obtain a detection result of the target object.

[0235] In some possible implementations, the determination module is further configured to determine that the indicator light is in a fault state when the detection result of the target object only includes the detection result corresponding to the indicator light base;

[0236] In a case where the detection results of the target object only include detection results corresponding to indicator lights in a lit state, it is determined that the scene state of collecting the input image is a dark state.

[0237] In some possible implementations, the recognition module is further used to determine a classifier matching the target object based on the type of the target object in the detection result of the target object;

[0238] The image features of the target area in the input image are identified by using a matching classifier to obtain indication information of the target object.

[0239] In some possible implementations, the recognition module is further used to, when the type of the target object is an indicator light base, determine that the matching classifier includes a first classifier for identifying an arrangement of the indicator lights in the indicator light base, and use the first classifier to identify image features of a target area where the target object is located to determine the arrangement of the indicator lights in the indicator light base; and / or,

[0240] The matched classifier includes a second classifier for identifying the scene where the indicator light is located. The second classifier is used to identify image features of the target area where the target object is located to determine scene information where the indicator light is located.

[0241] In some possible implementations, the recognition module is further configured to, when the type of the target object is a circular spot light or a pedestrian light, determine that the matching classifier includes a third classifier for identifying a color attribute of the circular spot light;

[0242] The third classifier is used to identify the image features of the target area where the target object is located, and determine the color attribute of the circular spot light or the pedestrian light.

[0243] In some possible implementations, the recognition module is further configured to, when the type of the target object is an arrow light, determine that the matching classifier includes a fourth classifier for identifying a color attribute of the arrow light and a fifth classifier for identifying a direction attribute;

[0244] The fourth classifier and the fifth classifier are used to identify the image features of the target area where the target object is located, and the color attribute and the direction attribute of the arrow light are determined respectively.

[0245] In some possible implementations, the recognition module is further used to, when the type of the target object is a digital light, determine that the matching classifier includes a sixth classifier for identifying a color attribute of the digital light and a seventh classifier for identifying a numerical attribute;

[0246] Based on the sixth classifier and the seventh classifier, the image features of the target area where the target object is located are identified, and the color attribute and the numerical attribute of the digital light are determined respectively.

[0247] In some possible implementations, the device further includes a matching module, which is used to determine, for a first indicator light base, an indicator light in a lighting state that matches the first indicator light base when the input image includes at least two indicator light bases; the first indicator light base is one of the at least two indicator light bases;

[0248] The indication information of the first indicator light base and the indication information of the indicator light in the lighting state that matches the first indicator light base are combined to obtain the combined indication information.

[0249] In some possible implementations, the matching module is further used for:

[0250] Based on the position of the target area where the target object is located in the detection result of the target object, determine a first area of ​​intersection between the target area where the at least one indicator light in the lit state is located and the target area where the first indicator light base is located, and determine a second area of ​​the target area where the at least one indicator light in the lit state is located;

[0251] In the case where a ratio of the first area between the first indicator light in the lit state and the first indicator light base to the second area of ​​the first indicator light in the lit state is greater than a set area threshold, determining that the first indicator light in the lit state matches the first indicator light base;

[0252] The first indicator light in the lighting state is one of the at least one indicator light in the lighting state.

[0253] in addition, Fig.11 A block diagram of a driving control device according to an embodiment of the present disclosure is shown; the driving control device comprises:

[0254] An image acquisition module 100, which is provided in the intelligent driving device and is used to acquire a driving image of the intelligent driving device;

[0255] An image processing module 200, which is used to perform the indicator light indication information recognition method as described in any one of the first aspects on the driving image to obtain indication information for the driving image;

[0256] The control module 300 is used to generate a control instruction for controlling the intelligent driving device using the indication information.

[0257] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0258] The embodiment of the present disclosure also provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0259] The embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured as the above method.

[0260] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0261] Fig.12 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0262] Reference Fig.12 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0263] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0264] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0265] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0266] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0267] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0268] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0269] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0270] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0271] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0272] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above method.

[0273] Fig.13 Another block diagram of an electronic device according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server. Fig.13, the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0274] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0275] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0276] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0277] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0278] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0279] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0280] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0281] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0282] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0283] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0284] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for identifying indication information of an indicator light, characterized in that: include: Get the input image; Based on the input image, determining a detection result of a target object, the target object comprising at least one of an indicator light base and an indicator light in a lit state, the detection result comprising a type of the target object and a position of a target area where the target object is located in the input image; Based on the detection result of the target object, identifying the target area where the target object is located in the input image, and obtaining indication information of the target object; Wherein, in response to the input image including at least two indicator light bases, the method further includes: For a first indicator light base, determining an indicator light of a lighting state matching the first indicator light base; the first indicator light base is one of the at least two indicator light bases; combining the indication information of the first indicator light base and the indication information of the indicator light in the lighting state that matches the first indicator light base to obtain combined indication information; The step of determining an indicator light of a lighting state matching the first indicator light base comprises: Based on the position of the target area where the target object is located in the detection result of the target object, determine a first area of ​​intersection between the target area where the at least one indicator light in the lit state is located and the target area where the first indicator light base is located, and determine a second area of ​​the target area where the at least one indicator light in the lit state is located; In response to a ratio between the first area between the first indicator light in the illuminated state and the first indicator light base and the second area of ​​the first indicator light in the illuminated state being greater than a set area threshold, determining that the first indicator light in the illuminated state matches the first indicator light base; The first indicator light in the lighting state is one of the at least one indicator light in the lighting state.

2. The method according to claim 1, characterized in that The step of determining a detection result of the target object based on the input image includes: Extracting image features of the input image; Determining a first position of each candidate region of at least one candidate region of the target object based on an image feature of the input image; Based on the image features at the first position corresponding to each candidate area in the input image, an intermediate detection result of each candidate area is determined, wherein the intermediate detection result includes a predicted type of the target object and a predicted probability that the target object is the predicted type; the predicted type is an indicator light base and any one of N indicator lights in lighting states, where N is a positive integer; A detection result of the target object is determined based on an intermediate detection result of each candidate area in the at least one candidate area and a first position of each candidate area.

3. The method according to claim 2, characterized in that Determining an intermediate detection result of each candidate region based on an image feature at a first position corresponding to each candidate region in the input image includes: For each candidate area, based on the image feature at the first position corresponding to the candidate area, the target object in the candidate area is classified to obtain a predicted probability that the target object is each of the at least one preset type; wherein the preset type includes at least one of an indicator light base and indicator lights in N lighting states, where N is a positive integer; The preset type with the highest prediction probability among the at least one preset type is used as the prediction type of the target object in the candidate area, and the prediction probability of the prediction type is obtained.

4. The method according to claim 2 or 3, characterized in that: Before determining the detection result of the target object based on the intermediate detection result of each candidate area in at least one candidate area and the first position of each candidate area, the method further includes: Determining a position deviation of a first position of each candidate region based on an image feature of the input image; The first position of each candidate region is adjusted using the position deviation corresponding to each candidate region.

5. The method according to claim 2 or 3, characterized in that: The determining the detection result of the target object based on the intermediate detection result of each candidate area in the at least one candidate area and the first position of each candidate area includes: In response to there being at least two candidate regions for the target object, based on an intermediate detection result of each of the at least two candidate regions, or based on the intermediate detection result of each candidate region and a first position of each candidate region, screening out the target region from the at least two candidate regions; The predicted type of the target object in the target area is used as the type of the target object, and the first position of the target area is used as the position of the target area where the target object is located, so as to obtain a detection result of the target object.

6. The method according to any one of claims 1 to 3, characterized in that: After determining the detection result of the target object based on the input image, the method further includes at least one of the following: In response to the detection results of the target object only including the detection results corresponding to the indicator light base, determining that the indicator light is in a fault state; In response to the detection results of the target object only including the detection results corresponding to the indicator lights in the lit state, it is determined that the scene state of collecting the input image is a dark state.

7. The method according to any one of claims 1 to 3, characterized in that The step of identifying a target area where the target object is located in the input image based on the detection result of the target object to obtain indication information of the target object includes: Determining a classifier that matches the target object based on the type of the target object in the detection result of the target object; The image features of the target area in the input image are identified by using a matching classifier to obtain indication information of the target object.

8. The method according to claim 7, characterized in that The step of identifying a target area where the target object is located in the input image based on the detection result of the target object to obtain indication information of the target object includes: In response to the type of the target object being an indicator light base, determining that the matching classifier includes a first classifier for identifying an arrangement of indicator lights in the indicator light base, using the first classifier to identify image features of a target area where the target object is located, and determining an arrangement of the indicator lights in the indicator light base; and / or, The matched classifier includes a second classifier for identifying the scene where the indicator light is located. The second classifier is used to identify image features of the target area where the target object is located to determine scene information where the indicator light is located.

9. The method according to claim 7, characterized in that: The step of identifying a target area where the target object is located in the input image based on the detection result of the target object to obtain indication information of the target object includes: In response to the type of the target object being a circular spot light or a pedestrian light, determining that the matching classifier includes a third classifier for identifying a color attribute of the circular spot light; The third classifier is used to identify the image features of the target area where the target object is located, and determine the color attribute of the circular spot light or the pedestrian light.

10. The method according to claim 7, characterized in that The step of identifying a target area where the target object is located in the input image based on the detection result of the target object to obtain indication information of the target object includes: In response to the type of the target object being an arrow light, determining that the matching classifier includes a fourth classifier for identifying a color attribute of the arrow light and a fifth classifier for identifying a direction attribute; The fourth classifier and the fifth classifier are used to identify the image features of the target area where the target object is located, and the color attribute and the direction attribute of the arrow light are determined respectively.

11. The method according to claim 7, characterized in that The step of identifying a target area where the target object is located in the input image based on the detection result of the target object to obtain indication information of the target object includes: In response to the type of the target object being a digital light, determining that the matching classifiers include a sixth classifier for identifying a color attribute of the digital light and a seventh classifier for identifying a numerical attribute; Based on the sixth classifier and the seventh classifier, the image features of the target area where the target object is located are identified, and the color attribute and the numerical attribute of the digital light are determined respectively.

12. A driving control method, characterized in that: include: Using the image acquisition device in the intelligent driving device to collect driving images; Executing the indicator light indication information recognition method according to any one of claims 1 to 11 on the driving image to obtain indication information for the driving image; The instruction information is used to generate a control instruction for the intelligent driving device.

13. A device for identifying indication information of an indicator light, characterized in that: include: An acquisition module, which is used to acquire an input image; a determination module, configured to determine a detection result of a target object based on the input image, wherein the target object includes at least one of an indicator light base and an indicator light in a lit state, and the detection result includes a type of the target object and a position of a target area where the target object is located in the input image; an identification module, which is used to identify the target area where the target object is located in the input image based on the detection result of the target object, and obtain indication information of the target object; The device further comprises a matching module, which is used to determine, for a first indicator light base, an indicator light in a lighting state that matches the first indicator light base when at least two indicator light bases are included in the input image; the first indicator light base is one of the at least two indicator light bases; combining the indication information of the first indicator light base and the indication information of the indicator light in the lighting state that matches the first indicator light base to obtain combined indication information; The matching module is also used for: Based on the position of the target area where the target object is located in the detection result of the target object, determine a first area of ​​intersection between the target area where the at least one indicator light in the lit state is located and the target area where the first indicator light base is located, and determine a second area of ​​the target area where the at least one indicator light in the lit state is located; In the case where a ratio of the first area between the first indicator light in the lit state and the first indicator light base to the second area of ​​the first indicator light in the lit state is greater than a set area threshold, determining that the first indicator light in the lit state matches the first indicator light base; The first indicator light in the lighting state is one of the at least one indicator light in the lighting state.

14. The device according to claim 13, characterized in that The determining module is also used for: Extracting image features of the input image; Determining a first position of each candidate region of at least one candidate region of the target object based on an image feature of the input image; Based on the image features at the first position corresponding to each candidate area in the input image, an intermediate detection result of each candidate area is determined, wherein the intermediate detection result includes a predicted type of the target object and a predicted probability that the target object is the predicted type; the predicted type is an indicator light base and any one of N indicator lights in lighting states, where N is a positive integer; A detection result of the target object is determined based on an intermediate detection result of each candidate area in the at least one candidate area and a first position of each candidate area.

15. The device according to claim 14, characterized in that The determination module is further used to: for each candidate area, based on the image feature at the first position corresponding to the candidate area, classify the target object in the candidate area to obtain the predicted probability that the target object is each preset type of the at least one preset type; wherein the preset type includes at least one of an indicator light base and indicator lights of N lighting states, where N is a positive integer; The preset type with the highest prediction probability among the at least one preset type is used as the prediction type of the target object in the candidate area, and the prediction probability of the prediction type is obtained.

16. The device according to claim 14 or 15, characterized in that The determination module is further configured to: determine a position deviation of the first position of each candidate region based on an image feature of the input image before determining the detection result of the target object based on the intermediate detection result of each candidate region in the at least one candidate region and the first position of each candidate region; The first position of each candidate region is adjusted using the position deviation corresponding to each candidate region.

17. The device according to claim 14 or 15, characterized in that The determination module is further configured to, when there are at least two candidate regions for the target object, screen out the target region from the at least two candidate regions based on an intermediate detection result of each of the at least two candidate regions, or based on the intermediate detection result of each candidate region and a first position of each candidate region; The predicted type of the target object in the target area is used as the type of the target object, and the first position of the target area is used as the position of the target area where the target object is located, so as to obtain a detection result of the target object.

18. The device according to any one of claims 13 to 15, characterized in that The determination module is further configured to determine that the indicator light is in a fault state when the detection result of the target object only includes the detection result corresponding to the indicator light base; In a case where the detection results of the target object only include detection results corresponding to indicator lights in a lit state, it is determined that the scene state of collecting the input image is a dark state.

19. The device according to any one of claims 13 to 15, characterized in that The recognition module is further used to determine a classifier matching the target object based on the type of the target object in the detection result of the target object; The image features of the target area in the input image are identified by using a matching classifier to obtain indication information of the target object.

20. The device according to claim 19, characterized in that The recognition module is further used to, when the type of the target object is an indicator light base, determine that the matching classifier includes a first classifier for identifying an arrangement of indicator lights in the indicator light base, and use the first classifier to identify image features of a target area where the target object is located to determine an arrangement of the indicator lights in the indicator light base; and / or, The matched classifier includes a second classifier for identifying the scene where the indicator light is located. The second classifier is used to identify image features of the target area where the target object is located to determine scene information where the indicator light is located.

21. The device according to claim 19, characterized in that The recognition module is further used to determine, when the type of the target object is a circular spot light or a pedestrian light, that the matching classifier includes a third classifier for identifying a color attribute of the circular spot light; The third classifier is used to identify the image features of the target area where the target object is located, and determine the color attribute of the circular spot light or the pedestrian light.

22. The device according to claim 19, characterized in that The recognition module is further used for determining, when the type of the target object is an arrow light, that the matching classifier includes a fourth classifier for identifying a color attribute of the arrow light and a fifth classifier for identifying a direction attribute; The fourth classifier and the fifth classifier are used to identify the image features of the target area where the target object is located, and the color attribute and the direction attribute of the arrow light are determined respectively.

23. The device according to claim 19, characterized in that The recognition module is further used for, when the type of the target object is a digital light, determining that the matching classifier includes a sixth classifier for identifying the color attribute of the digital light and a seventh classifier for identifying the numerical attribute; Based on the sixth classifier and the seventh classifier, the image features of the target area where the target object is located are identified, and the color attribute and the numerical attribute of the digital light are determined respectively.

24. A driving control device, characterized in that: include: An image acquisition module, which is provided in the intelligent driving device and is used to acquire a driving image of the intelligent driving device; An image processing module, configured to execute the indicator light indication information recognition method according to any one of claims 1 to 11 on the driving image to obtain indication information for the driving image; A control module is used to generate control instructions for the intelligent driving device using the indication information.

25. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 11, or to execute the method described in claim 12.

26. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method described in any one of claims 1 to 11 is implemented, or the method described in claim 12 is implemented.

Images