Traffic signal lamp detection method and device, computer equipment and readable storage medium

Abstract

The invention discloses a traffic signal lamp detection method and device, computer equipment and a readable storage medium, and the method is realized through a pre-trained traffic signal lamp detection model, and the model comprises a visual feature extraction network and a fusion perception detection network which are connected in sequence. Comprising the following steps: acquiring map data and a plurality of real shot images of the intelligent vehicle in a driving process; on the basis of a visual feature extraction network, feature extraction is carried out on each real shot image to obtain multiple visual features; based on a fusion perception detection network, performing fusion coding on map data and various visual features to obtain a plurality of fusion features; and on the basis of the fusion perception detection network, predicting each fusion feature to obtain a traffic signal lamp detection result. The problem of missing detection of the traffic signal lamps is significantly reduced, and the accuracy of traffic signal lamp detection is improved; the data driving characteristic is presented, and the detection effect can be improved through a large amount of training; moreover, the method is high in real-time performance, and almost has no delay.

Description

Technical Field

[0001] The present application relates to the field of intelligent automobile technology, and in particular to a traffic light detection method, device, computer equipment and readable storage medium. Background Art

[0002] Traffic light recognition is the most basic function in autonomous driving. There are currently two main methods. One is based on vehicle-to-everything (V2X) technology, and the other is based on visual image recognition technology.

[0003] V2X technology uses traffic lights to send signal status in real time, and then the vehicle receives the signal with low latency to identify the traffic lights. This method requires the modification of traffic lights, which not only places too high demands on the signal capabilities of the basic equipment, but also on network transmission.

[0004] It is based on visual image recognition, that is, the various status information of traffic lights is obtained by detecting and classifying 2D images. However, this method is easily affected by the weather environment outside the car. For example, strong sunlight or weak light makes it difficult to distinguish the colors of the imaging light panel, interference from the light tubes next to the traffic lights at night, interference from strong lights when capturing violations at night, interference from water stains on the camera, etc. Summary of the invention

[0005] To solve the above problems, the embodiments of the present application provide a traffic light detection method, apparatus, computer device and readable storage medium to overcome or partially overcome the deficiencies of the prior art.

[0006] In a first aspect, an embodiment of the present application provides a traffic light detection method, which is implemented by a pre-trained traffic light detection model, wherein the traffic light detection model includes a visual feature extraction network and a fusion perception detection network connected in sequence; the method includes:

[0007] Obtain map data and multiple real-time images of smart vehicles during driving;

[0008] Based on the visual feature extraction network, feature extraction is performed on each of the real-shot images to obtain a variety of visual features;

[0009] Based on the fusion perception detection network, the map data and the various visual features are fused and encoded to obtain a plurality of fusion features;

[0010] Based on the fusion perception detection network, each of the fusion features is predicted to obtain a traffic light detection result.

[0011] In a second aspect, an embodiment of the present application further provides a traffic light detection device, wherein the device is deployed with a pre-trained traffic light detection model, wherein the traffic light detection model includes a visual feature extraction network and a fusion perception detection network connected in sequence; the device includes:

[0012] An acquisition unit, used to acquire map data and multiple real-shot images of the intelligent vehicle during driving;

[0013] A feature extraction unit, configured to extract features from each of the real-shot images based on the visual feature extraction network to obtain a plurality of visual features;

[0014] A fusion unit, configured to perform fusion encoding on the map data and various visual features based on the fusion perception detection network to obtain fusion features;

[0015] A prediction unit is used to predict the fusion feature based on the fusion perception detection network to obtain a traffic light detection result.

[0016] In a third aspect, an embodiment of the present application further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned traffic light detection method when executing the computer program.

[0017] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the above-mentioned traffic light detection method when instructed by a processor.

[0018] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0019] The present application constructs and trains a traffic light detection model, which includes a visual feature extraction network and a fusion perception detection network connected in sequence. The visual feature extraction network serves as the input end of the model, and the fusion perception detection network serves as the output end of the model. During the driving process of the intelligent vehicle, the map data and multiple real-shot images during the driving process are obtained, and then the map data and multiple real-shot images are input into the traffic light detection model. The visual feature extraction network extracts features from each image to obtain multiple visual features. Then, the fusion perception detection network performs fusion encoding on the map data and various visual features to obtain multiple fusion features. Each fusion feature is predicted to obtain the traffic light detection result. The present application introduces an intelligent model into the detection of traffic lights, which significantly reduces the problem of missed detection of traffic lights and can greatly improve the accuracy of traffic light detection. The present application presents a data-driven feature, that is, by adding more training data, the detection performance of the traffic light detection model can be continuously improved. The present application has strong real-time performance and almost no delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 A schematic diagram of a flow chart of a traffic light detection method according to an embodiment of the present application is shown;

[0022] Figure 2 A schematic diagram of the structure of a traffic light detection model according to an embodiment of the present application is shown;

[0023] Figure 3 Shows that according to the data Figure 2 The flow diagram of the traffic light detection model 200 is shown;

[0024] Figure 4 A schematic diagram showing a visual feature extraction network 210 extracting visual features according to an embodiment of the present application is shown;

[0025] Figure 5 A schematic diagram of the structure of a traffic light detection device according to an embodiment of the present application is shown;

[0026] Figure 6 A schematic structural diagram of a computer device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0028] In order to enable ordinary technicians in the field to more clearly understand the technical solutions provided by the embodiments of the present application, the technical solution concept of the present application is first explained.

[0029] In the existing technology, traffic lights are detected by using vehicle network V2X technology or visual image recognition technology. Among them, V2X has high requirements on equipment performance; visual image recognition is greatly affected by the weather environment outside the vehicle and has low detection accuracy.

[0030] In this regard, the present application provides a traffic light detection method. Figure 1 A flow chart of a traffic light detection method according to an embodiment of the present application is shown. Figure 1 It can be seen that the present application at least includes steps S110 to S140:

[0031] Step S110: Acquire map data and multiple real-shot images of the smart vehicle during driving.

[0032] The traffic light detection method of the present application can be implemented by a traffic light detection model, which is trained with a large amount of data. Figure 2 A schematic diagram of the structure of a traffic light detection model according to an embodiment of the present application is shown. Figure 2 It can be seen that the traffic light detection model 200 includes a visual feature extraction network 210 and a fusion perception detection network 220 connected in sequence, the visual feature extraction network 210 serves as the input end of the traffic light detection model 200, and the fusion perception detection network 220 serves as the output end of the traffic light detection model 200.

[0033] In the present application, the input of the traffic light detection model 200 is the map data and multiple real-shot images of the intelligent vehicle during driving.

[0034] Among them, in some embodiments of the present application, the map data can be local map data (i.e., offline map) preloaded for the smart vehicle according to the driving task; or, real-time map data formed by multiple collected data collected by multiple sensors installed on the smart vehicle; the local map data and the real-time map data can be electronic maps of different levels of accuracy such as high-definition maps and ordinary maps.

[0035] In some embodiments of the present application, the multiple real-shot images may be obtained by photographing the target area with multiple cameras at different angles. In other embodiments, the multiple real-shot images may also be continuous multiple-frame images taken by one camera, but this method will bring certain delays; therefore, in the present application, it is preferred to use multiple cameras mounted on a smart vehicle to obtain multiple real-shot images at different angles.

[0036] Step S120: Based on the visual feature extraction network, feature extraction is performed on each of the real-shot images to obtain a variety of visual features.

[0037] After the map data and multiple real-shot images are input into the traffic light detection model 200, the multiple real-shot images are first processed based on the visual feature extraction network 210, that is, the detection results of multiple frames are output to the downstream fusion perception detection network 220 in the form of features.

[0038] The visual feature extraction network can be an image target detection network in the prior art, such as the DD3D model. DD3D is a single-stage and end-to-end network. By design, the parameters learned by the depth estimation module in pre-training can be shared with the 3D detection module. Finally, the network can benefit from the depth estimation pre-training based on a large amount of data like the pseudo-radar method, while having the advantages of simplicity and strong generalization of the end-to-end method.

[0039] By performing feature extraction on each real-shot image through a visual feature extraction network, a variety of visual features in each frame of the real-shot image can be obtained, and the visual features include but are not limited to 3D position information, classification features, etc.

[0040] Step S130: Based on the fusion perception detection network, the map data and the various visual features are fused and encoded to obtain a plurality of fusion features.

[0041] Then, the various visual features and map data in each frame of the real-shot image simultaneously enter the fusion perception detection network 220 to perform feature fusion and result prediction to obtain the traffic light detection result.

[0042] Figure 3 Shows that according to the data Figure 2The flow diagram of the traffic light detection model 200 shown in FIG. Figure 3 It can be seen that multiple real-shot images enter the visual feature extraction network 210 for feature extraction, and after obtaining multiple visual features of each real-shot image, these features and map data enter the fusion perception detection network 220 for feature fusion and result detection.

[0043] Regarding feature fusion, in some embodiments of the present application, map data can be fused with multiple visual features of each real-shot image to obtain multiple fused features, each of which contains the features of the map data and the visual features of a real-shot image.

[0044] In other embodiments of the present application, multiple visual features of each real-shot image can also be fused, such as fusing the 3D position information of multiple real-shot images to obtain a fusion feature of the 3D position information of the real-shot image, and fusing different visual features accordingly to obtain fusion features of multiple real-shot images, and then finally fusing the fusion features of the multiple real-shot images with the map data to obtain multiple fusion features.

[0045] For feature fusion, a multi-layer fully connected network can be used but is not limited to be implemented.

[0046] Step S140: Based on the fusion perception detection network, predict each of the fusion features to obtain a traffic light detection result.

[0047] Finally, multiple fusion features are tested to obtain traffic light detection results, which include the location of the traffic light, the color of the traffic light, and the shape of the traffic light (arrow, circle, etc.).

[0048] In some embodiments of the present application, in the above method, the fusion perception detection network is formed based on a multi-layer graph convolutional neural network and an attention mechanism.

[0049] Graph convolutional neural network (GCN) is a first-order local approximation of spectral graph convolution. It is a multi-layer graph convolutional neural network. Each convolutional layer only processes first-order neighborhood information. By stacking several convolutional layers, multi-order neighborhood information can be transferred. Its formula is shown in formula (1):

[0050]

[0051] in, N represents the number of nodes, and C represents the dimension of the node's feature vector, such as C-dimensional. represents the trainable parameter matrix, F represents the output dimension, It is the signal after convolution. The computational complexity of this formula is O(|ε|FC). Sparse matrix multiplication can be performed, so the actual calculation speed will be very fast.

[0052] The attention mechanism obtains the importance weight of the vector by calculating the similarity between vectors, thereby realizing information fusion between vectors. Its formula is shown in formula (2):

[0053]

[0054] in, n represents the length of the input sequence, d model represents the feature dimension of the input vector, d k Represents Key i The dimension of the feature, d v Represents Value i The dimension of the feature.

[0055] The above-mentioned fusion perception detection network is used to predict each of the fusion features to obtain a traffic light detection result, including: based on the multi-layer graph convolutional neural network, first-order neighborhood information extraction and multi-order neighborhood information propagation of each of the fusion features; based on the attention model, according to the similarity between each of the fusion features, the importance weight of each of the fusion features is determined to obtain the fusion information between each of the fusion features; based on the fusion information, the traffic light detection result is obtained.

[0056] The fusion feature of a real-shot image (Frame) and a map data is taken as a node (node), and the multi-layer graph convolutional neural network extracts the first-order neighborhood information and propagates the multi-order neighborhood information for each fusion feature, and the attention model determines the importance weight of each fusion feature according to the similarity between each fusion feature, and the fusion information between each fusion feature is obtained. It can be seen from formula (2) that when calculating the importance weight between two fusion features, one is used as a query (Query) and the other is used as a key value (Key). According to formula (2), the similarity between the two fusion features can be calculated. The importance weight can be directly measured by similarity. Simply put, the higher the similarity, the greater the importance weight. After prediction, the prediction result of the traffic light can be obtained. In some embodiments of the present application, the prior art is compared with the method of the present application. The prior art, such as using a multi-head attention model Sequence Model for prediction, needs to accumulate 35 frames of tracking image (Tracklet) data after the target is exposed, which brings at least 1.75s delay; while the method of the present application only needs to use the image obtained by real-time shooting, without the image in the historical time, and the target can be output in real time after exposure, with almost no delay. In addition, when using existing technologies, missed detections often occur, especially in noisy environments with a lot of interfering noise. After experiments, this application can improve the detection accuracy by more than 30% based on the multi-head attention model.

[0057] The traffic light detection model of this application presents data-driven characteristics. According to business needs, more training data can be added during the training phase to achieve continuous improvement in detection accuracy. It should be noted that the training process of the traffic light detection model is consistent with the business logic of the prediction process, which will not be repeated here.

[0058] Depend on Figure 1It can be seen from the method shown that the present application constructs and trains a traffic light detection model, which includes a visual feature extraction network and a fusion perception detection network connected in sequence, the visual feature extraction network serves as the input end of the model, and the fusion perception detection network serves as the output end of the model; during the driving process of the intelligent vehicle, the map data and multiple real-shot images during the driving process are obtained, and then the map data and multiple real-shot images are input into the traffic light detection model, the visual feature extraction network extracts features from each image to obtain multiple visual features, and then the fusion perception detection network fuses and encodes the map data and various visual features to obtain multiple fusion features; and each fusion feature is predicted to obtain the traffic light detection result. The present application introduces an intelligent model into the detection of traffic lights, which significantly reduces the problem of missed detection of traffic lights and can greatly improve the accuracy of traffic light detection; and the present application presents a data-driven feature, that is, by adding more training data, the detection performance of the traffic light detection model can be continuously improved; and the present application has strong real-time performance and almost no delay.

[0059] In some embodiments of the present application, in the above method, the visual feature extraction network is a 3D target feature extraction network based on a deep neural network; based on the visual feature extraction network, feature extraction is performed on each of the real-shot images to obtain a variety of visual features, including: based on the 3D target feature extraction network, feature extraction is performed on each of the real-shot images to obtain a variety of visual features of each frame of the real-shot image, and the visual features include: basic visual features, light group visual features and wick visual features; wherein the basic visual features include: camera ID, image ID and timestamp information; the light group visual features include: a first 2D visual frame, a first 3D position information, a first area of ​​interest, and on and off attributes; the wick visual features include: a second 2D visual frame, a second 3D position information, a second area of ​​interest, color attributes, and shape attributes.

[0060] Figure 4 A schematic diagram showing a visual feature extraction network 210 extracting visual features according to an embodiment of the present application is shown. Figure 4It can be seen that when multiple real-shot images are input into the visual feature extraction network and processed by the multi-layer graph neural network, for one real-shot image, three groups of visual features are obtained after extraction, including basic visual features, light group visual features and wick visual features, wherein the basic visual features include but are not limited to: camera ID, image ID and timestamp information, etc.; the light group visual features mainly refer to the overall information of the entire group of traffic lights to be identified, including but not limited to the first 2D visual frame, the first 3D position information, the first region of interest, and the on-off attribute of the entire group of traffic lights, wherein the on-off attribute mainly represents whether the traffic light is on (on) or off (off); the wick visual features mainly refer to the information of the wicks (mainly the lit wicks) of the entire group of traffic lights to be identified, including but not limited to the second 2D visual frame, the second 3D position information, the second region of interest, the color attributes, and the shape attributes of the wicks, wherein the color attributes mainly refer to the colors of red, yellow or green; the shape attributes mainly refer to the colors of circles, straight arrows and turning arrows, etc.

[0061] In some embodiments of the present application, in the above method, based on the fusion perception detection network, the map data and the various visual features are fused and encoded to obtain fused features, including: preprocessing the map data and the visual features of each frame of the real-shot image; for a frame of the real-shot image, using a splicing function, the map data, the first 2D visual frame, the first 3D position information, the first region of interest, the second 2D visual frame, the second 3D position information, and the second region of interest are fused and encoded to obtain image structural features; and the structural features, the brightness attributes, the color attributes, and the shape attributes are fused and encoded to obtain the fused features.

[0062] In the process of information fusion, the map data and the visual features of each real-shot image are fused separately to obtain multiple fused features. First, multiple features in the map data and visual features are preprocessed, including but not limited to: transforming the map data into a vehicle coordinate system (ego_pose transform); normalizing the first 2D visual frame and the second 2D visual frame (normalization); and encoding the first 3D position information and the second 3D position information in the vehicle coordinate system (pos embedding).

[0063] After preprocessing, each feature (including map data) is presented in the form of a matrix or a vector. The concat function can be used to fuse and encode the first 2D visual frame, the first 3D position information, the first region of interest, the second 2D visual frame, the second 3D position information, and the second region of interest to obtain the image structure feature (struct_feature). Specifically, the matrices of each feature are "placed" together in a certain order. For matrices and vectors of different dimensions, the missing elements can be set to "empty" or "specific value".

[0064] The aforementioned brightness attributes, color attributes, and shape attributes can be considered as classification features of the real-shot image. After obtaining the structural features, the structural features and the classification features are fused and encoded to obtain the final fused features.

[0065] For each real-shot image, the above method is used to perform feature fusion to obtain the fusion features corresponding to each real-shot image.

[0066] Figure 5 A schematic diagram of the structure of a traffic light detection device according to an embodiment of the present application is shown, wherein the device is deployed with a pre-trained traffic light detection model 200 ( Figure 2 ) is implemented, the traffic light detection model includes a visual feature extraction network and a fusion perception detection network connected in sequence; Figure 5 It can be seen that the traffic signal light detection device 500 includes:

[0067] An acquisition unit 510 is used to acquire map data and a plurality of real-shot images of the intelligent vehicle during its driving process;

[0068] A feature extraction unit 520, configured to extract features from each of the real-shot images based on the visual feature extraction network to obtain a plurality of visual features;

[0069] A fusion unit 530, configured to perform fusion encoding on the map data and various visual features based on the fusion perception detection network to obtain fusion features;

[0070] The prediction unit 540 is used to predict the fusion feature based on the fusion perception detection network to obtain a traffic light detection result.

[0071] In some embodiments of the present application, in the above-mentioned device, the visual feature extraction network is a 3D target feature extraction network based on a deep neural network; the feature extraction unit 520 is used to perform feature extraction on each of the real-shot images based on the 3D target feature extraction network to obtain multiple visual features of each frame of the real-shot image, and the visual features include: basic visual features, light group visual features and wick visual features; wherein the basic visual features include: camera ID, image ID and timestamp information; the light group visual features include: a first 2D visual frame, a first 3D position information, a first area of ​​interest, and on and off attributes; the wick visual features include: a second 2D visual frame, a second 3D position information, a second area of ​​interest, color attributes, and shape attributes.

[0072] In some embodiments of the present application, in the above-mentioned device, the fusion unit 530 is used to pre-process the map data and the visual features of each frame of the real-shot image; for a frame of the real-shot image, a splicing function is used to fuse and encode the map data, the first 2D visual frame, the first 3D position information, the first region of interest, the second 2D visual frame, the second 3D position information, and the second region of interest to obtain image structural features; and the structural features, the brightness attributes, the color attributes, and the shape attributes are fused and encoded to obtain the fused features.

[0073] In some embodiments of the present application, in the above-mentioned device, the fusion unit 530 is used to perform vehicle coordinate system conversion processing on the map data; standardize the first 2D visual frame and the second 2D visual frame; and perform position encoding processing on the first 3D position information and the second 3D position information in the vehicle coordinate system.

[0074] In some embodiments of the present application, in the above-mentioned device, the fusion perception detection network is formed based on a multi-layer graph convolutional neural network and an attention network.

[0075] In some embodiments of the present application, in the above-mentioned device, the prediction unit 540 is used to extract the first-order neighborhood information and propagate the multi-order neighborhood information for each of the fused features based on the multi-layer graph convolutional neural network; based on the attention network, determine the importance weight of each of the fused features according to the similarity between each of the fused features, and obtain the fusion information between each of the fused features; based on the fusion information, obtain the traffic light detection result.

[0076] In some embodiments of the present application, in the above-mentioned device, the map data is local map data preloaded by the smart vehicle; or, real-time map data formed based on multiple collected data collected by multiple sensors installed on the smart vehicle; the real-shot image is obtained by shooting with multiple cameras installed on the smart vehicle.

[0077] It should be noted that the above-mentioned traffic signal light detection device 500 can implement the above-mentioned traffic signal light detection methods one by one, which will not be described in detail here.

[0078] Figure 6 A schematic diagram showing the structure of a computer device according to an embodiment of the present application is shown. Figure 6 As shown, the computer device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, the functions or steps of the traffic light detection method are implemented.

[0079] In one embodiment, the computer device provided by the present application includes a memory and a processor, the memory stores a database and a computer program executable on the processor, and the above method is implemented when the processor executes the computer program.

[0080] The above application Figure 5The method performed by the traffic light detection device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor to be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above traffic light detection method in combination with its hardware.

[0081] In one embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned traffic light detection method are implemented.

[0082] It should be noted that the above functions or steps that can be implemented by the computer device or computer-readable storage medium can refer to the relevant description in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0083] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0084] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0085] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A traffic light detection method, characterized in that: This is achieved by using a pre-trained traffic light detection model, wherein the traffic light detection model includes a visual feature extraction network and a fusion perception detection network connected in sequence; the method includes: Obtain map data and multiple real-time images of smart vehicles during driving; Based on the visual feature extraction network, feature extraction is performed on each of the real-shot images to obtain a variety of visual features; Based on the fusion perception detection network, the map data and the various visual features are fused and encoded to obtain a plurality of fusion features; Based on the fusion perception detection network, each of the fusion features is predicted to obtain a traffic light detection result.

2. The method according to claim 1, characterized in that The visual feature extraction network is a 3D target feature extraction network based on a deep neural network; Based on the visual feature extraction network, feature extraction is performed on each of the real-shot images to obtain a variety of visual features, including: Based on the 3D target feature extraction network, feature extraction is performed on each of the real-shot images to obtain a plurality of visual features of each frame of the real-shot image, wherein the visual features include: basic visual features, light group visual features, and wick visual features; The basic visual features include: camera ID, image ID and timestamp information; The visual features of the light group include: a first 2D visual frame, first 3D position information, a first region of interest, and a light-on / off attribute; The wick visual features include: a second 2D visual frame, second 3D position information, a second region of interest, color attributes, and shape attributes.

3. The method according to claim 1, characterized in that Based on the fusion perception detection network, the map data and various visual features are fused and encoded to obtain fusion features, including: Preprocessing the map data and visual features of each frame of the real-shot image; For a frame of the real-shot image, using a splicing function, the map data, the first 2D visual frame, the first 3D position information, the first region of interest, the second 2D visual frame, the second 3D position information, and the second region of interest are fused and encoded to obtain image structural features; And the structural feature, the light-on and light-off attribute, the color attribute, and the shape attribute are fused and encoded to obtain the fused feature.

4. The method according to claim 3, characterized in that The preprocessing of the map data and the visual features of each frame of the real-shot image includes: Performing vehicle coordinate system conversion processing on the map data; Performing standardization processing on the first 2D visual frame and the second 2D visual frame; And performing position encoding processing on the first 3D position information and the second 3D position information in a vehicle coordinate system.

5. The method according to claim 1, characterized in that The fusion perception detection network is formed based on a multi-layer graph convolutional neural network and an attention network.

6. The method according to claim 5, characterized in that The method of predicting each of the fusion features based on the fusion perception detection network to obtain a traffic light detection result includes: Based on the multi-layer graph convolutional neural network, first-order neighborhood information extraction and multi-order neighborhood information propagation are performed on each of the fusion features; Based on the attention network, according to the similarity between the fused features, the importance weight of each fused feature is determined to obtain the fusion information between the fused features; The traffic light detection result is obtained according to the fusion information.

7. The method according to any one of claims 1 to 6, characterized in that: The map data is local map data preloaded in the smart vehicle; or real-time map data formed based on a plurality of collected data collected by a plurality of sensors loaded on the smart vehicle; The real-shot images are obtained by photographing with multiple cameras mounted on the smart vehicle.

8. A traffic light detection device, characterized in that: The device is deployed with a pre-trained traffic light detection model, wherein the traffic light detection model includes a visual feature extraction network and a fusion perception detection network connected in sequence; the device includes: An acquisition unit, used to acquire map data and multiple real-shot images of the intelligent vehicle during driving; A feature extraction unit, configured to extract features from each of the real-shot images based on the visual feature extraction network to obtain a plurality of visual features; A fusion unit, configured to perform fusion encoding on the map data and various visual features based on the fusion perception detection network to obtain fusion features; A prediction unit is used to predict the fusion feature based on the fusion perception detection network to obtain a traffic light detection result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the traffic light detection method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: The computer program, when instructed by a processor, implements the traffic light detection method according to any one of claims 1 to 7.

Images