Intelligent driving method and device based on traffic sign
By extracting the image information and navigation relationships of traffic signs, generating descriptive statements and generating intelligent driving assistance strategies, the problem of poor driving assistance effects in the existing technology is solved, and the safety and comfort of intelligent driving are improved.
Patent Information
- Application Number
- CN202410335416.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-22
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-03-22
AI Technical Summary
In the prior art, the mode of mechanically comparing traffic signs with existing databases results in poor driving assistance effects and inability to effectively assist drivers and intelligent driving decision-making systems.
By obtaining image information of traffic signs, the traffic indication information, including elements and navigation relationships, generate description statements, and generate intelligent driving assistance policies based on the description statements, which are used to prompt the driver or automatically adjust the driving status of the vehicle.
The fine-grained classification of traffic signs is realized, and more accurate road conditions information is generated, thereby improving the effectiveness of intelligent driving assistance strategies and improving driving safety and comfort.
Smart Images

Figure CN118331105B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of intelligent driving, and in particular to an intelligent driving method and device based on traffic signs. Background Art
[0002] Traffic signs guide vehicles to use the road in an orderly manner, can provide auxiliary information for intelligent driving, and effectively improve driving safety and comfort. The current traffic sign recognition method is mostly to detect and recognize the sign text or symbol, and then perform simple classification, and mechanically compare the traffic sign with the existing database to assist the driver and the intelligent driving decision-making system. However, the existing driving assistance through the comparison mode has a poor driving assistance effect. Summary of the invention
[0003] The present application provides an intelligent driving method and device based on traffic signs to solve the problem that the existing technology mechanically compares traffic signs with existing databases to assist drivers and intelligent driving decision-making systems, but the driving assistance effect is poor.
[0004] In the first aspect, the present application provides an intelligent driving method based on traffic signs, comprising: obtaining image information of a traffic sign in front of a vehicle, and extracting traffic indication information from the image information, wherein the traffic indication information includes elements in the traffic sign and navigation relationships between the elements, and the elements include: traffic indication symbols, traffic indication arrows and texts; the navigation relationship represents the association relationship and directional relationship between the elements; generating a corresponding description statement according to the navigation relationship between the elements in the traffic indication information; generating an intelligent driving assistance strategy according to the description statement, wherein the intelligent driving assistance strategy includes at least one of the following: prompt information for prompting the driver, control information for automatically adjusting the driving status of the vehicle.
[0005] In the second aspect, the present application provides an intelligent driving device based on a traffic sign, including: a processing module, used to obtain image information of a traffic sign in front of a vehicle, and extract traffic indication information from the image information, wherein the traffic indication information includes elements in the traffic sign and the navigation relationship between the elements, and the elements include: traffic indication symbols, traffic indication arrows and text; the navigation relationship represents the association relationship and directional relationship between the elements; a first generation module, used to generate a corresponding description statement according to the navigation relationship between the elements in the traffic indication information; a second generation module, used to generate an intelligent driving assistance strategy according to the description statement, wherein the intelligent driving assistance strategy includes at least one of the following: prompt information for prompting the driver, control information for automatically adjusting the driving status of the vehicle.
[0006] In a third aspect, the present application provides an electronic device comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the processor is configured to execute the intelligent driving method based on traffic signs described in the first aspect of the present application.
[0007] In a fourth aspect, the present application also provides a computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the intelligent driving method based on traffic signs described in the first aspect of the present application.
[0008] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: through the embodiment of the present application, after obtaining the image information of the traffic sign and extracting the traffic sign information from the image information, the fine-grained classification of the image information of the traffic sign is achieved, and then the corresponding description statement is generated according to the navigation relationship between the elements of the traffic sign information, so that the generated description statement is more accurate and more accurate and reasonable road condition information is obtained, so that a more effective intelligent driving assistance strategy can be generated later. That is, a safer intelligent driving experience can be achieved through the embodiment of the present application, which solves the problem of poor driving assistance effect in the prior art of mechanically comparing traffic signs with existing databases to assist drivers and intelligent driving decision-making systems. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0010] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0011] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.
[0012] Figure 1 A flow chart of an intelligent driving method based on traffic signs provided in an embodiment of the present application;
[0013] Figure 2A schematic diagram of a traffic sign provided in an embodiment of the present application;
[0014] Figure 3 A schematic diagram of a traffic sign after marking provided in an embodiment of the present application;
[0015] Figure 4 An optional flow chart of an intelligent driving method based on traffic signs provided in an embodiment of the present application;
[0016] Figure 5 A flowchart of an intelligent driving decision-making assistance method based on traffic sign understanding provided in an embodiment of the present application;
[0017] Figure 6 A flowchart for understanding the sign provided in the embodiment of the present application;
[0018] Figure 7 A schematic diagram of the structure of an intelligent driving decision assistance system based on traffic sign understanding provided in an embodiment of the present application;
[0019] Figure 8 A schematic diagram of integrating and understanding a traffic sign and outputting a description statement in a specific example provided in an embodiment of the present application;
[0020] Fig. 9 A schematic diagram of the structure of a traffic sign-based intelligent driving device provided in an embodiment of the present application;
[0021] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0023] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.
[0024] This application provides an intelligent driving method based on traffic signs, such as Figure 1 As shown, the steps of the method include:
[0025] Step 101, obtaining image information of a traffic sign in front of the vehicle, and extracting traffic instruction information from the image information, wherein the traffic instruction information includes elements in the traffic sign and navigation relationships between the elements, the elements including: traffic instruction symbols, traffic instruction arrows and texts; the navigation relationship represents the association relationship and directional relationship between the elements;
[0026] In the embodiment of the present application, traffic signs may include signs such as intersections, service areas, parking lots, no parking, speed limits, etc., that is, traffic signs refer to road sign information other than directions indicated by signs. Traffic arrows refer to signs in traffic signs that are arrows, and the arrows have directions, such as forward, left front, right front, and U-turn. Text is used to indicate information that needs to be described by words, such as road names and street names.
[0027] Furthermore, the association relationship refers to the relationship between various elements, such as Figure 2 As shown in the figure, the association relationship means that Wanshou Road, Shugang Road, Wansha Road, Zoulu Road and Binhai Avenue are located around Wanxin Road. Another example is that Wanshou Road is close to Huashan Road. The directional relationship refers to a directional description, for example, Huashan Road and Wanshou Road are located to the north of Wanxin Road. It can be seen that the combination of association relationship and directional relationship can clearly obtain the navigation information between the elements in the traffic indication information.
[0028] Step 102, generating a corresponding description statement according to the navigation relationship between each element information in the traffic instruction information;
[0029] In this regard, in a specific example, Figure 2 As shown, the generated description sentence can be: Go straight ahead to enter Wanshou Road, or turn right ahead to enter Zoulu Road, etc. If there are other elements on the current traffic sign, such as speed limit, no parking, or no honking, the generated description sentence can be: Go straight ahead to enter Wanshou Road, the current road speed limit is 40km / h, and honking is prohibited throughout the journey.
[0030] Step 103, generating an intelligent driving assistance strategy according to the description statement, wherein the intelligent driving assistance strategy includes at least one of the following: prompt information for prompting the driver, and control information for automatically adjusting the driving state of the vehicle.
[0031] It can be seen that after generating the corresponding description sentence, intelligent assistance strategy can be implemented. For example, the current description sentence is "Go straight ahead into Wanshou Road, the current road speed limit is 40km / h, and honking is prohibited throughout the journey." If the current vehicle speed is 50km / h, in addition to the voice prompt, the corresponding control command can also be generated to reduce the vehicle speed to 40km / h.
[0032] Through the above steps 101 to 103 of the embodiment of the present application, after obtaining the image information of the traffic sign and extracting the traffic sign information from the image information, the fine-grained classification of the image information of the traffic sign is achieved, and then the corresponding description sentence is generated according to the navigation relationship between the elements of the traffic sign information, so that the generated description sentence is more accurate and more accurate and reasonable road condition information is obtained, so that a more effective intelligent driving assistance strategy can be generated later. That is, a safer intelligent driving experience can be achieved through the embodiment of the present application, which solves the problem of poor driving assistance effect in the prior art of mechanically comparing traffic signs with existing databases to assist drivers and intelligent driving decision-making systems.
[0033] In an optional implementation of the embodiment of the present application, the method of extracting traffic indication information from the image information involved in the above step 101 may further include:
[0034] Step 101, dividing the elements representing road sign information from the image information into traffic indication symbols, dividing the elements representing directions into traffic indication arrows, and dividing the elements representing text information into text;
[0035] Step 12, marking a traffic sign by a first marking line, marking a traffic arrow by a second marking line, and marking a text by a third marking line;
[0036] Step 13: Identify the association relationship between the elements through the fourth identification line, and identify the directional relationship between the elements through the fifth identification line.
[0037] For the above steps 101 to 13, if Figure 3 As shown, the first identification line of Yangzhou Taizhou Airport, Yiling, 1km, S233, Exit, 215 is text, the second identification line is a traffic indication arrow, the third identification line is a traffic indication symbol, the fourth identification line is used to indicate that Yangzhou Taizhou Airport and Yiling are 1km ahead, and the fifth identification line is used to indicate the right front. It should be noted that the identification lines in the embodiment of the present application are distinguished by the shapes of different lines, and in specific application scenarios, other methods can also be distinguished by the color of the lines, which is not limited in the present application.
[0038] In an optional implementation of the embodiment of the present application, the method of generating a corresponding description statement according to the navigation relationship between the elements in the traffic indication information involved in the above step 102 may further include:
[0039] Step 21, constructing a corresponding scene graph according to the traffic indication information, wherein the nodes in the scene graph are used to represent the elements in the traffic indication information, and the edges between the nodes in the scene graph are used to represent the navigation relationship between the elements in the traffic indication information;
[0040] Step 22, inputting the scene graph into a graph attention network, and outputting a first feature vector, wherein the first feature vector is associated with the node feature and the edge feature;
[0041] Step 23, normalizing and encoding the traffic instruction information into the same feature space to obtain a second feature vector;
[0042] Step 24, input the first feature vector and the second feature vector, as well as the text characters obtained at the previous moment, into the target Transformer until a description sentence is output.
[0043] It should be noted that the target Transformer in the embodiment of the present application is a multimodal Transformer.
[0044] It can be seen from the above steps 21 to 24 that in the embodiment of the present application, the relationship between different elements can be constructed using a scene graph, nodes are used to represent the elements in the traffic sign, and the edges between the nodes represent the relationship between the sign elements. The constructed scene graph is input into the graph attention network (GAT) for further learning, and the node features obtained are Contains element visual features , location features and semantic features , which can be expressed as ; The obtained edge features include visual features and position features, which can be expressed as .
[0045] Furthermore, in a specific graph convolutional neural network, node features and edge features can be obtained by the following formula:
[0046]
[0047]
[0048] in, and is the corresponding fully connected network, yes right The attention coefficient, It is the edge arrive Features, is the feature of the i-th node.
[0049] After that, we need to input the acquired feature information and the character sentence encoding of the previous moment into the multimodal Transformer for interactive fusion and reasoning, and then output the description sentence through the decoder. Figure 3 For example, the character obtained at the last moment is " <start>After 1 km, take exit 215 on the right front, and follow "S233". Then the decoder can get the next character after "S233", and then iterate and output the final description sentence "After 1 km, take exit 215 on the right front, and follow S233 to Yangzhou Taizhou Airport and Yiling".
[0050] That is, the method of inputting the first feature vector and the second feature vector, and the character obtained at the previous moment into the target Transformer until the description sentence is outputted in the above step 24 may further include:
[0051] S1, based on the last text character in the text characters obtained at the previous moment, calculating the score of each element in the traffic indication information through a decoder;
[0052] S2, selecting the text character corresponding to the element with the highest score from the calculation results, and determining it as the next text character of the last text character;
[0053] S3, repeat S1 and S2 until the description statement is output.
[0054] In this regard, in a specific example, the first eigenvector, the eigenvector and the second eigenvector obtained, as well as the character obtained at the previous moment, are input into the target Transformer to output the vector , among which, among which, It represents the features obtained by inputting various elements into the scene graph and then inputting them into the graph attention network after detection, that is, the first feature vector, which can effectively obtain the spatial and semantic relationship between the elements. It represents the detection features obtained by mapping the basic features extracted by the backbone to the same feature space, that is, the second feature vector. Indicates the normalized encoding of the characters output before the current moment. In particular, if t=0, that is, no characters are output initially, we use a fixed meaningless character <start>to replace .
[0055] Further, =LN(F_ocr)+ LN(F_sign)+ LN(F_arrow), where F_ocr, F_sign, and F_arrow represent the features of text, symbol, and arrow, respectively. Specifically, each type of element (text, symbol, and arrow) will obtain the appearance features and bounding box features of the corresponding element, and the text will also obtain the corresponding semantic features. After LN (LayerNorm, layer normalization), the input features are converted into data with a mean of 0 and a variance of 1, so as to achieve normalized mapping to the same feature space, and then transmitted to the multimodal Transformer.
[0056] Furthermore, in an optional implementation of the embodiment of the present application, the method of calculating the score of each element in the traffic indication information by the decoder involved in the above step S1 may further include:
[0057] Step 31, when the element is text, predicting the score of the text based on the dynamic pointer network in the decoder;
[0058] Step 32, in the case where the element is non-text, the score of the non-text is determined based on the linear layer in the decoder, and the corresponding text is selected from a preset dictionary.
[0059] It can be seen that the decoder in the embodiment of the present application is divided into two branches. If there is text in the image, the scene text score is directly obtained through the dynamic pointer network. ; The dynamic pointer network is decoded and output And the output representation of the OCR characters after GAT enhancement The bilinear interaction between them predicts the score of each text character as:
[0060]
[0061] in, Represents the concatenation operation of vectors, and are the corresponding learnable parameters respectively.
[0062] If there is no text in the image, the score is calculated directly through the linear layer (equivalent to a linear function) , select a word from the preset dictionary as the current word. is the transpose of the learnable parameters, is the output of the multimodal Transformer, and b is a constant parameter. Finally, the text with the highest score is selected from all the texts as the current text, and then the remaining texts are generated through iteration until a description sentence is formed.
[0063] In an optional implementation of an embodiment of the present application, the method of generating an intelligent driving assistance strategy based on the description statement involved in the above step 103 may further include: identifying the urgency of the traffic sign based on the description statement, and generating prompt information based on the urgency and / or generating corresponding vehicle control instructions based on the control information.
[0064] In this regard, in the specific example, based on the generated description statement, on the one hand, the driver can be reminded to pay attention to changes in the current road conditions; on the other hand, the intelligent driving system can be used to adjust his driving strategy.
[0065] 1) Driver prompts: After identifying and understanding traffic signs, the first task is to provide clear and timely prompt information to the driver. Immediate prompts for important signs: For important signs such as speed limit, no-entry, and turn, the system should use voice prompts and dashboard displays to ensure that the driver can quickly and accurately obtain sign information. Double prompts for emergency signs: For signs that require immediate response, such as emergency stop and road closure, the system should not only prompt the driver through sound and vision, but may also need to automatically take safety measures such as emergency braking.
[0066] 2) Driving strategy adjustment: After understanding the content of the sign, the intelligent driving system needs to adjust the driving strategy according to the specific content of the sign. The specific situations may include the following:
[0067] Interpretation of traffic rules: First accurately understand the traffic rules such as speed limit, no-entry, turn, etc. on the signboards, and automatically adjust the vehicle's driving status according to these rules.
[0068] Speed control adjustment: When a speed limit sign changes, the system should automatically adjust the vehicle speed control strategy to ensure that the vehicle always drives within the speed limit. This may involve accelerating, decelerating, or maintaining the current speed.
[0069] Navigation route adjustment: For navigation signboards (such as ramps, roundabouts, etc.), we can not only get the direction information, but also the specific road distance numbers. This allows the system to update the route planning based on the sign information to ensure that the vehicle can travel in the correct direction. Specifically, this may include recalculating the driving path, adjusting lane selection, etc.
[0070] Adaptive driving mode: According to the content of the sign, the system can also adjust the vehicle's driving mode, such as switching from highway mode to city driving mode, or from automatic driving mode to manual driving mode.
[0071] Through the in-depth understanding of the content of traffic signs in the embodiments of this application, the intelligent driving system can adjust the driving strategy more accurately to ensure driving safety and efficiency, which not only improves the intelligence level of intelligent driving, but also provides strong support for the development of future intelligent transportation systems.
[0072] In an optional implementation manner of the embodiment of the present application, Figure 4 As shown, the method of the embodiment of the present application may also include:
[0073] Step 401, adjusting the monitoring frequency of acquiring the image information of the traffic sign by at least one of the following: the driving speed of the vehicle, the road environment information of the vehicle, and a customized monitoring frequency.
[0074] Among them, the monitoring frequency adjustment according to the vehicle's driving speed can be: when the vehicle is driving at high speed, due to the rapid changes in the road environment, the monitoring frequency of traffic signs should be increased to ensure that the sign information is captured and understood in a timely manner. In low-speed or congested conditions, the vehicle is driving slowly and the road environment is relatively stable. At this time, the monitoring frequency can be appropriately reduced to reduce unnecessary calculation burden.
[0075] The frequency of monitoring the road environment information of the vehicle can be adjusted by: obtaining more comprehensive road environment information through various sensors (such as radar, laser radar, etc.) carried by the vehicle. When the sensor detects that the road environment is complex and changeable (such as curves, intersections, etc.), the frequency of signboard monitoring should be increased; when the environmental perception system determines that the road environment is relatively simple and stable, the monitoring frequency can be appropriately reduced.
[0076] The monitoring frequency adjustment for the user-defined monitoring frequency can be: In this application, the user-defined monitoring frequency is supported to meet the personalized needs of different users. Or by continuously collecting user feedback and driving data, the monitoring frequency is adaptively adjusted to optimize the driving experience and safety.
[0077] Through the above-mentioned intelligent monitoring and frequency adjustment method in the embodiment of the present application, it can be ensured that the intelligent driving decision-making assistance system based on traffic sign understanding can perform at its best in different driving scenarios, reduce energy consumption, and improve driving safety and comfort.
[0078] The present application is described below with reference to a specific implementation of an embodiment of the present application. The specific implementation provides an intelligent driving decision-making assistance method based on understanding of traffic signs, such as Figure 5 As shown, the steps of the method include:
[0079] Step 501, collecting and preprocessing the image data of the front-view camera;
[0080] Step 502, intelligently monitor whether there is a sign; if there is, execute step 503, if not, the process ends;
[0081] Specifically, when the vehicle is in motion, the environmental information obtained by the vehicle is used to determine whether to start the sign recognition. The environmental information mainly refers to the image information obtained by the forward-looking fisheye camera, which is then transmitted to the original detection model of the intelligent driving system to determine whether there is a traffic sign in front of the vehicle.
[0082] Step 503, start signboard understanding;
[0083] Step 504, reminding the driver;
[0084] Step 505, update the intelligent driving decision based on the understood information.
[0085] It can be seen that this application adds a signboard understanding module to the existing intelligent driving perception system, and uses the original perception module to detect in real time whether there is a traffic sign on the road ahead. If not, the process ends; if so, the signboard understanding is started, and the collected image data is first dedistorted, cropped (processed into a specified size) and data enhanced, and then transmitted to the signboard understanding model, and finally the obtained image understanding sentence is output to the driver and the intelligent driving decision-making system as a reference, so as to achieve the purpose of optimizing intelligent driving decisions.
[0086] For the understanding of the identification plate in the above step 503, Figure 6 As shown, the following steps may further be included:
[0087] Step 601, obtaining an original front-view camera image;
[0088] Step 602, image preprocessing;
[0089] Step 603, crop the sign area from the original image;
[0090] It can be seen that in the embodiment of the present application, after starting the signboard understanding, it is first necessary to obtain an image and perform preprocessing. Specifically, the image is obtained through the vehicle's forward-looking fisheye camera, and optimization processing such as de-distortion and denoising is performed. Since the acquired image contains a variety of irrelevant objects, it is also necessary to crop the detected signboard as the input image of the signboard understanding model.
[0091] Step 604, detecting and identifying symbols, arrows and texts on the cropped sign image;
[0092] In the embodiment of the present application, in order to accurately describe the elements in the traffic sign, the present application abandons the previous mode of mechanically comparing the traffic sign with the existing database, and chooses to perform fine-grained classification: it is generally divided into 3 categories: symbols, arrows and text. For symbols, we divide them into 32 subcategories such as intersections, service areas, parking lots, etc., and divide the arrow group into 8 subcategories according to the direction, such as front, left front, and right rear; among them, the fine-grained classification of symbols can make the road condition information have a clearer understanding, and the fine-grained direction division of arrows provides the spatial position relationship between the elements in the image.
[0093] Step 605: perform fusion reasoning on the detected and recognized content and generate corresponding description sentences.
[0094] In this regard, Figure 7 The overall architecture diagram of the signboard understanding model shown in the figure is combined with Figure 8 As can be seen, the signboard is first taken out separately by cropping as the input for signboard understanding. Then, in order to distinguish different element types, three detection heads are used to detect symbols, arrows, and text (OCR) respectively. After obtaining the detected information, the scene graph and graph attention network are used to further explore the relationship between the elements. Then, these features are normalized and encoded into the same feature space, and then combined with the characters obtained at the previous moment (such as Figure 7 in <start>1 km later, take exit 215 in front of the right, and follow the "") to the multimodal Transformer, and the final character is obtained through the decoder (such as Figure 8 in S233 ", the characters after "S233" are unpredicted characters), through iterative processing, until the final description sentence is output as "1km later, take exit 215 in front of the right, and go to Yangzhou Taizhou Airport and Yiling along S233". That is, the acquired features and the characters obtained at the previous moment are input into the target Transformer to output the vector , among which, among which, It means that after detection, various elements are combined into a scene graph and input into the graph attention network to obtain the features, which can effectively obtain the spatial and semantic relationship between the elements. It represents the detection feature quantity obtained by mapping the basic features extracted by the backbone to the same feature space. Represents the normalized encoding of the characters output before the current moment. If t=0, that is, no characters are output initially, we use a fixed meaningless character <start>to replace .
[0095] It can be seen that in the embodiment of the present application, a signboard understanding is added to the existing intelligent driving perception system, which can detect various elements (especially text) in the signboard in real time, and predict the relationship between the elements through reasoning, so as to obtain more accurate and reasonable road condition information for assisting driving decision-making. In addition, in order to further improve the efficiency and accuracy of information processing, a multimodal Transformer structure is adopted in the embodiment of the present application, which can effectively interact, fuse and reason the feature information obtained from various sensors and signboard understanding, and finally output it to the decoder to assist driving decision-making. Furthermore, the intelligent monitoring mechanism intelligently adjusts the monitoring frequency of the signboard according to the real-time speed of the vehicle. This not only helps to reduce unnecessary calculations and resource consumption, but also ensures that the system can obtain road condition information in a timely and accurate manner at critical moments. At the same time, combined with the results of signboard understanding, the driving strategy can also be dynamically adjusted to adapt to different road conditions and traffic rules, further improving the safety and efficiency of intelligent driving.
[0096] Corresponding to the above Figure 1 , the embodiment of the present application also provides an intelligent driving device based on traffic signs, such as Fig. 9 As shown, the device comprises:
[0097] The processing module 902 is used to obtain image information of a traffic sign in front of the vehicle and extract traffic instruction information from the image information, wherein the traffic instruction information includes elements in the traffic sign and navigation relationships between the elements, and the elements include: traffic instruction symbols, traffic instruction arrows and texts; the navigation relationship represents the association relationship and directional relationship between the elements;
[0098] A first generating module 904, configured to generate a corresponding description statement according to the navigation relationship between the elements in the traffic instruction information;
[0099] The second generation module 906 is used to generate an intelligent driving assistance strategy according to the description statement, wherein the intelligent driving assistance strategy includes at least one of the following: prompt information for prompting the driver, and control information for automatically adjusting the driving state of the vehicle.
[0100] In an optional implementation manner of an embodiment of the present application, the processing module 902 in the embodiment of the present application may further include: a division unit, used to divide elements representing road sign information from the image information into traffic signs, and divide elements representing directions into traffic arrows, and divide elements representing text information into text; a first identification unit, used to identify traffic signs through a first identification line, identify traffic arrows through a second identification line, and identify text through a third identification line; a second identification unit, used to identify the association relationship between elements through a fourth identification line, and identify the directional relationship between elements through a fifth identification line.
[0101] In an optional implementation manner of an embodiment of the present application, the first generation module 904 in the embodiment of the present application may further include: a construction unit, used to construct a corresponding scene graph according to the traffic indication information, wherein the nodes in the scene graph are used to represent the elements in the traffic indication information, and the edges between the nodes in the scene graph are used to represent the navigation relationship between the elements in the traffic indication information; a first processing unit, used to input the scene graph into the graph attention network and output a first feature vector, wherein the first feature vector is associated with the node feature and the edge feature; a second processing unit, used to normalize and encode the traffic indication information into the same feature space to obtain a second feature vector; a third processing unit, used to input the first feature vector and the second feature vector, as well as the text characters obtained at the previous moment, into the target Transformer until a description sentence is output.
[0102] In an optional implementation manner of the embodiment of the present application, the third processing unit in the embodiment of the present application is used to perform the following steps:
[0103] S1, based on the last text character in the text characters obtained at the previous moment, calculating the score of each element in the traffic indication information through a decoder;
[0104] Among them, when the element is text, the score of the text is predicted based on the dynamic pointer network in the decoder; when the element is non-text, the score of the non-text is determined based on the linear layer in the decoder, and the corresponding text is selected from the preset dictionary.
[0105] S2, selecting the text character corresponding to the element with the highest score from the calculation results, and determining it as the next text character of the last text character;
[0106] S3, repeat S1 and S2 until the description statement is output.
[0107] In an optional implementation of an embodiment of the present application, the second generation module in the embodiment of the present application may further include: a generation unit for identifying the urgency of the traffic sign based on the description statement, and generating prompt information based on the urgency and / or generating corresponding vehicle control instructions based on the control information.
[0108] In an optional implementation of the embodiment of the present application, the device in the embodiment of the present application may also include: an adjustment module, used to adjust the monitoring frequency of obtaining the image information of the traffic sign through at least one of the following: the vehicle's driving speed, the road environment information of the vehicle's environment, and a customized monitoring frequency.
[0109] like Fig.10 As shown, an embodiment of the present application provides an electronic device, including a processor 1011, a communication interface 1012, a memory 1013 and a communication bus 1014, wherein the processor 1011, the communication interface 1012, and the memory 1013 communicate with each other through the communication bus 1014.
[0110] Memory 1013, used for storing computer programs;
[0111] In one embodiment of the present application, the processor 1011 is used to execute the program stored in the memory 1013 to implement the intelligent driving method based on traffic signs provided by any of the aforementioned method embodiments, and the role it plays is similar and will not be repeated here.
[0112] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the intelligent driving method based on traffic signs provided in any of the aforementioned method embodiments are implemented.
[0113] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0114] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.
[0115] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.
[0116] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.< / start> < / start> < / start> < / start>
Claims
1. An intelligent driving method based on traffic signs, It is characterized in that include: Acquire image information of a traffic sign ahead of the vehicle, and extract traffic instruction information from the image information, wherein the traffic instruction information includes elements in the traffic sign and navigation relationships between the elements, the elements including: traffic instruction symbols, traffic instruction arrows and texts; the navigation relationship represents the association relationship and directional relationship between the elements; Generate a corresponding description statement according to the navigation relationship between the elements in the traffic instruction information; Generate an intelligent driving assistance strategy according to the description statement, wherein the intelligent driving assistance strategy includes at least one of the following: prompt information for prompting the driver, and control information for automatically adjusting the driving state of the vehicle; Wherein, generating a corresponding description statement according to the navigation relationship between each element in the traffic instruction information includes: constructing a corresponding scene graph according to the traffic instruction information, wherein the nodes in the scene graph are used to represent the elements in the traffic instruction information, and the edges between the nodes in the scene graph are used to represent the navigation relationship between each element in the traffic instruction information; inputting the scene graph into a graph attention network, and outputting a first feature vector, wherein the first feature vector is associated with node features and edge features; normalizing and encoding the traffic instruction information into the same feature space to obtain a second feature vector; inputting the first feature vector and the second feature vector, as well as the text characters obtained at the previous moment, into a target Transformer until the description statement is output; The first feature vector and the second feature vector, as well as the character obtained at the previous moment, are input into the target Transformer until the description statement is output, including: S1, based on the last text character among the text characters obtained at the previous moment, calculating the score of each element in the traffic instruction information through a decoder; S2, selecting the text character corresponding to the element with the highest score from the calculation result, and determining it as the next text character of the last text character; S3, repeatedly executing S1 and S2 until the description statement is output; Among them, the score of each element in the traffic instruction information is calculated by the decoder, including: when the element is text, predicting the score of the text based on the dynamic pointer network in the decoder; when the element is non-text, determining the score of the non-text based on the linear layer in the decoder, and selecting the corresponding text from a preset dictionary.
2. The method according to claim 1, It is characterized in that Extracting traffic indication information from the image information includes: Dividing the elements representing the road sign information from the image information into the traffic indication symbols, dividing the elements representing the directions into the traffic indication arrows, and dividing the elements representing the text information into the text; Identify the traffic sign by a first identification line, identify the traffic arrow by a second identification line, and identify the text by a third identification line; The association relationship between the elements is identified by the fourth identification line, and the pointing relationship between the elements is identified by the fifth identification line.
3. The method according to claim 1, It is characterized in that Generating an intelligent driving assistance strategy according to the description statement includes: The urgency of the traffic sign is identified according to the description statement, and the prompt information is generated according to the urgency and / or a corresponding vehicle control instruction is generated based on the control information.
4. The method according to claim 1, It is characterized in that The method further comprises: The monitoring frequency for acquiring the image information of the traffic sign is adjusted by at least one of the following: the driving speed of the vehicle, the road environment information of the vehicle, and a customized monitoring frequency.
5. An intelligent driving device based on traffic signs, It is characterized in that include: A processing module, used to obtain image information of a traffic sign in front of the vehicle, and extract traffic instruction information from the image information, wherein the traffic instruction information includes elements in the traffic sign and navigation relationships between the elements, the elements including: traffic instruction symbols, traffic instruction arrows and texts; the navigation relationship represents the association relationship and directional relationship between the elements; A first generating module, used for generating a corresponding description statement according to the navigation relationship between the elements in the traffic instruction information; A second generating module is used to generate an intelligent driving assistance strategy according to the description statement, wherein the intelligent driving assistance strategy includes at least one of the following: prompt information for prompting the driver, and control information for automatically adjusting the driving state of the vehicle; The first generation module includes: a construction unit, which is used to construct a corresponding scene graph according to the traffic instruction information, wherein the nodes in the scene graph are used to represent the elements in the traffic instruction information, and the edges between the nodes in the scene graph are used to represent the navigation relationship between the elements in the traffic instruction information; a first processing unit, which is used to input the scene graph into the graph attention network and output a first feature vector, wherein the first feature vector is associated with the node feature and the edge feature; a second processing unit, which is used to normalize and encode the traffic instruction information into the same feature space to obtain a second feature vector; a third processing unit, which is used to input the first feature vector and the second feature vector, as well as the text characters obtained at the previous moment, into the target Transformer until a description sentence is output; The third processing unit is used to perform the following steps: S1, based on the last text character among the text characters obtained at the previous moment, calculate the score of each element in the traffic instruction information through the decoder; wherein, in the case where the element is text, predict the score of the text based on the dynamic pointer network in the decoder; in the case where the element is non-text, determine the score of the non-text based on the linear layer in the decoder, and select the corresponding text from the preset dictionary; S2, select the text character corresponding to the element with the highest score from the calculation results, and determine it as the next text character of the last text character; S3, repeat S1 and S2 until the description sentence is output; The S1 includes: when the element is text, predicting the score of the text based on the dynamic pointer network in the decoder; when the element is non-text, determining the score of the non-text based on the linear layer in the decoder, and selecting the corresponding text from a preset dictionary.
6. An electronic device, include: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor coupled to the at least one bus; At least one memory connected to the at least one bus, wherein the processor is configured to execute the intelligent driving method based on traffic signs as described in any one of claims 1 to 4 above.
7. A computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the intelligent driving method based on traffic signs as described in any one of claims 1 to 4 above.
