Traffic question and answer method, storage medium, product and vehicle

Abstract

The invention relates to a traffic question and answer method, a storage medium, a product and a vehicle. And the vehicle receives the traffic problem input by the user. In response to the traffic question, the vehicle outputs a target answer. The target answer is obtained by the target model based on the detection result of the traffic signal on the first image and the traffic question. On the basis, the vehicle can answer the traffic problem of the user through the target model, so that the vehicle can better dialogue with the user, the interactivity between the vehicle and the user is improved, and the user experience is ensured.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and more particularly to a traffic question-and-answer method, storage medium, product, and vehicle. Background Technology

[0002] Visual question answering (VQA) refers to a model generating and outputting corresponding answers based on a given image and a question about that image. With the development of technology, visual question answering is gradually being applied in vehicle driving fields such as assisted driving and autonomous driving.

[0003] Currently, in the field of vehicle driving, the models used for visual question answering have limited capabilities, resulting in poor interaction between vehicles and users and a low user experience. Summary of the Invention

[0004] This application provides a traffic question-answering method, storage medium, product, and vehicle. The vehicle can answer users' traffic questions through a target model, improving the interactivity between the vehicle and the user and providing a high-quality user experience.

[0005] To achieve the above objectives, according to a first aspect of this application, a traffic problem method is provided, applied to a vehicle, the method comprising: receiving a traffic problem input by a user;

[0006] In response to the traffic problem, output the target answer;

[0007] The target answer is obtained by the target model based on the detection results of traffic signals on the first image and the traffic question.

[0008] Optionally, the method further includes:

[0009] Send the first image and the traffic problem to the server;

[0010] Receive the target response from the server.

[0011] Alternatively, the first image may be captured by a camera on the vehicle.

[0012] Optionally, the target model is obtained by training a first model using sample image information and its corresponding traffic question-and-answer dataset;

[0013] The traffic question-and-answer dataset is generated based on the sample image information; the sample image information includes labeled images with traffic signals and enhanced images of the labeled images, the enhanced images being obtained by performing data augmentation on the labeled images.

[0014] Optionally, the enhanced image includes a first enhanced image; wherein the first enhanced image is obtained by performing viewpoint switching data enhancement on the labeled image; the viewpoint switching data enhancement is used to change the orientation of traffic signals in the labeled image and reduce the clarity of the labeled image.

[0015] Optionally, the enhanced image includes a second enhanced image; wherein the second enhanced image is obtained by performing long-tailed label data enhancement on the labeled image; the long-tailed label data enhancement is used to replace traffic signals on the labeled image with long-tailed labels; the long-tailed labels are determined based on the frequency of occurrence of the traffic signals.

[0016] Optionally, the enhanced image includes a third enhanced image; wherein the third enhanced image is obtained by performing long-tailed label data enhancement on the first enhanced image; and the first enhanced image is obtained by performing viewpoint switching data enhancement on the labeled image.

[0017] Optionally, the process of determining the first enhanced image may include:

[0018] The labeled image is segmented to obtain multiple first sub-images of the labeled image;

[0019] Each of the plurality of first sub-images is magnified to obtain a first enhanced image corresponding to each first sub-image; wherein the size of the first enhanced image is the same as the size of the labeled image.

[0020] Optionally, the first subgraph includes a positive subgraph and a negative subgraph; the positive subgraph represents a subgraph marked with traffic signals; the negative subgraph represents a subgraph without traffic signals marked.

[0021] Optionally, segmenting the labeled image to obtain multiple first sub-images of the labeled image includes:

[0022] A sliding window is used to slide over the labeled image;

[0023] Based on the enhancement weight of the second subgraph within the sliding window, the second subgraph is used as the first subgraph;

[0024] The enhancement weight of the second subgraph is negatively correlated with the frequency of traffic signal occurrence in the second subgraph, and the enhancement weight of the second subgraph is positively correlated with the probability of using the second subgraph as the first subgraph.

[0025] Optionally, the first subgraph includes a positive subgraph;

[0026] The step of using the second subgraph as the first subgraph based on the enhancement weight of the second subgraph within the sliding window includes:

[0027] When a traffic signal is marked on the second sub-image within the sliding window, the second sub-image is selected as a candidate positive sub-image based on its enhancement weight; wherein the enhancement weight of the second sub-image is positively correlated with the probability of selecting the second sub-image as the candidate positive sub-image.

[0028] Randomly select p candidate positive subgraphs to obtain the positive subgraph, where p is a positive integer.

[0029] Optionally, the traffic signals on the above candidate subgraphs are complete.

[0030] Optionally, the process of determining the negative subgraph includes:

[0031] If the second sub-map within the sliding window does not have traffic signals marked, the second sub-map will be used as a candidate negative sub-map.

[0032] q candidate negative subgraphs are randomly selected to obtain the negative subgraph, where p is a positive integer.

[0033] Optionally, the long-tail sign includes traffic signals that appear less frequently than a preset frequency in all labeled images; the method further includes:

[0034] Replace the traffic signals on the image to be replaced with long-tailed signs; wherein the image to be replaced represents a first enhanced image containing a number of traffic signals.

[0035] Optionally, the sample image information also includes the annotation file corresponding to the labeled image and the annotation file corresponding to the enhanced image, wherein the annotation file includes information on the traffic signals annotated on the image corresponding to the annotation file.

[0036] Optionally, the labeled image is obtained by fully labeling the original labeled image.

[0037] Optionally, the method further includes:

[0038] The sample image information and preset rules are input into the second model to obtain the traffic question-and-answer dataset.

[0039] Optionally, the number of parameters in the second model is greater than the number of parameters in the first model.

[0040] Optionally, the second model is used to perform a first filter (or hard filter) on the sample image information and generate a traffic question-and-answer dataset based on the filtered sample image information; wherein, the first filter means filtering out traffic signals in the sample image information whose size is smaller than a preset size.

[0041] Optionally, the traffic question-and-answer dataset includes traffic question samples and corresponding answer samples.

[0042] Optionally, the enhanced image of the labeled image includes a positive sample image of the labeled image and a negative sample image of the labeled image. The positive sample image includes one or more of a first enhanced image, a second enhanced image, and a third enhanced image. The first enhanced image is obtained based on the positive sub-image of the labeled image, and the negative sample image is obtained by enlarging the negative sub-image of the labeled image.

[0043] The traffic question-and-answer dataset includes a first traffic question sample for traffic signals on the labeled image and a first answer sample corresponding to the first traffic question sample.

[0044] A second traffic question sample and a second answer sample corresponding to the traffic signal on the positive sample image of the labeled image;

[0045] And a third traffic problem sample on the negative sample image of the labeled image and a third answer sample corresponding to the third traffic problem sample;

[0046] The first sample response includes traffic signals on the labeled image, and / or driving advice;

[0047] The second response sample includes traffic signals on a positive sample image of the labeled image, and / or driving advice;

[0048] The third response sample includes preset response sample information and / or driving suggestions.

[0049] Optionally, the above-mentioned training of the first model using the sample image information and its corresponding traffic question-and-answer dataset to obtain the target model includes:

[0050] Based on LoRA fine-tuning technology, the first model is fine-tuned using the sample image information and its corresponding traffic question-and-answer dataset to obtain the target model.

[0051] According to a second aspect of this application, a model training method is provided, comprising: acquiring sample image information and a traffic question-and-answer dataset corresponding to the sample image information; wherein the traffic question-and-answer dataset is generated based on the sample image information; the sample image information includes annotated images with traffic signals and enhanced images of the annotated images, wherein the enhanced images are obtained by performing data augmentation on the annotated images;

[0052] The first model is trained using the sample image information and its corresponding traffic question-and-answer dataset to obtain the target model.

[0053] Optionally, the target model is used to identify traffic signals in images captured by cameras on the vehicle, obtain detection results, and generate and output corresponding answers based on the detection results and user-input questions.

[0054] According to a third aspect of this application, a traffic question-and-answer method is provided, applied to a server, the method comprising:

[0055] Receive the first image sent by the vehicle and the traffic problem input by the user;

[0056] Send the target response to the vehicle;

[0057] The target answer is obtained by the target model based on the detection results of traffic signals on the first image and the traffic question.

[0058] According to a fourth aspect of this application, a dataset generation method is provided, comprising: generating a traffic question-and-answer dataset based on sample image information; wherein the sample image information includes annotated images labeled with traffic signals and enhanced images of the annotated images, the enhanced images being obtained by performing data augmentation on the annotated images.

[0059] According to a fifth aspect of this application, an electronic device is provided, comprising:

[0060] A memory on which computer programs are stored;

[0061] A processor for executing the computer program in the memory to implement the method as described in the first or second aspect above.

[0062] According to a sixth aspect of this application, a vehicle is provided, including the electronic equipment described above.

[0063] According to a seventh aspect of this application, a server is provided, comprising:

[0064] A memory on which computer programs are stored;

[0065] A processor for executing the computer program in the memory to implement the method as described in the first or second aspect above.

[0066] According to an eighth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method described above.

[0067] According to an eighth aspect of this application, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described above.

[0068] In summary, according to the embodiments of this application, through the above technical solution, after receiving a traffic question input by the user, the vehicle responds to the traffic question by outputting a target answer. This target answer is obtained by the target model based on the detection results of traffic signals on the first image and the traffic question. In other words, the target model has high capability and can answer the user's traffic question well based on the detection results of traffic signals on the first image, ensuring the accuracy of the answer and improving the interactivity between the vehicle and the user, thereby enhancing the user experience.

[0069] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] To gain a more complete understanding of this application and its beneficial effects, the following description will be provided in conjunction with the accompanying drawings, wherein the same reference numerals in the following description denote the same parts.

[0072] Figure 1 This is a flowchart of a traffic question-and-answer method provided in an exemplary embodiment of this disclosure. Figure 1 ;

[0073] Figure 2 This is a schematic diagram of a full annotation process provided in an exemplary embodiment of this disclosure. Figure 1 ;

[0074] Figure 3 This is a schematic diagram of a full annotation process provided in an exemplary embodiment of this disclosure. Figure 2 ;

[0075] Figure 4This is a schematic diagram of a data augmentation process provided in an exemplary embodiment of this disclosure. Figure 1 ;

[0076] Figure 5 This is a schematic diagram of a data augmentation process provided in an exemplary embodiment of this disclosure. Figure 2 ;

[0077] Figure 6 This is a schematic diagram illustrating the process of generating a traffic question-and-answer dataset according to an exemplary embodiment of this disclosure;

[0078] Figure 7 This is a flowchart of a traffic question-and-answer method provided in an exemplary embodiment of this disclosure. Figure 2 ;

[0079] Figure 8 This is a schematic diagram of a vehicle-cloud collaboration process provided in an exemplary embodiment of this disclosure;

[0080] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this disclosure. Figure 1 ;

[0081] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this disclosure. Figure 2 ;

[0082] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this disclosure. Figure 3 ;

[0083] Figure 12 This is a schematic diagram of the chip system architecture provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0084] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0085] To facilitate understanding of this application, some of the terms used in this application will be introduced below.

[0086] A vision-language model (VLM) is a multimodal artificial intelligence model that integrates computer vision and natural language processing techniques. It can simultaneously process and understand image and text information. By combining knowledge from visual and language modalities, VLM can perform tasks such as image captioning and visual question answering, that is, generating descriptive text from images or analyzing and reasoning about images based on text instructions.

[0087] Long-tail distribution refers to a situation in a specific dataset where a few items occupy a large number of occurrences or have high importance (i.e., the "head"), while a large number of other items make up a significant proportion of the overall dataset (i.e., the "tail"). Long-tail distribution is common in many real-world datasets. In traffic signal recognition datasets, common traffic signals such as speed limit signs and zebra crossing markings often occupy the head, while a large number of other signals are distributed in the tail.

[0088] Data augmentation is a technique commonly used in machine learning and deep learning. It expands the size and diversity of the training dataset by purposefully transforming existing data or generating new data samples. It helps models learn a wider range of data features, reduces overfitting, and improves the model's generalization ability.

[0089] Multimodal interaction is a human-computer interaction method that integrates multiple sensory channels (such as vision, hearing, and touch) and data modalities (such as text, images, and voice). By simultaneously processing and utilizing multiple input and output formats, it enables machines to more comprehensively understand user intentions and provide a more natural and richer interactive experience. Compared to traditional unimodal interaction, multimodal interaction allows users to communicate with devices through voice commands, gestures, text input, and other methods. Simultaneously, devices can also provide feedback through voice, images, text, and other formats, greatly enhancing the flexibility of interaction and the user experience.

[0090] Fine-tuning refers to the process of further training a pre-trained model for a specific downstream task or dataset. Pre-trained models typically learn a wide range of fundamental knowledge on large-scale general data, while fine-tuning transfers this knowledge to a specific domain or task. By continuing training on the target dataset, the model's parameters are adjusted to better adapt to the new task requirements, thereby improving the model's performance and accuracy in specific scenarios.

[0091] LoRA (low-rank adaptation) fine-tuning is a method for optimizing large-scale pre-trained models. It fine-tunes some weights of the model by introducing a low-rank matrix, rather than updating all parameters of the entire model. Specifically, LoRA technology allows the model to adapt to specific tasks or datasets by inserting a small number of trainable parameters (usually a low-rank matrix) into the original model, thus achieving efficient personalized tuning. Compared to traditional full-model fine-tuning, LoRA significantly reduces the required storage space and computational cost while maintaining high performance.

[0092] While traditional traffic signal recognition systems have achieved effective detection of common traffic signs and traffic lights to a certain extent, they have significant limitations. Traditional traffic signal recognition mainly relies on conventional image preprocessing, feature extraction, and deep learning-based image classification models. Common image preprocessing operations include, but are not limited to, grayscale conversion, noise reduction, and contrast enhancement. Image preprocessing can effectively remove interfering factors from the original image, such as changes in lighting and weather conditions, making the traffic signals in the original image clearer and more visible.

[0093] Feature extraction refers to extracting information that characterizes traffic signal properties from preprocessed images, such as color, shape, and texture. Then, traffic signals can be determined based on the extracted features using deep learning-based image classification models (such as YOLO (You Only Look Once) and convolutional neural networks (CNNs) such as ResNet and VGG). However, deep learning-based image classification models have a narrow recognition range and insufficient scalability. They typically can only recognize common traffic signs or traffic lights, failing to recognize all traffic signals and thus unable to provide users with comprehensive road information. Furthermore, due to the long-tail distribution of training data, the lack of sufficient training data for some categories prevents deep learning-based image classification models from accurately recognizing certain traffic signals. Additionally, while deep learning-based image classification models are generally trained on publicly available large-scale traffic signal datasets, these typically cover limited geographical areas, and the road traffic information contained in the annotation files is incomplete. Models trained on low-information-density annotation data cannot provide comprehensive and accurate analysis results on road traffic signs and traffic information, making it difficult to support driving suggestions that require strong logic. Furthermore, deep learning-based image classification models may only output static results, such as traffic signal labels or bounding boxes, lacking the ability to interact with users. This means that when drivers need to understand the specific meaning of a particular signal or ask how to deal with certain road conditions, deep learning-based image classification models cannot meet their needs.

[0094] With the rapid development of intelligent transportation systems and autonomous driving technologies, the level of intelligence in the intelligent cockpit, as the core interface connecting the driver and the vehicle, is becoming a crucial standard for measuring the competitiveness of modern automobiles. An intelligent cockpit not only needs to provide basic safety and comfort guarantees but also should offer a highly personalized interactive experience to meet diverse user needs. Compared to deep learning-based image classification models, visual language models can not only recognize image content but also understand questions based on context and provide detailed answers, offering users a more personalized service experience.

[0095] Therefore, to address the aforementioned problems, this application provides a target model, which is a visual language model. The images are fully annotated, and data augmentation is performed on the annotated images to obtain sample image information. Then, guided by rules, a rich traffic question-and-answer dataset is generated based on this sample image information to ensure the comprehensiveness of the training samples. Subsequently, the model is fine-tuned using this sample image information and its corresponding traffic question-and-answer dataset to obtain the target model, improving its generalization ability and thus enhancing the accuracy of traffic signal recognition and user interactivity. Furthermore, it improves the target model's ability to analyze roads, outputs highly logical driving suggestions, improves driving automation and intelligence, and ensures a better user experience.

[0096] The following section will detail the process of vehicle interaction with user using a target model, which is the traffic question-answering method provided in this application. For example, ... Figure 1 As shown, the method may include S101-S102.

[0097] S101, the vehicle receives traffic problems input by the user.

[0098] In this embodiment, when a user wants to interact with the vehicle, the user can input a traffic question (or user input question) to ask in real time. The input method can be voice, text, etc., and this application is not limited to these methods. Optionally, when the input method is non-textual, such as voice, the vehicle can convert the user's input traffic question into text. It is understood that the vehicle can be equipped with a voice-to-text conversion tool to convert the voice (user-input traffic question) into text.

[0099] S102, in response to the traffic problem, the vehicle outputs the target answer. The target answer is obtained by the target model based on the detection results of traffic signals on the first image and the traffic problem.

[0100] Optionally, the first image may be captured by a camera on the vehicle (or an in-vehicle camera).

[0101] In this embodiment, after receiving a traffic question input by a user, the vehicle can acquire a first image captured in real time by its onboard camera. This first image, along with the traffic question, is used as real-time multimodal data for the target model to perform inference. Specifically, the model detects traffic signals in the first image, obtains the corresponding traffic signal detection results, and then generates a corresponding target answer based on these detection results and the user-input traffic question. The vehicle can then output the target answer, thus effectively engaging in dialogue with the user. It is understood that when the traffic question is input via voice, the vehicle can use the converted text traffic question and the first image as real-time multimodal data.

[0102] Optionally, outputting the target response may include displaying the target response and / or outputting the target response verbally. Similarly, as described above, the vehicle may also be equipped with text-to-speech tools to convert the target response into speech.

[0103] In some embodiments, the detection result of the traffic signal can be detection result 1 or detection result 2. Detection result 1 indicates a detected traffic signal; that is, when a traffic signal is detected in the first image, detection result 1 can be generated, indicating the detected traffic signal. Optionally, detection result 1 indicates a traffic signal related to a traffic problem input by the user. For example, the traffic problem is whether there is a speed limit sign ahead. When a speed limit sign is detected in the first image, detection result 1 indicates the presence of a speed limit sign.

[0104] Detection result 2 indicates that no traffic signal was detected. That is, when no traffic signal is detected in the first image, detection result 2 can be generated, indicating that no traffic signal was detected. Optionally, detection result 2 indicates that no traffic signal was detected related to the traffic problem input by the user.

[0105] In some embodiments, the target answer may include traffic signals on the first image and / or driving suggestions. When the target answer includes traffic signals on the first image, the traffic signals may be traffic signals related to the traffic question input by the user. For example, the traffic question might be whether there is a speed limit sign ahead. If a speed limit sign is detected on the first image, the target answer indicates that there is a speed limit sign ahead. Or, for example, if the traffic question is what traffic signals are ahead, the target answer indicates that there are traffic signals ahead.

[0106] The specifics of the target answer generated by the target model through inference can be found in the section below explaining the process of determining the target model. For example, the target answer can be referenced from answer samples in the traffic question-answering dataset. These details will not be elaborated upon here.

[0107] In this embodiment, the target model has high capabilities and can effectively answer users' traffic questions based on the detection results of traffic signals on the first image, ensuring the accuracy of the answers and improving the interactivity between vehicles and users, thereby enhancing the user experience.

[0108] The reasoning process for the target model has been introduced above. The process of determining the target model will be described below.

[0109] The target model described above was obtained by training the first model using sample image information and its corresponding traffic question-and-answer dataset. The traffic question-and-answer dataset was generated based on the sample image information. The sample image information includes labeled images with traffic signals and augmented images of the labeled images; the augmented images are obtained by performing data augmentation on the labeled images.

[0110] In this embodiment of the application, after obtaining the labeled image and the enhanced image of the labeled image, a corresponding traffic question and answer dataset can be generated based on the labeled image and the enhanced image of the labeled image. The labeled image, the enhanced image of the labeled image, and the traffic question and answer dataset can be used as training samples for the first model.

[0111] In this embodiment, on the one hand, the first model is trained using not only labeled images of traffic signals but also enhanced images of those labeled images, resulting in richer training samples for traffic signals and improving the accuracy of the first model in recognizing traffic signals. On the other hand, by simultaneously utilizing traffic question-and-answer datasets corresponding to both the labeled images and the enhanced images of those labeled images, the training samples for question-and-answer dialogues are also richer, enhancing the interactivity of the first model. Based on this, the capabilities of the target model are improved, enabling it not only to accurately recognize traffic signals but also enhancing its interactivity with the user, resulting in a superior user experience.

[0112] In some embodiments, the sample image information further includes annotation files corresponding to the annotated image and annotation files corresponding to the enhanced image. The annotation files include information about the traffic signals annotated on the image corresponding to the annotation file. Optionally, the traffic signal information may include one or more of the following annotation information: a traffic signal label, its location, and a traffic signal description. The traffic signal label indicates the specific type of traffic signal, such as a traffic sign, road marking, or traffic light. The traffic signal description indicates the specific details of the traffic signal. For example, if the traffic signal annotated in the annotated image includes a speed limit sign of 40 km / h, then the label for that traffic signal could be a speed limit sign, and the description could be a speed limit of 40 km / h.

[0113] In some embodiments, the aforementioned labeled images may be obtained by fully labeling the original labeled images. Correspondingly, the label files corresponding to the labeled images may also be obtained by updating the label files corresponding to the original labeled images. It should be understood that the label file corresponding to each labeled image describes in detail the traffic signals labeled on that labeled image, and the traffic signals labeled on that labeled image can be visualized through the label file corresponding to the labeled image.

[0114] For example, the above-described full-scale annotation process may include a pre-annotation preparation process and an image annotation process. For example... Figure 2 As shown, the preparatory work before labeling includes S201-S203.

[0115] S201. Determine the labels and descriptions of all traffic signals to obtain the label comparison table L.

[0116] S202. For complex traffic signals, closely spaced and scattered traffic signal combinations, and some special cases, the annotation rule Z is compiled.

[0117] For example, a road sign may have different lane markings indicating different lanes, representing a complex traffic signal. Accordingly, the labeling rule Z includes a description of each marking in that complex traffic signal, which may follow certain rules (such as from left to right, from top to bottom).

[0118] At intersections, multiple closely spaced signs (such as no parking, no honking, and speed limit signs) that are not on the same sign are framed as a whole and described as a combination. Accordingly, labeling rule Z can include descriptions of each traffic signal in the combination, which can follow certain rules (such as from left to right or from top to bottom).

[0119] Special cases may include non-traffic signals that are easily confused. For example, a bus may have a speed limit sign, but the speed limit is for the bus itself, not a traffic signal. Therefore, labeling rule Z may include the speed limit sign on the bus and describe the speed limit sign, such as "This is a speed limit sign for a bus, not a traffic signal".

[0120] S203. Select the annotation tool.

[0121] After determining the annotation reference table L and annotation rules Z, a unified annotation tool can be selected from among open-source annotation tools. This tool, combined with the annotation reference table L and annotation rules Z, allows annotators to perform detailed annotations on the original annotated images, achieving full annotation. Specifically, for example... Figure 3 As shown, the image annotation process includes S301-S305 to achieve batch annotation of the original image.

[0122] S301. Select a batch of original images, perform image filtering and pre-annotation on the selected original images to obtain the original annotated images and their corresponding annotation files.

[0123] A batch of original images is selected from the original images. The selected original images are first filtered and pre-annotated. That is, the original images are pre-annotated using an open-source traffic sign recognition model. Images without obvious traffic signals are filtered out to obtain the original annotated images (or pre-annotated images) and their corresponding annotation files.

[0124] S302. Use open-source annotation tools to perform detailed annotations on the original annotated image to obtain an annotated image with detailed annotations and its corresponding annotation file.

[0125] Annotators use open-source annotation tools in accordance with annotation reference table L and annotation rules Z to perform detailed annotations on the original annotated images, resulting in an annotated image with detailed annotations and its corresponding annotation file, thus achieving full annotation.

[0126] S303. Determine if there is an unlabeled original image.

[0127] If all original images have been annotated, then S304 can be executed to end the image annotation process. If there are unannotated original images, then the annotation of the unannotated original images can continue, i.e., S305 can be executed.

[0128] S304. Compile all annotated images and their corresponding annotation files.

[0129] S305, Return to S301.

[0130] Furthermore, the original annotated image and its corresponding annotation file can also be directly determined from publicly available annotated images and their corresponding annotation files. The above describes the annotation process of the original annotated image using annotation tools by annotators as an example. However, the original annotated image can also be automatically annotated using relevant tools, and this application does not limit this.

[0131] In this embodiment, the original labeled images are annotated with high information density to construct fully annotated images, meaning that these annotated images have high information density and provide reliable data for model training. Furthermore, the full-scale annotation scheme provided in this application has strong scalability and fundamental logical consistency.

[0132] The process of determining labeled images has been described above. Data augmentation after image annotation is a crucial step in improving the accuracy and generalization of subsequent model recognition. The following section will continue to describe the process of data augmentation for labeled images.

[0133] In some embodiments, the data augmentation described above may include viewpoint switching data augmentation (or viewpoint zoom-in switching data augmentation) and / or long-tail labeling data augmentation. Viewpoint switching data augmentation is used to change the orientation of traffic signals in the labeled image (i.e., change the viewpoint of the traffic signals in the labeled image) and reduce the sharpness of the labeled image. Long-tail labeling data augmentation is used to replace traffic signals in the labeled image with long-tail labels. Long-tail labels are determined based on the frequency of occurrence of traffic signals.

[0134] In one example, the data augmentation described above may include viewpoint-switching data augmentation. Accordingly, the augmented image of the labeled image may include a first augmented image. The first augmented image is obtained by performing viewpoint-switching data augmentation on the labeled image.

[0135] In another example, the data augmentation described above may include data augmentation of long-tail labels. Accordingly, the augmented image of the labeled image may include a second augmented image. The second augmented image is obtained by performing long-tail label data augmentation on the labeled image.

[0136] In another example, the data augmentation described above may include data augmentation for viewpoint switching. Correspondingly, the augmented image of the labeled image may include a first augmented image and a second augmented image. Alternatively, the augmented image of the labeled image may include a third augmented image. The third augmented image is obtained by performing long-tail label data augmentation on the first augmented image. The following will use the example of an augmented image including a third augmented image to first introduce the implementation process of viewpoint switching data augmentation, and then introduce the implementation process of long-tail label data augmentation.

[0137] For example, the process of determining the first enhanced image described above may include:

[0138] For each labeled image, the labeled image is segmented to obtain multiple first sub-images of the labeled image.

[0139] Each of the multiple first sub-images is magnified to obtain the first enhanced image corresponding to each first sub-image, which is also the first enhanced image of the labeled image. The size of the first enhanced image is the same as the size of the labeled image.

[0140] Based on this, by segmenting and then enlarging the labeled image, not only can the clarity of the traffic signal on the labeled image be changed, but also the position of the traffic signal on the labeled image can be changed (that is, the position of the traffic signal on the first enhanced image has changed compared to the position of the traffic signal on the labeled image). This is equivalent to changing the perspective and bringing the perspective closer, so that the first model can learn traffic signals under different clarity and different perspectives, and avoid the first model being unable to recognize traffic signals with lower clarity due to the labeled image being too high-definition.

[0141] Alternatively, the above-mentioned labeled image can be segmented according to the magnification ratio 'a'.

[0142] Optionally, the first sub-image mentioned above may include a positive sub-image and a negative sub-image. A positive sub-image represents a sub-image labeled with traffic signals. A negative sub-image represents a sub-image without traffic signals. Correspondingly, the enhanced image of the labeled image may include a positive sample image of the labeled image and a negative sample image of the labeled image. The positive sample image is obtained by enlarging the positive sub-image of the labeled image, and the negative sample image is obtained by enlarging the negative sub-image of the labeled image. This ensures a balance between positive and negative samples.

[0143] In some embodiments, the above-described segmentation of the labeled image to obtain multiple first sub-images of the labeled image includes:

[0144] A sliding window is used to slide across the labeled image.

[0145] Based on the enhancement weights of the second subgraph within the sliding window, the second subgraph is designated as the first subgraph. The enhancement weights of the second subgraph are negatively correlated with the frequency of traffic signal occurrences in the second subgraph, and positively correlated with the probability of designating the second subgraph as the first subgraph.

[0146] Specifically, the enhancement weight of the second subgraph is negatively correlated with the frequency of traffic signal occurrences in the second subgraph; the higher the frequency of traffic signal occurrences in the second subgraph, the lower its enhancement weight. Conversely, the lower the frequency of traffic signal occurrences in the second subgraph, the higher its enhancement weight. The enhancement weight of the second subgraph is positively correlated with the probability of it being used as the first subgraph; the higher the enhancement weight, the higher the probability of it being used as the first subgraph. Conversely, the lower the enhancement weight, the lower the probability of it being used as the first subgraph.

[0147] Based on this, the enhancement weight of the second subgraph is determined according to the frequency of traffic signal occurrence in the second subgraph. Then, the enhancement weight of the second subgraph is used to determine whether to use the second subgraph as the first subgraph, suppressing high-frequency traffic signals and enhancing low-frequency traffic signals to ensure the generalization of training samples.

[0148] Optionally, if the second sub-image within the sliding window is marked with a traffic signal, the second sub-image is selected as a candidate positive sub-image based on its enhancement weight. The enhancement weight of the second sub-image is positively correlated with the probability of it being selected as a candidate positive sub-image; the greater the enhancement weight, the higher the probability that the second sub-image will be saved as a candidate positive sub-image. P candidate positive sub-images are randomly selected to obtain the positive sub-image, where p is a positive integer.

[0149] Optionally, the size of the aforementioned sliding window (hereinafter referred to as the sliding window) can be determined according to the magnification ratio a.

[0150] Optionally, the traffic signals in the candidate positive subgraph are complete. That is, the second subgraph is randomly saved according to the enhancement weight of the second subgraph, and if the second image includes complete traffic signals, the saved second subgraph can be used as a candidate positive subgraph.

[0151] Furthermore, the process of determining the negative submap described above may include: if the second submap within the sliding window does not have traffic signals marked, the second submap is used as a candidate negative submap. Then, q candidate negative submaps are randomly selected from the candidate negative submaps of the marked image to obtain the negative submap, where p is a positive integer.

[0152] In one possible implementation, the frequency of occurrence of the traffic signal can be the number of times the traffic signal appears on all labeled images. Optionally, the frequency of occurrence of each traffic signal can be obtained by counting the number of times each traffic signal appears on the labeled images corresponding to each labeled image. Alternatively, a preset frequency range to which the frequency of occurrence of the traffic signal belongs can be found, and the enhancement weight corresponding to the preset frequency range can be used as the enhancement weight of the traffic signal. That is, different preset frequency ranges all have corresponding enhancement weights.

[0153] For example, the following will combine Figure 4 This section details a possible implementation process for the data augmentation involving perspective switching described above. For example... Figure 4 As shown, the data enhancement process for this perspective switching can include S401-S412.

[0154] S401. Determine the hyperparameters of data augmentation (including the magnification ratio a for bringing the viewpoint closer, the augmentation factor p for positive samples, and the augmentation factor q for negative samples).

[0155] S402. Select a batch of labeled images and their corresponding labeled files from all labeled images and their corresponding labeled files, perform batch data analysis, analyze the frequency of occurrence of each traffic signal, and determine the enhancement weight w of each traffic signal.

[0156] Based on this, the enhancement weight w of a traffic signal is determined by utilizing its occurrence frequency. Data augmentation is then performed using this weight w to suppress high-frequency categories and enhance low-frequency categories. Optionally, the occurrence frequency of a traffic signal can represent the number of times the traffic signal appears in a selected batch of labeled images.

[0157] S403. For each labeled image in the batch data, determine the sliding window size according to the magnification ratio a, and then continuously slide the window to obtain the second sub-image of the labeled image.

[0158] After determining the second sub-image of the labeled image, the positive and negative samples in the second sub-image can be identified, and image enhancement can be performed by filtering the positive and negative samples.

[0159] S404. For a second sub-map that includes traffic signals, where the second sub-map should contain complete traffic signals and the second sub-map is randomly saved according to the enhancement weight of the second sub-map, the second sub-map is used as a candidate positive sub-map of the labeled image.

[0160] Since traffic signals on the second subgraph are represented by category sign dividers, meaning each category sign divider includes one traffic signal, a complete traffic signal can be described as a complete category sign divider.

[0161] S405. Randomly select p candidate positive sub-images from all candidate positive sub-images of the labeled image as positive sub-images of the labeled image.

[0162] S406. For a second subgraph that does not contain any traffic signals, the second subgraph is taken as a candidate negative subgraph.

[0163] S407. Select q candidate negative sub-images from all candidate negative sub-images of the labeled image as negative sub-images of the labeled image.

[0164] S408. Enlarge the retained positive and negative sub-images to the original image size to obtain the positive and negative sample images of the labeled image.

[0165] After filtering the positive and negative samples, the retained positive and negative sub-images are enlarged to the original image size to obtain the positive and negative sample images (or positive and negative enhanced images) of the labeled image.

[0166] S409. Synchronously store the positive and negative sample images and their corresponding annotation files.

[0167] The annotation files for positive sample images are obtained by updating existing annotation files. This means that only the positions of the traffic signals need to be updated. The annotation files for negative sample images can be considered as empty.

[0168] S410. Determine if there is an unenhanced labeled image.

[0169] If all annotated images have been augmented, then S411 can be executed to end the data augmentation process involving zooming in and switching perspectives. If there are still annotated images that have not been augmented, then the process returns to the step of selecting a batch of annotated images and their corresponding annotation files from all annotated images and their corresponding annotation files, i.e., S412 is executed.

[0170] S411. Summarize the positive and negative sample images of each labeled image and their corresponding label files.

[0171] S412, Return to S402.

[0172] It should be noted that when there is only one traffic signal in the second sub-graph, the enhancement weight of that traffic signal can be determined based on its frequency of occurrence, and this enhancement weight can be used to randomly save the second sub-graph. When there are multiple traffic signals in the second sub-graph, if there are low-frequency traffic signals, then the second sub-graph can be directly saved randomly based on those low-frequency traffic signals. Alternatively, when there are multiple traffic signals in the second sub-graph, the enhancement weights of the multiple traffic signals can be averaged, weighted, or summed to obtain processed enhancement weights, which can then be used to randomly save the second sub-graph.

[0173] The above describes the process of performing viewpoint switching data augmentation on the labeled image to obtain the first augmented image of the labeled image (including the positive and negative sample images of the labeled image mentioned above). The following describes the process of further performing long-tail label data augmentation on the first augmented image of the labeled image.

[0174] Replace the traffic signals in the image to be replaced with long-tailed signs. The image to be replaced represents a first enhanced image containing only one number of traffic signals. Long-tailed signs include traffic signals that appear less frequently than a preset frequency in all labeled images. The frequency of traffic signal occurrence can be found in the relevant description above.

[0175] Based on this, data augmentation using long-tailed markers can enhance low-frequency traffic signals, thereby improving the generalization of training samples and enabling the first model to learn rare traffic signals, thus enhancing the accuracy of the first model in recognizing traffic signals.

[0176] For example, the image to be replaced could be a positive sample image with only one traffic signal. Figure 5 As shown, the data augmentation process for long-tailed tags may include S501-S507.

[0177] S501. Select positive sample images with only one label as images to be replaced.

[0178] To reduce the impact of the replaced long-tailed signs on the overall road information logic of the image, positive sample images with only one sign were selected as the images to be replaced.

[0179] S502, Perform long-tail tag filtering and generation.

[0180] S503. Select at least one image to be replaced, and combine the at least one image to be replaced with a long-tail sign, and cover the long-tail sign on a single traffic signal of the image to be replaced.

[0181] S504. Modify the label and description of the traffic signal in the annotation file corresponding to the image to be replaced to the label and description of the long-tail sign, and store the long-tail enhanced positive sample image and its corresponding annotation file.

[0182] Among them, the positive sample image can also become the third enhanced image.

[0183] S505. Determine if there are any remaining long-tailed flags.

[0184] If yes, return to the step of selecting at least one image to be replaced to continue covering the remaining long-tailed markers onto the image to be replaced, i.e., execute S506. If no, then execute S507.

[0185] S506, Return to S503.

[0186] S507. Summarize the long-tail enhanced positive sample images of each image to be replaced and their corresponding annotation files.

[0187] Optionally, for Figure 2 If a traffic signal is identified in the image and its frequency of occurrence in all labeled images is lower than a preset frequency, then that traffic signal is designated as a long-tail label.

[0188] In some embodiments, since the third enhanced image includes long-tailed signs, i.e., traffic signals, it can also serve as a positive sample image. Similarly, the second enhanced image described above can also serve as a positive sample image. That is, a positive sample image can include a first enhanced image, a second enhanced image, and / or a third enhanced image obtained by magnifying a positive sub-image of the labeled image.

[0189] It should be noted that the process of determining the second enhanced image is similar to that of determining the third enhanced image described above. To avoid redundancy, the process of determining the second enhanced image will not be described again.

[0190] In this embodiment of the application, in order to address the issues of small volume of original high-quality annotation data and long-tail distribution of sign categories, the limited original annotation data is better utilized by switching perspectives and enhancing the data of long-tail signs. This can generate long-tail category data that is as realistic and diverse as possible, while enhancing the ability of the multimodal model to analyze road information from various perspectives.

[0191] The previous section introduced the determination of labeled images and the data augmentation process. The following section will continue to describe the process of generating a rich traffic question-and-answer dataset based on labeled images and their augmented forms.

[0192] In some embodiments, sample image information and preset rules are input into a second model to obtain a traffic question-and-answer dataset, thereby realizing the generation of a rule-guided traffic question-and-answer dataset.

[0193] Optionally, the second model can be a randomly selected multimodal large model, i.e., a visual language model.

[0194] The second model has more parameters than the first model.

[0195] In this application, a traffic question-and-answer dataset is generated using the capabilities of a large model (i.e., the second model), and a small model (the first model) is trained using this traffic question-and-answer dataset. The trained first model (i.e., the target model) is then used for reasoning. This process is equivalent to knowledge distillation. The target model utilizes the capabilities of the large model to ensure the accuracy and rationality of the reasoning.

[0196] In some embodiments, the traffic question-and-answer dataset mentioned above includes a first traffic question-and-answer dataset for traffic signals on labeled images (including a first traffic question sample and a first answer sample corresponding to the first traffic question sample). The first traffic question-and-answer dataset may also be referred to as a general description dialogue.

[0197] The second traffic question-and-answer dataset (including second traffic question samples and corresponding second answer samples) is a collection of traffic signals on positive sample images of labeled images. This second traffic question-and-answer dataset can also be referred to as positive sample information dialogue.

[0198] And a third traffic question-and-answer dataset (including third traffic question samples and corresponding third answer samples) on negative sample images of labeled images. The third traffic question-and-answer dataset can also be called negative sample information dialogue.

[0199] The first response sample includes traffic signals on the labeled image, and / or driving advice. The second response sample includes traffic signals on a positive sample image of the labeled image, and / or driving advice. The third response sample includes preset response sample information, and / or driving advice.

[0200] Based on this, the traffic question-and-answer dataset can include traffic question-and-answer datasets corresponding to labeled images and traffic question-and-answer datasets corresponding to positive and negative sample images, ensuring the richness of the traffic question-and-answer dataset, thereby ensuring the richness of the question-and-answer dialogue, and further ensuring the generalization of the target model, improving the interactivity between the target model and the user, and enabling the target model to accurately provide the answers that the user needs.

[0201] In one possible design approach, the aforementioned preset rules may include a traffic rule base T, a traffic signal description base S (or traffic sign description base S), and a set of traffic signals O requiring enhanced negative sample dialogue (or a set of signs O requiring enhanced negative sample dialogue). The traffic rule base S is consistent with actual traffic rules, thus providing accurate driving advice. The traffic signal description base S can be obtained based on the shape and color of traffic signals (such as traffic signs). The set of traffic signals O requiring enhanced negative sample dialogue can be obtained based on high-frequency traffic signals in labeled images and their corresponding enhanced images. In this application, high-frequency traffic signals can be traffic signals that occur more frequently than a certain frequency. It should be understood that the preset rules can be compiled by relevant personnel or automatically compiled by relevant equipment (such as servers or other equipment).

[0202] For example, such as Figure 6 As shown, the process of generating the traffic question-and-answer dataset can include S601-S604.

[0203] S601. Obtain preset rules (including traffic rule base T, traffic sign description base S, and sign set O that needs to enhance negative sample dialogue).

[0204] S602. Input the preset rules, sample image information and their corresponding traffic question and answer dataset into the second model.

[0205] S603. The second model iterates through each image and its corresponding annotation file in the sample image information. For each image and its corresponding annotation file, the second model performs hard filtering on the image and its corresponding annotation image to obtain the filtered sample image information.

[0206] Hard filtering refers to filtering out traffic signals in the image whose size is smaller than a preset size (or, in other words, filtering out traffic signals whose proportion in the image is smaller than a preset proportion) and the annotation information of that traffic signal in the corresponding annotation file; that is, ignoring smaller traffic signals. The proportion of a traffic signal indicates the ratio of its size to the entire image. Additionally, the image in the sample image information in S603 may be referred to as the dialogue image to be generated.

[0207] S604. The second model can generate multi-turn dialogues based on the filtered sample image information to obtain the traffic question-and-answer dataset corresponding to each image.

[0208] The traffic question-answering dataset includes overall descriptive dialogue generation for labeled images, positive sample information dialogues for each positive sample image of the labeled image, and negative sample information dialogues for each negative sample image of the labeled image. The overall descriptive dialogue generation for the labeled image, the positive sample information dialogues for each positive sample image of the labeled image, and the negative sample information dialogues for each negative sample image of the labeled image can be integrated as multi-turn dialogue training data for the labeled image. Once all images in the sample image information have been traversed, it indicates that the traffic question-answering dataset for each image has been obtained, and dialogue generation can then end.

[0209] For example, in the first traffic question in the overall description dialogue, questions need to be asked about all traffic signals in the labeled image. The corresponding answer (i.e., the first answer sample) includes the recognition results of traffic signals on the labeled image. Guided by the corresponding rules selected from the traffic rule base T, soft filtering of traffic signals is performed, that is, ignoring non-critical traffic signals, and providing comprehensive driving advice based on the driving environment. For example, if the first traffic question is what signs are ahead, the corresponding first answer sample would be: There are signs for a narrow road, speed limit, and weight limit ahead, and the road ahead is about to narrow, so the driver should pay attention to the speed limit of 30 km / h and try to stay in the left lane. There is also a 20-ton weight limit sign ahead, so please ensure the vehicle meets the requirements, and no traffic lights are seen ahead (driving advice). It should be understood that because the second model has basic learning capabilities, it can filter out non-critical traffic signals, such as speed limit and no-parking signs ahead, so the no-parking sign can be considered a non-critical traffic signal.

[0210] For the second traffic question in the dialogue based on the positive sample information of the labeled image, the question should be asked under the guidance of relevant descriptions filtered from the traffic sign description library S, such as asking about traffic signals in the positive sample image. The corresponding answer (i.e., the second answer sample) includes the recognition result of the traffic signal on the positive sample image, and under the guidance of the corresponding rules filtered from the relevant traffic rules T, combined with the driving environment, it provides key driving suggestions. For example, the second traffic question is, "Is there a weight limit ahead?" The corresponding second answer sample is, "There is a weight limit sign slightly to the right ahead, with a weight limit of 20 tons. Please pay attention to controlling the total weight of your vehicle to meet the requirements (driving suggestion)." In addition, driving suggestions can also include important information that has been discovered, such as, "The road ahead is about to narrow; please drive carefully."

[0211] For dialogue based on negative sample information, the second model can traverse the label set O. If the traffic signal in label set O is not present in the negative sample image, a question about that traffic signal is posed (i.e., a third traffic question is generated). The corresponding response must indicate that no relevant traffic signal was found (i.e., pre-defined response sample information), and provide appropriate driving advice based on the traffic signals in the corresponding positive sample image (i.e., the positive sample image of the labeled image to which the negative sample image belongs). For example, the third traffic question might be: What traffic signal is ahead? The third response sample would indicate that no relevant traffic signal was found ahead, along with important information found (e.g., the road ahead is about to narrow; please proceed with caution).

[0212] It is understandable that the operations performed by the second model are actually performed by the device on which the second model resides. For example, if the second model is located on a server, then the operations performed by the second model are operations performed by the server.

[0213] In this embodiment, considering that users focus on the more important information in an image during conversations, and that important information typically constitutes a larger proportion of the content, and given the word limit in user conversations, extracting important information is performed, which is equivalent to hard filtering. Furthermore, it should be understood that hard filtering is an optional step.

[0214] In some embodiments, after obtaining sample image information and its corresponding traffic question-and-answer dataset, the first model can be trained using the sample image information and its corresponding traffic question-and-answer dataset to obtain the target model.

[0215] For example, the first model is fine-tuned and trained using sample image information and its corresponding traffic question-and-answer dataset, and the trained first model is used as the target model. Optionally, the target model can be a visual language model.

[0216] Optionally, to save fine-tuning time, LoRA technology can be used to fine-tune the first model to improve training speed.

[0217] In some embodiments, the training process of the first model described above may be performed on a vehicle, server, or other device, and this application does not limit it. Similarly, the sample image information and traffic question-and-answer dataset described above may also be performed on a vehicle, server, or other device.

[0218] In this application, a traffic question-and-answer dataset is generated based on the labeled images and their corresponding labeled files, the enhanced images of the labeled images and their corresponding labeled files. That is, a multimodal large model single-graph multi-turn dialogue is generated for the overall traffic signal situation, individual traffic signals, and traffic signals that may not exist. This ensures the reliability, breadth and balance of the fine-tuning data. The model is fine-tuned by constructing a balanced set of positive and negative samples, which can effectively alleviate the illusion problem of large models.

[0219] In some embodiments, the target model described above can be deployed on vehicles, or it can be deployed in the cloud. Vehicle-cloud collaboration can implement traffic question-answering methods. For example... Figure 7 As shown, the traffic question-and-answer method may include S701-S702.

[0220] S701, the vehicle sends a first image and a traffic problem input by the user to the server. Correspondingly, the server receives the first image from the vehicle and the traffic problem input by the user.

[0221] S702, The server sends the target response to the vehicle. Correspondingly, the vehicle receives the target response from the server.

[0222] The target answer is obtained by the server using real-time multimodal data as input parameters to the target model, running the target model, and obtaining the corresponding target answer.

[0223] Optionally, the server can concatenate the system prompt and the traffic question, using the concatenated text and the first image as input parameters for the target model. The system prompt indicates the target model's role in the dialogue and its intended function. This prompt can be adjusted as needed and is generally concise. For example, the system prompt could state that, as an intelligent driving assistant, it needs to help the user analyze traffic information ahead and provide driving assistance. The system prompt can be preset or determined based on the user's input of traffic questions or other requirements; simply put, the system prompt corresponds to the traffic question.

[0224] The following example illustrates the process of traffic signal recognition and interaction. Figure 8As shown, the process can be divided into three stages. In stage one, the target model is deployed in the cloud, and system prompts are provided to the large model during interactions. Additionally, speech-to-text and text-to-speech conversion tools are deployed in the vehicle. In stage two, the vehicle acquires the user's input of a traffic question (either text or speech-converted) and the first image captured in real-time by the vehicle's camera, transmitting this real-time multimodal data to the cloud. The cloud performs the necessary user authentication operations required for cloud services and concatenates the system prompts and traffic questions as text information used for target model inference. In stage three, cloud-based target model inference is performed first. The input parameters for target model inference include the text information and image data. The target model's inference output generates an accurate target answer based on the real-time multimodal data. This target answer undergoes sensitive information evaluation and interception before being output to the vehicle to determine if it meets the requirements. If the target answer meets the requirements, it is transmitted back from the cloud to the vehicle, displayed on the vehicle's screen, and can be converted into speech output.

[0225] Additionally, if the target answer does not meet the requirements, the cloud can process it to make it acceptable, such as by deleting the non-compliant parts. The cloud can then send the processed target answer back to the vehicle. Alternatively, the cloud can perform other operations, such as directly sending a preset answer back to the vehicle, indicating that the question cannot be answered.

[0226] In this embodiment, the vehicle only performs voice-to-text and text-to-speech conversion, user questioning, and real-time image acquisition and uploading. The deployment and inference of the target model (or multimodal large model) are all done in the cloud. The target model can not only recognize image content, but also understand questions based on context and provide detailed answers, providing users with a more personalized service experience (such as generating comprehensive and personalized driving suggestions for drivers). At the same time, there are no vehicle-side computing power limitations, reducing vehicle-side costs and increasing applicability.

[0227] It should be noted that the above example illustrates the question-answering process using a target model deployed in the cloud. The target model can also be deployed on vehicles, allowing them to directly utilize it for reasoning and achieve real-time responses. The process of reasoning using the target model in a vehicle is similar to the cloud-based process described above, and will not be repeated here.

[0228] In some embodiments, this application provides a traffic question-answering system, which includes one or more of the following: a data annotation model, a data augmentation module, a traffic question-answering dataset construction module, a multimodal large model fine-tuning module, and a vehicle-cloud collaborative interactive question-answering module. The data annotation module is used to implement the full-scale annotation described above. The data augmentation module is used to implement the data augmentation described above. The traffic question-answering dataset construction module is used to generate the traffic question-answering dataset described above. The multimodal large model fine-tuning module is used to obtain the target model described above. The vehicle-cloud collaborative interactive question-answering module is used to implement the traffic question-answering described above.

[0229] In some embodiments, the above method can be applied to other types of electronic devices, to systems containing electronic devices, or to chips within electronic devices. For example, it can be applied to chips in vehicles. This application does not limit the scope of the application.

[0230] For example, Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. For example, the electronic device may be included in a vehicle.

[0231] like Figure 9 As shown, the electronic device 100 may include a processor 110. Optionally, the electronic device 100 may also include a memory 120 and a display screen 130, etc.

[0232] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0233] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0234] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0235] In some embodiments, processor 110 may include one or more interfaces. These one or more interfaces can be used to connect processor 110 to memory 120, display 130, etc.

[0236] In some embodiments of this application, the processor 110 can be used to run a model, thereby generating first traffic information. Please refer to the following text for details.

[0237] The memory 120 can be used to store computer executable program code, which includes instructions. The memory 120 may include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as image playback function), etc. The data storage area may store data created during the use of the electronic device 100. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the memory 120 and / or instructions stored in memory located in the processor.

[0238] Electronic device 100 implements display functions through a GPU, display screen 130, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 130 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0239] The display screen 130 is used to display images, videos, etc. The display screen 130 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 130, where N is a positive integer greater than 1.

[0240] In some embodiments of this application, the display screen 130 can be used to display various interfaces. For example, displaying the target answer.

[0241] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include... Figure 9 The diagram shows more or fewer components, or combinations of components, or separate components, or different arrangements of components. The components shown can be implemented in hardware, software, or a combination of both.

[0242] like Figure 10 This application illustrates one possible structure of a vehicle according to an embodiment of the present application, which includes the device / electronic equipment described in any of the embodiments herein. The vehicle possesses all the beneficial effects of the end-side electronic equipment, etc., as described in this embodiment. In other embodiments, the vehicle may also have some or all of the cloud-based functions / effects; for example, models may also be deployed within the vehicle. Further details are omitted here.

[0243] For example, the vehicle may be a plug-in hybrid electric vehicle or a new energy vehicle, etc., and this application does not specifically limit it.

[0244] In some embodiments, this application also provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the traffic signal recognition method described above. This non-transitory computer-readable storage medium possesses all the beneficial effects of the traffic signal recognition method described above, which will not be elaborated further here. This computer-readable storage medium can be used in a vehicle or as a server.

[0245] In some embodiments, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the traffic signal recognition method described above. This computer program product possesses all the beneficial effects of the traffic signal recognition method described above, which will not be elaborated upon further herein. This computer program product can be used in a vehicle or on a server.

[0246] In some embodiments, this application also provides an electronic device, including: a memory and a processor, wherein a computer program is stored in the memory; the processor is used to execute the computer program in the memory to implement the steps of the traffic signal recognition method described above. This electronic device possesses all the beneficial effects of the traffic signal recognition method described above, which will not be elaborated upon further herein.

[0247] In some embodiments, this application also provides a server, including a memory and a processor, wherein a computer program is stored in the memory; the processor is used to execute the computer program in the memory to implement the steps of the traffic signal recognition method described above. This server possesses all the beneficial effects of the traffic signal recognition method described above, which will not be elaborated upon further herein.

[0248] In some embodiments, this application also provides a vehicle that includes the electronic device described in any of the above embodiments. This vehicle possesses all the beneficial effects of the aforementioned electronic device, etc., which will not be elaborated upon further herein.

[0249] The aforementioned computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof, and this application does not specifically limit it. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0250] In some embodiments of this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used or combined with an instruction execution system, apparatus, or device.

[0251] The aforementioned computer-readable storage medium may be included in the aforementioned electronic device, or it may exist independently without being assembled into the electronic device. The aforementioned computer-readable storage medium carries one or more programs that, when executed by the electronic device, cause the control chip in the electronic device to communicate with the intelligent network system via Ethernet through a physical layer device.

[0252] Computer program code for performing operations of some embodiments of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0253] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function.

[0254] It should also be noted that in some alternative implementations, the functions marked in the box may occur in a different order than those marked in the attached figures.

[0255] For example, two consecutively represented blocks can actually be executed in substantially parallel order, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, as well as combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0256] The units described in some embodiments of this application can be implemented in software or in hardware.

[0257] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0258] like Figure 11The diagram shown is a structural schematic of another electronic device provided in an embodiment of this application. This electronic device 2200 can be used to implement the methods described in the above method embodiments. For example, the electronic device 2200 may specifically include a processing unit 2201. Optionally, the electronic device 2200 may further include a display unit 2202.

[0259] The processing unit 2201 is used to support the electronic device 2200 in performing operations. Figures 2 to 10 The processing function described in any one of the following.

[0260] The display unit 2202 is used to support the electronic device 2200 in performing its functions. Figures 2 to 10 The display function described in any one of the following statements.

[0261] Optional, Figure 11 The illustrated electronic device 2200 may also include a communication unit ( Figure 11 (Not shown in the image), this communication unit is used to support electronic device 2200 in performing the steps of communication between electronic device and other electronic devices in the embodiments of this application.

[0262] Optional, Figure 11 The illustrated electronic device 2200 may further include a storage unit 2203 that stores programs or instructions. When the processing unit 2201 executes the program or instructions, it causes... Figure 11 The electronic device 2200 shown can perform the method described in the above-described method embodiments.

[0263] Figure 11 The technical effects of the electronic device 2200 shown can be referred to the technical effects of the method shown in the above method embodiments, and will not be repeated here. Figure 11 The processing unit 2201 involved in the illustrated electronic device 2200 can be implemented by a processor or processor-related circuit components, and can be a processor or processing module. The communication unit can be implemented by a transceiver or transceiver-related circuit components, and can be a transceiver or transceiver module. The display unit 2202 can be implemented by display screen-related components.

[0264] This application also provides a chip system, such as... Figure 12As shown, the chip system includes at least one processor 2301 and at least one interface circuit 2302. The processor 2301 and the interface circuit 2302 are interconnected via lines. For example, the interface circuit 2302 can be used to receive signals from other devices. As another example, the interface circuit 2302 can be used to send signals to other devices (e.g., the processor 2301). Exemplarily, the interface circuit 2302 can read instructions stored in memory and send those instructions to the processor 2301. When the instructions are executed by the processor 2301, the electronic device can perform the various steps performed by the electronic device in the above embodiments. Of course, the chip system may also include other discrete devices, and this application embodiment does not specifically limit this.

[0265] Optionally, the chip system may contain one or more processors. These processors can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0266] Optionally, the chip system may contain one or more memories. The memory may be integrated with the processor or disposed separately from it; this application does not limit this. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or disposed separately on different chips. This application does not specifically limit the type of memory or the arrangement of the memory and processor.

[0267] For example, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0268] It should be understood that each step in the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The method steps disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0269] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified. In the above embodiments, the descriptions of each embodiment have their own emphasis; parts not described in detail in a certain embodiment can be referred to the relevant descriptions of other embodiments.

[0270] The embodiments, implementation methods, and related technical features of this application can be combined and substituted for each other without conflict.

[0271] The above are merely preferred embodiments of this application and are not intended to limit this application in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of this application without departing from the scope of the technical solution of this application shall still fall within the scope of the technical solution of this application.

Claims

1. A traffic question-and-answer method, characterized in that, include: Receive traffic-related questions input by users; In response to the traffic problem, output the target answer; The target answer is obtained by the target model based on the detection results of traffic signals on the first image and the traffic question.

2. The method according to claim 1, characterized in that, The method further includes: Send the first image and the traffic problem to the server; Receive the target response from the server.

3. The method according to claim 1, characterized in that, The target model is obtained by training the first model using sample image information and its corresponding traffic question-and-answer dataset. The traffic question-and-answer dataset is generated based on the sample image information; the sample image information includes labeled images with traffic signals and enhanced images of the labeled images, the enhanced images being obtained by performing data augmentation on the labeled images.

4. The method according to claim 3, characterized in that, The enhanced image includes a first enhanced image; The first enhanced image is obtained by performing viewpoint switching data enhancement on the labeled image; the viewpoint switching data enhancement is used to change the orientation of traffic signals in the labeled image and reduce the clarity of the labeled image.

5. The method according to claim 3 or 4, characterized in that, The enhanced image includes a second enhanced image; The second enhanced image is obtained by performing long-tailed marker data enhancement on the labeled image; the long-tailed marker data enhancement is used to replace traffic signals on the labeled image with long-tailed markers; the long-tailed markers are determined based on the frequency of occurrence of the traffic signals.

6. The method according to claim 3 or 4, characterized in that, The enhanced image includes a third enhanced image; The third enhanced image is obtained by performing long-tail label data enhancement on the first enhanced image; the first enhanced image is obtained by performing viewpoint switching data enhancement on the labeled image.

7. The method according to claim 6, characterized in that, The method further includes: The labeled image is segmented to obtain multiple first sub-images of the labeled image; Each of the plurality of first sub-images is enlarged to obtain a first enhanced image corresponding to each first sub-image; The size of the first enhanced image is the same as the size of the labeled image.

8. The method according to claim 7, characterized in that, The first subgraph includes a positive subgraph and a negative subgraph; the positive subgraph represents a subgraph marked with traffic signals; the negative subgraph represents a subgraph not marked with traffic signals.

9. The method according to claim 7, characterized in that, The segmentation of the labeled image to obtain multiple first sub-images of the labeled image includes: A sliding window is used to slide over the labeled image; Based on the enhancement weight of the second subgraph within the sliding window, the second subgraph is used as the first subgraph; The enhancement weight of the second subgraph is negatively correlated with the frequency of traffic signal occurrence in the second subgraph, and the enhancement weight of the second subgraph is positively correlated with the probability of using the second subgraph as the first subgraph.

10. The method according to claim 9, characterized in that, The first subgraph is included in the positive subgraph; The step of using the second subgraph as the first subgraph based on the enhancement weight of the second subgraph within the sliding window includes: When a traffic signal is marked on the second sub-image within the sliding window, the second sub-image is selected as a candidate positive sub-image based on its enhancement weight; wherein, the enhancement weight of the second sub-image is positively correlated with the probability of selecting the second sub-image as the candidate positive sub-image. Randomly select p candidate positive subgraphs to obtain the positive subgraph, where p is a positive integer.

11. The method according to claim 6, characterized in that, The long-tail sign includes traffic signals that appear less frequently than a preset frequency in all labeled images; the method further includes: Replace the traffic signals on the image to be replaced with long-tailed signs; wherein the image to be replaced represents a first enhanced image containing a number of traffic signals.

12. The method according to claim 3 or 4, characterized in that, The sample image information also includes the annotation file corresponding to the labeled image and the annotation file corresponding to the enhanced image. The annotation file includes information about traffic signals annotated on the image corresponding to the annotation file.

13. The method according to claim 3 or 4, characterized in that, The labeled image is obtained by fully annotating the original labeled image.

14. The method according to claim 3 or 4, characterized in that, The method further includes: The sample image information and preset rules are input into the second model to obtain the traffic question and answer dataset.

15. The method according to claim 14, characterized in that, The second model has more parameters than the first model.

16. The method according to claim 8, characterized in that, The enhanced image of the labeled image includes a positive sample image of the labeled image and a negative sample image of the labeled image. The positive sample image includes one or more of a first enhanced image, a second enhanced image, and a third enhanced image. The first enhanced image is obtained based on the positive sub-image of the labeled image, and the negative sample image is obtained by enlarging the negative sub-image of the labeled image. The traffic question-and-answer dataset includes a first traffic question sample for traffic signals on the labeled image and a first answer sample corresponding to the first traffic question sample. A second traffic question sample and a second answer sample corresponding to the traffic signal on the positive sample image of the labeled image; And a third traffic problem sample on the negative sample image of the labeled image and a third answer sample corresponding to the third traffic problem sample; The first sample response includes traffic signals on the labeled image, and / or driving advice; The second response sample includes traffic signals on a positive sample image of the labeled image, and / or driving advice; The third response sample includes preset response sample information and / or driving suggestions.

17. A traffic question-and-answer method, characterized in that, include: Receive the first image sent by the vehicle and the traffic problem input by the user; Send the target response to the vehicle; The target answer is obtained by the target model based on the detection results of traffic signals on the first image and the traffic question.

18. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the method as described in any one of claims 1 to 16.

19. A vehicle, characterized in that, Including the electronic device as described in claim 18.

20. A server, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the method as described in claim 17.

21. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1 to 17.

22. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1 to 17.

Images