A road safety hidden danger identification processing method, device and medium

Abstract

The application is suitable for the field of road safety technology, and provides a road safety hidden danger identification processing method, equipment and medium. The method comprises the following steps: training an original large language model by using a trained student model and an instruction fine-tuning to obtain a trained large language model; inputting image data of a road into the trained student model to obtain a road hidden danger classification result; inputting the image data into the trained large language model to obtain a road hidden danger description text; obtaining a similarity between the road hidden danger description text and the road hidden danger classification result; if the similarity does not satisfy a preset similarity condition, adjusting parameters of the large language model, taking the adjusted large language model as the trained large language model, and re-obtaining the road hidden danger description text of the image data; and if the preset similarity condition is satisfied, outputting a hidden danger solution scheme corresponding to the road hidden danger description text by using the trained large language model. The application can improve the efficiency and accuracy of road safety hidden danger investigation.

Description

Technical Field

[0001] This application belongs to the field of road safety technology, and in particular relates to a method, equipment and medium for identifying and handling road safety hazards. Background Technology

[0002] Statistics show that by the end of 2022, the total length of rural roads in my country had increased to 4.53 million kilometers, including 700,000 kilometers of county roads, 1.24 million kilometers of township roads, and 2.59 million kilometers of village roads. Low-grade roads accounted for over 80% of these, making them crucial channels for rural residents' daily travel. However, due to the generally low level of road infrastructure and technical standards in rural areas, coupled with insufficient traffic management resources and public safety awareness, road traffic safety has become a significant risk factor. In this complex traffic environment, identifying and addressing various potential safety hazards is crucial for effectively managing and resolving road safety issues.

[0003] Traditional road hazard investigations often rely on experienced professionals in traffic engineering to conduct on-site surveys. This method requires a large workforce and repetitive labor, covering a wide area of ​​the road network. Furthermore, expert observations can be influenced by subjective factors, leading to differing assessments of the same road by different experts.

[0004] In summary, the current methods for identifying road safety hazards suffer from low efficiency and low accuracy. Summary of the Invention

[0005] This application provides a method, device, and medium for identifying and processing road safety hazards, which can solve the problems of low efficiency and accuracy in identifying road safety hazards.

[0006] In a first aspect, embodiments of this application provide a method for identifying and processing road safety hazards, including:

[0007] The original large language model was trained using the trained student model and instruction fine-tuning to obtain the trained large language model; the instruction fine-tuning data was safety knowledge data in the traffic field; the trained student model was obtained by distilling the teacher model using multiple labeled road image training samples and the student model; the teacher model was a pre-trained image classification model.

[0008] The image data of the road that needs to be identified for safety hazards is input into the trained student model to obtain the road hazard classification results of the image data.

[0009] The image data is input into the trained large language model to obtain the road hazard description text of the image data;

[0010] Obtain the similarity between the description text of road hazards and the classification results of road hazards;

[0011] If the similarity does not meet the preset similarity conditions, the parameters of the large language model are adjusted, and the adjusted large language model is used as the trained large language model. Then, the process of inputting the image data into the trained large language model to obtain the road hazard description text of the image data is returned.

[0012] If the similarity meets the preset similarity conditions, the trained large language model will be used to output the hazard solution corresponding to the road hazard description text.

[0013] Optionally, the original large language model can be trained using the trained student model and instruction fine-tuning to obtain a trained large language model, including:

[0014] The original large language model was trained using road image samples and the trained student model to obtain an intermediate large language model;

[0015] Safety knowledge data in the transportation field is used as instruction fine-tuning data to fine-tune the intermediate large language model, resulting in the trained large language model.

[0016] Optionally, the original large language model is trained using road image samples and the trained student model to obtain an intermediate large language model, including:

[0017] Input road image samples into the trained student model to obtain road hazard classification results for the road image samples;

[0018] Input the road image samples into the original large language model to obtain the road hazard description text of the road image samples;

[0019] Calculate the similarity between the road hazard classification results and the road hazard description text of road image samples;

[0020] If the similarity between the road hazard classification result and the road hazard description text of the road image sample does not meet the preset similarity condition, the parameters of the original large language model are adjusted, and the adjusted large language model is used as the original large language model. Then, the process is returned to execute the step of inputting the road image sample into the original large language model to obtain the road hazard description text of the road image sample.

[0021] If the similarity between the road hazard classification result and the road hazard description text of the road image sample meets the preset similarity condition, then the original large language model will be used as the intermediate large language model.

[0022] Optionally, safety knowledge data from the transportation sector can be used as instruction fine-tuning data to fine-tune the intermediate large language model, resulting in a trained large language model, including:

[0023] Safety knowledge data in the transportation field is used as instruction fine-tuning data to fine-tune the intermediate large language model, resulting in a fine-tuned large language model.

[0024] The weights of the original large language model and the fine-tuned large language model are merged to obtain the trained large language model.

[0025] Optionally, obtain the similarity between the road hazard description text and the road hazard classification results, including:

[0026] Feature extraction is performed on the road hazard description text and the road hazard classification result by semantic extraction network to obtain the first feature vector of the road hazard description text and the second feature vector of the road hazard classification result.

[0027] Calculate the cosine similarity between the first feature vector and the second feature vector, and use the cosine similarity as the similarity between the road hazard description text and the road hazard classification result.

[0028] Optionally, the cosine similarity between the first and second eigenvectors is calculated, including:

[0029] Through formula The cosine similarity between the first and second eigenvectors was calculated. similarity(A,B) ;

[0030] Where A represents the first eigenvector, B represents the second eigenvector, ||A|| represents the norm of A, and ||B|| represents the norm of B.

[0031] Optionally, the preset similarity condition is: the similarity is greater than a preset threshold.

[0032] Secondly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned road safety hazard identification and processing method.

[0033] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned road safety hazard identification and processing method.

[0034] The above-mentioned solution in this application has the following beneficial effects:

[0035] In the embodiments of this application, the original large language model is trained using a student model obtained through distillation learning and instruction fine-tuning to obtain a trained large language model. Then, the trained large language model and the trained student model are used to process image data of roads requiring safety hazard identification, resulting in corresponding road hazard classification results and road hazard description text. When the similarity between the road hazard classification results and the road hazard description text meets a preset similarity condition, the trained large language model outputs the hazard solution corresponding to the road hazard description text. Specifically, the knowledge distillation technique transfers knowledge from the large image classification model to the large language model, optimizing the image classification task of the large language model, enhancing its performance in recognizing low-resolution and highly complex images, and improving the accuracy of road safety hazard investigation. Simultaneously, the use of automated algorithms for road safety hazard investigation significantly improves investigation efficiency compared to traditional methods.

[0036] Furthermore, this application embodiment utilizes safety knowledge data in the transportation field to fine-tune the large language model, enabling the large language model to deeply understand traffic scenarios, accurately analyze and predict traffic safety issues, and propose specific professional solutions for identified road hazards, thereby providing reliable support and guidance for relevant decision-making.

[0037] Other beneficial effects of this application will be described in detail in the following detailed description section. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A flowchart illustrating a road safety hazard identification and processing method provided in an embodiment of this application;

[0040] Figure 2 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0041] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0042] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0043] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0044] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0045] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0046] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0047] To address the low efficiency and accuracy of current road safety hazard identification methods, this application provides a road safety hazard identification and processing method. This method trains an original large language model using a student model obtained through distillation learning and instruction fine-tuning, resulting in a trained large language model. Then, the trained large language model and the trained student model are used to process image data of roads requiring safety hazard identification, yielding corresponding road hazard classification results and road hazard description text. When the similarity between the road hazard classification results and the road hazard description text meets a preset similarity condition, the trained large language model outputs the hazard solution corresponding to the road hazard description text. Specifically, knowledge distillation technology transfers knowledge from a large image classification model to the large language model, optimizing the image classification task of the large language model and enhancing its performance in recognizing low-resolution and highly complex images, thereby improving the accuracy of road safety hazard identification. Furthermore, the use of automated algorithms for road safety hazard identification significantly improves efficiency compared to traditional methods.

[0048] Furthermore, this application embodiment utilizes safety knowledge data in the transportation field to fine-tune the instructions of the large language model, enabling the large language model to deeply understand traffic scenarios, make accurate analyses and predictions of traffic safety issues, and propose specific professional solutions for identified road hazards, thereby providing reliable support and guidance for relevant decision-making.

[0049] The road safety hazard identification and processing method provided in this application will be illustrated below with reference to specific embodiments.

[0050] like Figure 1 As shown in the embodiments of this application, the road safety hazard identification and processing method includes the following steps:

[0051] Step 11: Use the trained student model and instruction fine-tuning to train the original large language model to obtain the trained large language model.

[0052] The fine-tuning data for the aforementioned instructions is safety knowledge data in the transportation field (including but not limited to traffic rules, vehicle driving patterns, and types of traffic accidents); the original large language model can be a generative AI model, such as the GPT model or the ChatGLM model. The GPT model is a natural language processing model based on the Transformer architecture, and the ChatGLM model is an open-source dialogue language model based on the GLM architecture. GLM is a general pre-training framework for natural language understanding and generation.

[0053] The trained student model is obtained by distilling the teacher model using multiple labeled road image training samples and the student model. The teacher model is a pre-trained image classification model (i.e., a trained image classification model). The output of this image classification model is the road hazard classification result. For example, the road hazard classification result can be: worn traffic markings, non-standard traffic markings, missing traffic markings, non-standard traffic sign settings, missing roadbed guardrails, insufficient roadbed guardrail height, insufficient configuration of at-grade intersection facilities, and non-standard speed limit facility settings, etc.

[0054] In some embodiments of this application, the image classification model described above can be obtained by training a deep convolutional neural network using a large amount of image training data and its corresponding labels. The specific questions and classification labels corresponding to the image training data are shown in Table 1.

[0055]

[0056] Table 1

[0057] When constructing a teacher-student structure based on a neural network model, the teacher model has three fully connected layers, with the number of neurons corresponding to the image pixel values. The student model also has three fully connected layers, but with far fewer neurons than the teacher model. For example, both the teacher and student models are deep convolutional neural networks.

[0058] During training, unlabeled image data can first be converted into tensor patterns that the model can process using the Transform function. This converted data is then input into the teacher model in an unsupervised pre-training manner to learn the general representation features of the images. The model's parameter values ​​are preserved and transferred to the student model. Then, the trained student model is obtained through distillation learning (i.e., knowledge distillation training). The knowledge distillation training process is as follows:

[0059] Labeled image data is input into teacher and student models for training, and the model loss is calculated. The model loss includes the hard target loss L. hard And soft target loss L soft L hard =-∑ i p i log(q stu，i ), p i Let q be the probability of the i-th category of the true label. stu，i q represents the probability of the i-th category output by the student model, T is the distillation temperature parameter, and q tea,i Let q represent the probability of the i-th category output by the teacher model, KL represent the divergence, i.e., the relative entropy between two probability distributions, and softmax(q / T) represent the probability vector q (i.e., the q above)stu，i q tea,i Softmax normalization is performed and divided by the temperature parameter. To better obtain the correlation between other parameters, the distillation temperature is usually taken as a value much greater than 1, and the value is determined according to the parameter scale and the size of the input image data. In this embodiment, the value is taken as 70 based on experience. The expression for the probability vector q is: z i To calculate the logits of the output for each category, where T is the distillation temperature parameter and q... i (i.e., the above q) stu，i q tea,i ) The probability output for each category.

[0060] Taking into account both hard target loss and soft target loss, they can be weighted and averaged to obtain the final model loss L. total The weights of the weighted average are controlled by a hyperparameter α, which typically takes a value between [0,1]. Empirically, α = 0.3. The formula for calculating the total loss function of the weighted average is:

[0061] L total =αL hard +(1-α)L soft

[0062] After obtaining the total loss function, set the number of iterations, perform backpropagation on the student model parameters, and update the student parameter model.

[0063] It should be noted that distillation learning is a commonly used model training method, therefore, the process will not be elaborated upon here. The process of training the original large language model using the trained student model and instruction fine-tuning will be described in detail later.

[0064] Step 12: Input the image data of the road that needs to be identified for safety hazards into the trained student model to obtain the road hazard classification results of the image data.

[0065] The image data of the roads requiring safety hazard identification can be collected in real time using sensors, cameras, and other equipment deployed on the roads. After inputting this image data into a trained student model, the model will identify the road hazard and obtain the classification result (such as non-standard traffic sign placement).

[0066] Step 13: Input the image data into the trained large language model to obtain the road hazard description text of the image data.

[0067] In some embodiments of this application, after inputting the image data of the road that needs to be identified into the trained large language model, the large language model will perform identification processing on the image data to obtain a text description of the image data (i.e., the above-mentioned road hazard description text), which is used to describe traffic safety hazards in the image data, such as wear and tear on traffic markings.

[0068] Step 14: Obtain the similarity between the road hazard description text and the road hazard classification results.

[0069] In some embodiments of this application, a semantic extraction network can be used to extract features from the road hazard description text and the road hazard classification result to obtain a first feature vector of the road hazard description text and a second feature vector of the road hazard classification result; then the cosine similarity between the first feature vector and the second feature vector is calculated, and the cosine similarity is used as the similarity between the road hazard description text and the road hazard classification result.

[0070] Specifically, it can be done through formulas The cosine similarity between the first and second eigenvectors was calculated. similarity(A,B) Where A represents the first eigenvector, B represents the second eigenvector, ||A|| represents the norm of A, and ||B|| represents the norm of B. cosine similarity(A,B) This value is between -1 and 1. The closer the value is to 1, the more similar the road hazard description text is to the road hazard classification result, while the closer the value is to -1, the less similar the road hazard description text is to the road hazard classification result.

[0071] Step 15: If the similarity does not meet the preset similarity conditions, adjust the parameters of the large language model and use the adjusted large language model as the trained large language model, then return to step 13.

[0072] Step 16: If the similarity meets the preset similarity conditions, the trained large language model is used to output the hazard solution corresponding to the road hazard description text.

[0073] In some embodiments of this application, the above-mentioned preset similarity condition can be: the similarity is greater than a preset threshold, which can be a value close to 1.

[0074] In some embodiments of this application, when processing image data of roads requiring safety hazard identification, if the similarity between the road hazard description text and the road hazard classification result meets a preset similarity condition, the road hazard description text is used as the identification result of the road safety hazard, and the corresponding hazard solution is output. If the similarity between the road hazard description text and the road hazard classification result does not meet the preset similarity condition, the parameters of the large language model are adjusted, and the adjusted large language model is used as the large language model in step 13. Step 13 is then returned to execution until the similarity between the road hazard description text and the road hazard classification result meets the preset similarity condition. At this point, the road hazard description text is used as the identification result of the road safety hazard, and the corresponding hazard solution is output.

[0075] For example, when the road hazard description text indicates that a level crossing lacks flashing lights and convex mirrors, the corresponding hazard solutions could be: install flashing lights to improve visibility; install convex mirrors to improve visibility. When the road hazard description text indicates that a road is adjacent to a cliff and lacks guardrails, slow-down signs, and yield signs, the corresponding hazard solutions could be: install guardrails to ensure road safety; install slow-down signs to remind drivers to pay attention to driving safety; install yield signs to remind drivers to yield.

[0076] The training process of a large language model is illustrated below with specific examples.

[0077] In some embodiments of this application, step 11 above, which involves training the original large language model using the trained student model and instruction fine-tuning to obtain the trained large language model, includes the following specific steps:

[0078] Step 11.1: Use road image samples and the trained student model to train the original large language model to obtain an intermediate large language model.

[0079] In some embodiments of this application, the process of obtaining the intermediate large language model is as follows:

[0080] The road image sample is input into the trained student model to obtain the road hazard classification result of the road image sample; the road image sample can be the image data of a road.

[0081] The road image samples are input into the original large language model to obtain the road hazard description text of the road image samples; the road hazard description text is used to describe the traffic safety hazards in the image data.

[0082] Calculate the similarity between the road hazard classification results and the road hazard description text of the road image samples; the calculation method for this similarity is the same as that for the similarity calculation in step 14, and the calculation process for the similarity will not be described again here.

[0083] If the similarity between the road hazard classification result and the road hazard description text of the road image sample does not meet the preset similarity condition, the parameters of the original large language model are adjusted, and the adjusted large language model is used as the original large language model. Then, the process is returned to execute the step of inputting the road image sample into the original large language model to obtain the road hazard description text of the road image sample.

[0084] If the similarity between the road hazard classification result and the road hazard description text of the road image sample meets the preset similarity condition, then the original large language model will be used as the intermediate large language model.

[0085] The aforementioned preset similarity condition can be: the similarity is greater than a preset threshold, which can be a value close to 1. During training, if the similarity between the road hazard classification result and the road hazard description text of the road image sample meets the preset similarity condition, then the original large language model at this time is used as an intermediate large language model. Otherwise, the parameters of the original large language model are adjusted, and the road hazard description text of the road image sample is obtained again using the large language model with adjusted parameters, until the similarity between the road hazard classification result and the road hazard description text of the road image sample meets the preset similarity condition, so that the output of the large language model (i.e., the road hazard description text) is more accurate and closer to the image content.

[0086] Step 11.2: Use safety knowledge data in the transportation field as instruction fine-tuning data to fine-tune the intermediate large language model, and obtain the trained large language model.

[0087] In some embodiments of this application, safety knowledge data in the field of transportation can be used as instruction fine-tuning data to fine-tune the intermediate large language model to obtain the fine-tuned large language model; then the weights of the original large language model and the fine-tuned large language model are merged to obtain the trained large language model.

[0088] To enable the large language model to evaluate and provide feedback on its output in the field of traffic and road safety, self-awareness training is necessary. This training strengthens the model's ability to analyze its own predictions and generated results, and to assess their accuracy, consistency, and rationality. The self-awareness training model involves fine-tuning through instructions, creating a dataset in the format {"instruction":"","input":"","output":""}. "instruction" poses questions related to individual cognition, "input" provides relevant background information, and "output" corresponds to answers to a series of questions, such as those from traffic and road safety hazard analysis experts.

[0089] Next, establish a dataset called "Safety Hazard Cause Analysis and Description - Solution Description". The typical input data format is: {"instruction":"","input":"","output":""}, where "instruction" corresponds to the safety hazard cause analysis and description (i.e., the road hazard description text mentioned above), "input" provides background information, and "output" corresponds to the solution description input.

[0090] To maintain efficient fine-tuning of the model, while keeping the pre-trained weights unchanged, a bypass is added next to the parameters of the original base model according to the LoRa method. This bypass trains a dimensionality reduction parameter matrix and an increased dimensionality parameter matrix, keeping the input and output dimensions of the model unchanged. When outputting, the increased / reduced dimensionality matrix is ​​superimposed on the original model parameters, resulting in better model performance on the given task.

[0091] Configure the environment, including other data processing and machine learning libraries. After deploying the model file locally, load the model and configure its path. Upload the training dataset (i.e., the aforementioned safety knowledge data in the transportation field, which can be obtained by extracting entities, relationships, and attributes from industry reports, professional papers, and technical documents using natural language processing techniques) to the project file. Next, select an appropriate number of training iterations and optimizer, and set training parameters such as save interval and learning rate settings, training rank set to 8, and random discarding of 0.1. After fine-tuning the instructions and starting training, use the loss curve to determine whether to terminate training early to prevent overfitting.

[0092] It should be noted that instruction fine-tuning is a common parameter adjustment method for large language models, and its principles will not be elaborated on here.

[0093] After fine-tuning the instructions, the fine-tuned large language model and the original large language model are weighted and merged. To reduce the covariance between models, a multivariate weight averaging method can be used. That is, the calculation weight of the parameters is obtained through the number of weights of the models, the parameter values ​​of the merged models are determined according to the weight values, and the final merged model is used as the trained large language model.

[0094] In some embodiments of this application, a user-friendly front-end interface can be designed during actual use, including areas for image uploading, text input, and result display; the trained security vulnerability identification model (mainly including the large language model and student model trained above) can be deployed to a server using tools such as TensorFlow Serving and ONNXRuntime; and the application programming interface of the model service can be configured for the front-end interface to call.

[0095] In summary, the road safety hazard identification and processing method provided in this application has the following advantages:

[0096] (1) High efficiency: The use of artificial intelligence technology enables efficient investigation of road safety hazards. Automated algorithms and processing capabilities allow for the coverage of a large road network in a short period of time, improving investigation efficiency;

[0097] (2) Comprehensiveness: Artificial intelligence technology can monitor and analyze road conditions in all aspects, including road surface conditions, bridge structure, intersections, etc. Compared with manual on-site investigation, it can obtain more comprehensive and accurate data, effectively reducing the possibility of overlooking hidden dangers;

[0098] (3) Cost-effectiveness: Compared to manual inspections, automated AI-powered inspection systems reduce labor costs and reliance on specialized personnel. In the long run, this can lead to more economical road safety management.

[0099] (4) Data analysis and decision support: By performing deep learning and data analysis on the collected data, artificial intelligence systems can generate detailed reports and statistics. This helps decision-makers better understand the road safety situation and develop targeted maintenance and improvement plans.

[0100] The relevant devices provided in this application will be described by way of example below with reference to specific embodiments.

[0101] like Figure 2 As shown, embodiments of this application provide a terminal device, such as... Figure 2 As shown, the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 2The diagram shows only one processor, a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 executes the computer program D102 to implement the steps in any of the above method embodiments.

[0102] Specifically, when the processor D100 executes the computer program D102, it trains the original large language model using a student model obtained through distillation learning and instruction fine-tuning to obtain a trained large language model. Then, it uses the trained large language model and the trained student model to process the image data of roads requiring safety hazard identification, obtaining corresponding road hazard classification results and road hazard description text. When the similarity between the road hazard classification results and the road hazard description text meets a preset similarity condition, the trained large language model outputs the hazard solution corresponding to the road hazard description text. In this process, the knowledge of the large image classification model is transferred to the large language model through knowledge distillation, optimizing the image classification task of the large language model, enhancing its performance in recognizing low-resolution and highly complex images, and improving the accuracy of road safety hazard investigation. Simultaneously, the use of automated algorithms for road safety hazard investigation significantly improves investigation efficiency compared to traditional methods.

[0103] The processor D100 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0104] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may be an external storage device of the terminal device D10, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device D10. Furthermore, the memory D101 may include both internal and external storage units of the terminal device D10. The memory D101 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory D101 can also be used to temporarily store data that has been output or will be output.

[0105] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0106] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0107] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0108] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0109] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0110] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0111] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0112] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

[0115] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for identifying and handling road safety hazards, characterized in that, include: The original large language model is trained using the trained student model and instruction fine-tuning to obtain the trained large language model; the instruction fine-tuning data is safety knowledge data in the traffic field; the trained student model is obtained by distilling the teacher model using multiple labeled road image training samples and the student model, and the teacher model is a pre-trained image classification model. The image data of the road that needs to be identified for safety hazards is input into the trained student model to obtain the road hazard classification result of the image data. The image data is input into the trained large language model to obtain the road hazard description text of the image data; Obtain the similarity between the road hazard description text and the road hazard classification result; If the similarity does not meet the preset similarity condition, the parameters of the large language model are adjusted, and the adjusted large language model is used as the trained large language model. Then, the step of inputting the image data into the trained large language model to obtain the road hazard description text of the image data is returned. If the similarity satisfies the preset similarity condition, the trained large language model will be used to output the hazard solution corresponding to the road hazard description text.

2. The method according to claim 1, characterized in that, The process of training the original large language model using the trained student model and instruction fine-tuning to obtain the trained large language model includes: The original large language model was trained using road image samples and the trained student model to obtain an intermediate large language model; Safety knowledge data in the transportation field is used as instruction fine-tuning data to fine-tune the intermediate large language model, resulting in a trained large language model.

3. The method according to claim 2, characterized in that, The process of training the original large language model using road image samples and the trained student model to obtain an intermediate large language model includes: The road image samples are input into the trained student model to obtain the road hazard classification results of the road image samples; The road image samples are input into the original large language model to obtain the road hazard description text of the road image samples; Calculate the similarity between the road hazard classification results and the road hazard description text of the road image samples; If the similarity between the road hazard classification result and the road hazard description text of the road image sample does not meet the preset similarity condition, then the parameters of the original large language model are adjusted, and the adjusted large language model is used as the original large language model. Then, the step of inputting the road image sample into the original large language model to obtain the road hazard description text of the road image sample is returned to be executed. If the similarity between the road hazard classification result and the road hazard description text of the road image sample meets the preset similarity condition, then the original large language model is used as the intermediate large language model.

4. The method according to claim 2, characterized in that, The step of using traffic safety knowledge data as instruction fine-tuning data to fine-tune the intermediate large language model to obtain the trained large language model includes: Safety knowledge data in the transportation field is used as instruction fine-tuning data to fine-tune the intermediate large language model, resulting in a fine-tuned large language model. The weights of the original large language model and the fine-tuned large language model are merged to obtain the trained large language model.

5. The method according to claim 1, characterized in that, The process of obtaining the similarity between the road hazard description text and the road hazard classification result includes: The road hazard description text and the road hazard classification result are extracted by a semantic extraction network to obtain a first feature vector of the road hazard description text and a second feature vector of the road hazard classification result. Calculate the cosine similarity between the first feature vector and the second feature vector, and use the cosine similarity as the similarity between the road hazard description text and the road hazard classification result.

6. The method according to claim 5, characterized in that, The calculation of the cosine similarity between the first feature vector and the second feature vector includes: Through formula The cosine similarity between the first feature vector and the second feature vector was calculated. similarity(A,B) ; Where A represents the first eigenvector, B represents the second eigenvector, ||A|| represents the norm of A, and ||B|| represents the norm of B.

7. The method according to claim 1, characterized in that, The preset similarity condition is: the similarity is greater than a preset threshold.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the road safety hazard identification and processing method as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the road safety hazard identification and processing method as described in any one of claims 1 to 7.

Images