Mountain-oriented new energy heavy truck multi-modal information intelligent analysis system

Abstract

The invention discloses a mountainous region-oriented new energy heavy truck multi-modal information intelligent analysis system, and relates to the technical field of data analysis, and the mountainous region-oriented new energy heavy truck multi-modal information intelligent analysis system comprises the following steps: collecting real-time multi-modal monitoring data through a multi-type sensor array, and generating a real-time driving state description text by using data standardization and text integration; semantic matching is carried out in the heavy truck mountain running strategy database, and K running strategies and an execution feedback information set thereof are obtained; and calculating strategy execution value scoring through two-dimensional analysis, performing centralized screening on a score set, and selecting a driving strategy corresponding to the maximum value in the score set as a multi-modal information analysis result. The technical problem that multi-modal data in a complex environment are difficult to process in real time and the optimal driving strategy is difficult to generate efficiently in the prior art is solved, and the technical effect of improving the driving safety and decision reliability of the heavy truck in the complex mountain environment is achieved.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and in particular to a multi-modal information intelligent analysis system for new energy heavy trucks for mountainous areas. Background Art

[0002] With the widespread use of new energy heavy trucks in complex mountainous environments, they need to deal with a variety of complex scenarios such as steep slopes, sharp turns, unstable road conditions, etc. during driving. Traditional single sensors or fixed rule systems have significant limitations in real-time processing of multimodal data and generating optimized driving strategies. In the existing technology, multimodal data (such as terrain, environment, vehicle status, etc.) are processed by independent modules respectively, and there is a lack of efficient coordination mechanism between modules, resulting in low data integration efficiency. Traditional methods mainly rely on fixed rules or simple mathematical models to analyze sensor data, which shows slow response and lack of flexibility in the face of complex environments with real-time dynamic changes. For example, for the combination of different slopes and curves, traditional models often find it difficult to generate targeted strategies and cannot fully optimize energy consumption, safety and efficiency.

[0003] Existing technologies make it difficult to achieve efficient semantic processing of multimodal data collected in real time. The correlation and contextual meaning between data cannot be effectively captured, resulting in the derivation of driving strategies relying on coarse-grained information and lack of intelligent analysis capabilities. For example, in the face of a combination of rainy and snowy weather and sharp turns, it is impossible to quickly generate a safety strategy that matches the current state, increasing driving risks. Summary of the invention

[0004] The present application provides a multimodal information intelligent analysis system for new energy heavy-duty trucks in mountainous areas, which is used to solve the technical problem that the existing technology is difficult to process multimodal data in complex environments in real time and efficiently generate optimal driving strategies.

[0005] In view of the above problems, the present application provides a multi-modal information intelligent analysis system for new energy heavy trucks for mountainous areas.

[0006] The present application provides a multi-modal information intelligent analysis system for new energy heavy trucks in mountainous areas, the system comprising: A data acquisition module, which is used to interactively deploy a multi-type sensor array of a target new energy heavy truck traveling in the mountains, collect the collected data of the multi-type sensor array at the current moment, and obtain a real-time multimodal monitoring data set; a text integration module, which is used to use the Pandas tool to perform standardized structure conversion on the real-time multimodal monitoring data set to obtain a real-time standardized multimodal monitoring data set, and perform text integration on the real-time standardized multimodal monitoring data according to a preset rule template based on a pre-trained large language model to obtain a real-time driving status description text; a semantic matching module, which uses the real-time driving status description text as a matching object, performs semantic matching in the heavy truck mountain driving strategy database, obtains K driving strategies and K strategy execution feedback information sets, wherein K is an integer greater than or equal to 1; a two-dimensional analysis module, the two-dimensional analysis module is used to traverse the K strategy execution feedback information sets to perform two-dimensional analysis of strategy execution timeliness and next state evaluation results, and obtain K strategy execution value score sets of the K driving strategies; a score concentration screening module, the score concentration screening module is used to perform score concentration screening on the K strategy execution value score sets, determine the K strategy execution value score concentration values, and select the driving strategy corresponding to the maximum value of the K strategy execution value score concentration values ​​as the multimodal information analysis result.

[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages: The present application interactively deploys a multi-type sensor array on a target new energy heavy truck traveling in a mountainous area, collects the collected data of the multi-type sensor array at the current moment, and obtains a real-time multimodal monitoring data set; uses the Pandas tool to perform standardized structure conversion on the real-time multimodal monitoring data set to obtain a real-time standardized multimodal monitoring data set, and performs text integration on the real-time standardized multimodal monitoring data according to a preset rule template based on a pre-trained large language model to obtain a real-time driving state description text; uses the real-time driving state description text as a matching object, performs semantic matching in a heavy truck mountain driving strategy database, and obtains K driving strategies and K strategy execution feedback information sets, where K is an integer greater than or equal to 1; traverses the K strategy execution feedback information sets to perform a two-dimensional analysis of the strategy execution timeliness and the next state evaluation results, and obtains K strategy execution value score sets of the K driving strategies; performs score concentration screening on the K strategy execution value score sets, determines the K strategy execution value score concentration values, and selects the driving strategy corresponding to the maximum value in the K strategy execution value score concentration values ​​as the multimodal information analysis result. The present invention solves the technical problem that the prior art is difficult to process multimodal data in a complex environment in real time and efficiently generate the optimal driving strategy. Real-time multimodal data is collected through a multi-type sensor array, and the Pandas tool is used to perform standardized structure conversion and text integration of a pre-trained large language model to generate a real-time driving status description text, and semantic matching is performed in a heavy-duty truck mountain driving strategy database. The strategy execution value score is calculated in combination with two-dimensional analysis, and the optimal strategy is selected through centralized screening through the score, so as to realize efficient processing and intelligent analysis of real-time multimodal data, and achieve the technical effect of improving the driving safety and decision-making reliability of heavy-duty trucks in complex mountain environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0009] Figure 1 A schematic diagram of the structure of a multi-modal information intelligent analysis system for new energy heavy trucks for mountainous areas provided in an embodiment of the present application; Figure 2 A schematic flow chart of the execution steps of a two-dimensional analysis module in a multi-modal information intelligent analysis system for new energy heavy trucks for mountainous areas provided in an embodiment of the present application.

[0010] Explanation of the reference numerals: data collection module 11, text integration module 12, semantic matching module 13, two-dimensional analysis module 14, scoring centralized screening module 15. DETAILED DESCRIPTION

[0011] The present application provides a multimodal information intelligent analysis system for new energy heavy-duty trucks in mountainous areas, aiming to solve the technical problem that the existing technology is difficult to process multimodal data in complex environments in real time and efficiently generate optimal driving strategies. Real-time multimodal data is collected through a multi-type sensor array, and the Pandas tool is used to perform standardized structure conversion and text integration of pre-trained large language models to generate real-time driving status description text, perform semantic matching in the heavy-duty truck mountain driving strategy database, combine two-dimensional analysis to calculate the strategy execution value score, and select the optimal strategy through centralized screening through scoring, so as to realize efficient processing and intelligent analysis of real-time multimodal data, and achieve the technical effect of improving the driving safety and decision-making reliability of heavy-duty trucks in complex mountainous environments.

[0012] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0013] It should be noted that any variations of the terms "include" and "have" are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules that are not explicitly listed or inherent to these processes, methods, products or devices.

[0014] Examples, such as Figure 1 As shown, the embodiment of the present application provides a multi-modal information intelligent analysis system for new energy heavy trucks in mountainous areas, the system comprising: The data acquisition module 11 is used to interactively deploy a multi-type sensor array on a target new energy heavy truck traveling in the mountains, collect data from the multi-type sensor array at the current moment, and obtain a real-time multi-modal monitoring data set.

[0015] In the embodiment of the present application, the data acquisition module 11 realizes comprehensive collection of real-time data during the operation of the vehicle through a multi-type sensor array interactively arranged on the target new energy heavy truck. The data acquisition module 11 is used to obtain multi-modal monitoring data of the current state and environment of the vehicle in real time, and integrate it into a real-time multi-modal monitoring data set.

[0016] Specifically, the multi-type sensor array deployed on the target new energy heavy truck includes terrain sensors, environmental sensors and vehicle status sensors. The terrain sensor is responsible for collecting the slope, curve radius and road type of the current road surface, such as gravel road or asphalt road; the environmental sensor is used to monitor weather conditions (such as sunny days, rain and snow), temperature and humidity, and lighting conditions in real time; the vehicle status sensor monitors the core parameters of the new energy heavy truck, including battery power, brake temperature, load weight and current speed. At every moment of the vehicle's driving, all sensors operate simultaneously, and the monitored terrain features, environmental features and vehicle status data are output in real time, such as "the slope is 15°, the curve radius is 50 meters, the road surface is gravel, the weather is sunny, the battery power is 70%, the brake temperature is 80℃, and the current speed is 30km / h". These real-time data are seamlessly transmitted through the interaction between the sensor and the data acquisition module 11, and finally form a real-time multimodal monitoring data set.

[0017] The text integration module 12 is used to use the Pandas tool to perform standardized structure conversion on the real-time multimodal monitoring data set to obtain a real-time standardized multimodal monitoring data set, and perform text integration on the real-time standardized multimodal monitoring data according to a preset rule template based on a pre-trained large language model to obtain a real-time driving status description text.

[0018] In the embodiment of the present application, the text integration module 12 first uses the Pandas tool to perform standardized structural conversion on the real-time multimodal monitoring data output by the data acquisition module 11. Through the powerful data processing capabilities of Pandas, the original data is loaded into the DataFrame format, and the data types are unified, such as converting some string values ​​into numeric types to ensure the consistency of the data format. At the same time, clear labels are added to various types of data for subsequent model input and processing, thereby forming a structured standardized multimodal data set.

[0019] Next, we introduce a pre-trained large language model to achieve the mapping from data to natural language. This large language model is based on the Transformer architecture and uses general language datasets (such as Wikipedia, OpenWebText) and vehicle-specific corpora. It is trained through a transfer learning algorithm, enabling the model to understand the semantics of multimodal data and generate natural language.

[0020] Then, the standardized data is mapped into a specific natural language framework by combining preset rule templates (such as terrain feature templates, environmental feature templates, and vehicle status templates). For example, the terrain feature template "The current slope is (slope), the road type is (road type), and the curve radius is (curve radius)." When receiving specific data, it is dynamically filled in as "The current slope is 15°, the road type is gravel road, and the curve radius is 50 meters." The use of this rule template ensures that the generated text has a clear structure and accurate semantics.

[0021] Finally, the text integration module 12 performs text integration based on conditional text generation and context semantic understanding to generate a complete real-time driving status description text. For example, when the input multimodal monitoring data includes a slope of 15°, a curve radius of 50 meters, clear weather, a temperature of 28°C, a battery charge of 70%, and a brake temperature of 120°C, the output text is "The current vehicle is in a steep slope area with a slope of 15°, a gravel road ahead, and a curve radius of 50 meters. The weather is clear, the temperature is 28°C, the battery charge is 70%, the brake temperature is 120°C, the current load is 20 tons, and the speed is 50km / h." The semantic matching module 13 uses the real-time driving status description text as a matching object, performs semantic matching in the heavy truck mountain driving strategy database, and obtains K driving strategies and K strategy execution feedback information sets, where K is an integer greater than or equal to 1.

[0022] In the embodiment of the present application, the semantic matching module 13 first uses the named entity recognition method to parse the real-time driving status description text, extract key entities and attributes, such as "slope", "gravel road", "brake temperature", "load weight", etc. These entities are annotated as terrain features, environmental features and vehicle status features, and a structured feature set is generated through semantic parsing.

[0023] Next, a pre-trained word embedding model (such as Word2Vec or BERT) is used to vectorize the parsed structured feature set. The word embedding model learns the contextual relationship in the semantic space through training, and maps each key entity in the text into a high-dimensional semantic vector. For example, "the slope is 15°" is represented as a specific vector , and the vector for “the road surface is gravel” is , the vector for "load is 20 tons" is Then, these entity vectors are combined into an overall semantic vector, that is, the three vectors are averaged by dimension to obtain the semantic vector of the state: The semantic features representing the current driving state are used as input for semantic matching.

[0024] Then, the semantic vector of the real-time driving status description text is matched with the semantic vector of each strategy in the strategy database through the cosine similarity method. Each strategy in the heavy truck mountain driving strategy database has been converted into a semantic vector through the same word embedding model in the preprocessing stage and stored as an efficient index structure. By calculating the cosine similarity between the real-time text semantic vector and the strategy semantic vector in the database, the semantic similarity between them is measured, and all strategies are sorted according to the similarity score to select the K driving strategies with the highest scores. For example, for the overall semantic vector A of the current driving state = , the corresponding feature vector B in the strategy is , then the dot product is 0.3×0.25+0.2667×0.30+0.3×0.35=0.26, and the modulus of vector A is , vector B modulus = , then similarity = . Similarly, similarity calculations are performed on all strategies to obtain K driving strategies.

[0025] After determining K driving strategies, further extract the corresponding strategy execution feedback information set for each driving strategy. Through the SQL query method, retrieve the execution effect data of each strategy in similar historical scenarios from the heavy truck mountain driving strategy database. For example, a driving strategy "reduce vehicle speed" may correspond to multiple strategy execution feedback information, such as "brake temperature reduced by 20% in a 15° slope scenario", "energy consumption increased by 5% in a gravel road scenario", and "brake wear reduced by 10% in a 20-ton load scenario". These feedback information are aggregated into the strategy execution feedback information set of the strategy.

[0026] Through the above process, an output result including K driving strategies and their corresponding K strategy execution feedback information sets is generated, wherein each driving strategy corresponds to a strategy execution feedback information set, and K is an integer greater than or equal to 1.

[0027] The two-dimensional analysis module 14 is used to traverse the K strategy execution feedback information sets to perform two-dimensional analysis of strategy execution timeliness and next state evaluation results, and obtain K strategy execution value score sets of the K driving strategies.

[0028] In the embodiment of the present application, the two-dimensional analysis module 14 is used to traverse K policy execution feedback information sets to perform two-dimensional analysis of policy execution timeliness and next state evaluation results.

[0029] Specifically, firstly, the state transition time is used as an index to extract the state transition time set from each strategy execution feedback information set, and the ratio of each state transition time to the set sum is calculated, and the reciprocal is taken as the strategy execution time factor to measure the efficiency of the strategy in achieving the goal. Then, the next state description text set is extracted from the feedback information set with the next state description text as an index, and these texts are analyzed using the pre-trained state evaluator. The state evaluator combines the preset state evaluation indicators (such as battery power usage rate, load unit energy consumption, noise and road condition matching, etc.) to generate the next state evaluation result corresponding to each text, and quantify the running effect after the strategy is executed. Finally, the strategy execution time factor and the next state evaluation result are weighted and comprehensively analyzed through the value scoring function to calculate the comprehensive value score of each strategy. Through this process, the two-dimensional analysis of the strategy execution time and the next state evaluation result is completed, and the K strategy execution value score sets of K driving strategies are output.

[0030] Further, such as Figure 2 As shown, in the system provided by the embodiment of the application, the two-dimensional analysis module 14 is also used for: Using the state transition time as an index, the K strategy execution feedback information sets are retrieved to obtain K state transition time sets; any state transition time in the K state transition time sets is divided by the sum of the corresponding state transition time sets, and the inverse of the calculated result is used as the K strategy execution time factor sets; using the next state description text as an index, the K strategy execution feedback information sets are retrieved to obtain K next state description text sets; using the state evaluator to analyze the K next state description text sets to obtain K next state evaluation result sets; using the value scoring function to perform a two-dimensional analysis on the K strategy execution time factor sets and the K next state evaluation result sets to obtain the K strategy execution value scoring sets.

[0031] In an embodiment of the present application, first, the state transition time is used as an index to retrieve K sets of strategy execution feedback information, extract the state transition time data of each strategy in different scenarios, and generate K sets of state transition time. State transition time refers to the time required from the start of strategy execution to the achievement of the target state, such as the length of time from the start of brake temperature drop to the time when it reaches the safe range, reflecting the execution efficiency of the strategy. Through the database retrieval method, K sets of strategy execution feedback information are retrieved, the state transition time data of each strategy in different scenarios are extracted, and the corresponding K sets of state transition time are generated.

[0032] Then, the ratio of any state transition time in each state transition time set to the set sum is calculated, and the inverse of the calculation result is taken to generate K policy execution time factor sets. Specifically, the formula for the ratio of a single state transition time to the set sum is ratio = single state transition time / sum of state transition time sets, and the inverse of the ratio is used as the policy execution time factor. Through this process, K policy execution time factor sets corresponding to K state transition time sets are obtained.

[0033] Next, we use the next state description text as an index to retrieve the K strategy execution feedback information sets, extract the next state description text corresponding to each strategy, and form K next state description text sets. The next state description text is a semantic description of the new operating state that may be achieved after the strategy is executed, such as "brake temperature decreases by 20%" or "energy consumption increases by 5%". These texts express the direct effect after the strategy is executed. By using the semantic index retrieval method to extract these texts from the feedback information, a data basis is provided for subsequent state evaluation, and K next state description text sets are obtained.

[0034] Next, the pre-trained state evaluator is used to analyze the K next-state description text sets, and the K next-state description text sets are input into the state evaluator respectively to obtain K next-state evaluation result sets.

[0035] Finally, the value scoring function is used to perform a two-dimensional comprehensive analysis on the K strategy execution time factor sets and the K next state evaluation result sets, and the comprehensive value score of each strategy is calculated to obtain the K strategy execution value score sets.

[0036] Furthermore, in the system provided in the application embodiment, the value scoring function is: ; in, is the strategy execution value score output by the value scoring function at the current moment according to the driving strategy, The strategy execution time factor for a strategy execution feedback information, The next state evaluation result of a policy execution feedback information, The weight of the balance strategy execution time factor and the next state evaluation result.

[0037] In the embodiment of the present application, the value scoring function is: ;in, is the strategy execution value score output by the value scoring function at the current moment according to the driving strategy, The strategy execution time factor for a strategy execution feedback information, The next state evaluation result of a policy execution feedback information, The weight is pre-set by technical experts to balance the strategy execution time factor and the next state evaluation result.

[0038] Based on this function, a two-dimensional analysis is performed on the K strategy execution time factor sets and the K next state evaluation result sets, and finally the K strategy execution value score sets are obtained. Specifically, the obtained time factor set and next state evaluation result set are substituted into the value scoring function, and for each strategy, the weighted sum of the time factor and the next state evaluation result in the feedback information is calculated one by one, and finally a comprehensive value score is generated for each strategy. By calculating the K strategies one by one, the K strategy execution value score sets are finally summarized.

[0039] Furthermore, in the system provided in the embodiment of the application, the two-dimensional analysis module 14 is also used for: Acquire multiple sample next-state description texts; evaluate the multiple sample next-state description texts using an expert survey method combined with a preset state evaluation indicator set to obtain multiple sample next-state evaluation results; use the multiple sample next-state description texts and the multiple sample next-state evaluation results to perform supervised training on a framework built based on a convolutional neural network, learn a one-to-one mapping relationship between the next-state description texts and the next-state evaluation results during training, until the training converges, and obtain a trained state evaluator.

[0040] Furthermore, in the system provided in the application embodiment, the preset state evaluation indicator set includes battery power utilization rate, load unit energy consumption, noise and road condition matching degree.

[0041] In an embodiment of the present application, a plurality of sample next-state description texts are first obtained, and these texts are derived from a set of strategy execution feedback information generated by new energy heavy trucks in actual operation. The feedback information set records the state changes of the vehicle after the strategy is executed under different operating conditions. The feedback information is parsed by semantic indexing technology to extract text content describing the vehicle state changes. These texts are semantic descriptions of the new operating state that the vehicle may enter after the strategy is executed, such as "brake temperature decreases by 20%" or "load unit energy consumption increases by 5%". The semantic indexing method combines keyword matching and context parsing technology to automatically locate, filter and extract text fragments from the feedback information to ensure that it accurately reflects the state changes after the strategy is executed. Through this process, a plurality of sample next-state description texts are obtained.

[0042] Next, the expert survey method is combined with the preset state evaluation index set to annotate multiple sample next state description texts and generate multiple sample next state evaluation results. Specifically, by inviting domain experts to conduct a comprehensive analysis of each description text according to the preset state evaluation index, and quantitatively score its effect. The preset state evaluation index set includes four indicators: battery power usage rate, load unit energy consumption, noise and road condition matching degree. These indicators are given the same weight to ensure the fairness of the evaluation process. The expert's analysis of the text covers multiple dimensions, such as the impact of the strategy on battery energy consumption in the evaluation description text, the change of energy consumption under different load conditions, the change of noise level during operation, and the degree of adaptation of the strategy to actual road and environmental conditions. Experts score each text based on these dimensions (such as a score between 0 and 1), and generate a comprehensive evaluation result by averaging the scores of the four indicators. For example, for the text describing "load unit energy consumption increases by 5%", the expert may mark the evaluation result as 0.75, indicating that the impact of the strategy on energy consumption is more significant. Through this evaluation process, multiple sample next state evaluation results are obtained.

[0043] Then, the framework built based on the convolutional neural network is supervised and trained using multiple sample next-state description texts and multiple sample next-state evaluation results. During the training process, multiple sample next-state description texts and multiple sample next-state evaluation results are divided into training sets and validation sets. The validation set randomly divides a part of samples from multiple sample next-state description texts and their evaluation results as validation set data, which is used to evaluate the performance of the model after each round of training. The training process includes converting the description text into a vector through word embedding technology, extracting semantic features using convolutional neural networks, and comparing it with the real evaluation results after mapping through the fully connected layer. The mean square error (MSE) is used as the loss function, and the model parameters are optimized through back propagation. During the training process, the change of the loss value of the validation set is monitored in real time. When the loss value changes less than the set threshold (such as 0.001) in several consecutive rounds, the model training is considered to have converged. Finally, the model parameters when the loss of the validation set is the lowest are saved to obtain the trained state evaluator, which accurately learns the one-to-one mapping relationship between the next-state description text and the evaluation results.

[0044] The scoring centralized screening module 15 is used to perform scoring centralized screening on the K strategy execution value scoring sets, determine the K strategy execution value scoring centralized values, and select the driving strategy corresponding to the maximum value among the K strategy execution value scoring centralized values ​​as the multimodal information analysis result.

[0045] In the embodiment of the present application, the scoring set screening module 15 first extracts the median value of each set from the K policy execution value scoring sets to obtain the median values ​​of the K policy execution value scores. The median value is calculated by selecting the middle value after sorting the scoring data.

[0046] Next, according to the preset centralized screening step length, a neighborhood is constructed for each median value, called the median neighborhood, with a range of [median value - step length, median value + step length]. The preset centralized screening step length is pre-set by technical experts. On this basis, the mean of the score in each median neighborhood is calculated, and the difference between the mean and the median is calculated to obtain K differences. The difference reflects the degree of deviation between the neighborhood mean and the median, and is an important basis for judging the concentration of the score set.

[0047] If the difference is less than or equal to the preset difference threshold, it is considered that the concentration of the score set has met the requirements. At this time, the concentration value calculation is further improved by neighborhood expansion. The neighborhood range is expanded along the neighborhood edge according to the preset step size to form the target neighborhood, and then the mean of all scores in the target neighborhood is calculated as the final strategy execution value score concentration value.

[0048] If the difference is greater than the preset difference threshold, the median score is iteratively updated, and a phased iterative neighborhood is constructed based on the updated score data. The density of each iterative neighborhood is calculated and compared with the median neighborhood density. If the iterative neighborhood density is greater than or equal to the median neighborhood density, the median score is updated according to the step size and the neighborhood range is expanded until the preset number of iterations is reached or the difference meets the threshold condition, and finally the target iterative strategy execution value score is obtained as the centralized value of the score set.

[0049] Finally, the concentrated values ​​of the K strategy execution value scores are summarized, and the driving strategy corresponding to the maximum value is selected as the result of multimodal information analysis.

[0050] Furthermore, in the system provided in the application embodiment, the scoring centralized screening module 15 is also used for: Extract median values ​​from the K policy execution value score sets respectively to obtain K policy execution value score medians; construct neighborhoods of the K policy execution value score medians according to a preset centralized screening step length to obtain K median neighborhoods; calculate the mean of the K median neighborhoods, and perform difference calculations on the calculation results and the K policy execution value score medians respectively to obtain K differences; when the K differences are less than or equal to a preset difference threshold, perform edge diffusion on the K median neighborhoods according to a preset centralized screening step length to obtain K target neighborhoods; calculate the mean of the K target neighborhoods to obtain the K policy execution value score centralized values.

[0051] In the embodiment of the present application, the median value is first extracted from each strategy execution value score set to obtain K strategy execution value score median values. The median value is a statistical indicator that can effectively reflect the concentration trend of the score set, and is calculated by sorting the score data in the set and selecting the middle value.

[0052] Next, according to the preset centralized screening step size, an initial neighborhood is constructed for each median value, called the median neighborhood. The range of the neighborhood is defined as [median-step size, median+step size]. It is used to capture the score data near the median value and reflect the local concentration of the score set. For example, if the median value is 0.75 and the step size is 0.1, the neighborhood range is [0.65, 0.85].

[0053] Then, the mean of the scores in the neighborhood of each median is calculated, that is, the average of all the score data in the neighborhood. By comparing the mean with the corresponding median, the difference between the two is calculated to obtain K differences.

[0054] When the difference is less than or equal to the preset difference threshold, it is considered that the central tendency of the score set has met the requirements, and the median neighborhood is further diffused. The diffusion operation expands the neighborhood range according to the preset step size. For example, the upper and lower limits of the neighborhood are expanded by 0.1 respectively to obtain the target neighborhood. In the target neighborhood, the mean of the score is recalculated as the final strategy execution value score concentration value. Edge diffusion can more comprehensively capture the distribution characteristics of the score data and ensure the accuracy of the concentration value. Among them, the preset difference threshold and the preset difference threshold are pre-set by technical experts.

[0055] Through the above steps, the centralized value calculation of each strategy execution value score set is completed, and K strategy execution value score centralized values ​​are generated.

[0056] Furthermore, in the system provided in the application embodiment, the scoring centralized screening module 15 is also used for: The K median neighborhoods are edge diffused according to a preset centralized screening step length to obtain K diffused neighborhoods; it is determined whether the neighborhood density of the K diffused neighborhoods is greater than or equal to the neighborhood density of the K median neighborhoods, if not, the diffusion is stopped, and the K median neighborhoods are used as K target neighborhoods; if so, the K diffused neighborhoods are edge diffused according to a preset centralized screening step length until a preset number of diffusions is met, and the neighborhood obtained by the last diffusion is used as the K target neighborhoods.

[0057] In an embodiment of the present application, the upper and lower limits of each median neighborhood are first expanded according to the preset centralized screening step size to form K diffuse neighborhoods. The expansion operation is implemented by linear interval calculation. For example, for the median neighborhood [0.7, 0.8], when the preset centralized screening step size is 0.1, the expanded neighborhood range is [0.6, 0.9]. The upper and lower limits are adjusted one by one using an iterative algorithm, and the scoring data of each neighborhood is re-divided to ensure that all data in the scoring set can find the corresponding distribution position in the expanded neighborhood. After this step is completed, K diffuse neighborhoods are obtained.

[0058] Next, calculate the neighborhood density of the diffusion neighborhood and compare it with the density of the corresponding median neighborhood. The neighborhood density is calculated by dividing the number of scores in the neighborhood by the length of the neighborhood range. The density value is obtained by counting the number of scores contained in the diffusion neighborhood and calculating the length of the neighborhood range in combination with the upper and lower limits of the neighborhood. For example, if the diffusion neighborhood range is [0.6, 0.9] and contains 10 score data, the neighborhood density is 10 / 0.3=33.33. By comparing the density of the diffusion neighborhood and the density of the median neighborhood, it is determined whether the expansion is effective. If the density of the diffusion neighborhood is less than that of the median neighborhood, it means that the median neighborhood has fully captured the concentrated distribution of the score data, so the diffusion stops and the median neighborhood is directly used as the target neighborhood.

[0059] If the density of the diffusion neighborhood is greater than or equal to the neighborhood density of the median neighborhood, it is considered that the current diffusion range is insufficient to fully reflect the distribution characteristics of the score set, so the diffusion neighborhood is further expanded according to the preset step size. The diffusion operation is repeated, and the neighborhood density after diffusion is continuously compared with the neighborhood density before diffusion. Specifically, if the neighborhood density after diffusion is less than the density of the neighborhood before diffusion, the current diffusion neighborhood is determined as the target neighborhood; if the neighborhood density after diffusion is greater than or equal to the density of the neighborhood before diffusion, the diffusion continues until the preset diffusion limit is reached (for example, 3 times), and the diffusion neighborhood obtained by the last diffusion is determined as the target neighborhood.

[0060] Through the above process, the expansion operation is finally completed and K target neighborhoods are obtained.

[0061] Furthermore, the system provided in the application embodiment is also used for: When the K difference values ​​are greater than a preset difference threshold, the median values ​​of the K strategy execution value scores are iteratively updated according to the preset centralized screening step length to obtain K stage iterative strategy execution value scores; according to the preset centralized screening step length, K stage iterative neighborhoods of the K stage iterative strategy execution value scores are constructed; it is determined whether the neighborhood density of the K stage iterative neighborhoods is greater than or equal to the neighborhood density of the K median neighborhoods; if so, the K stage iterative strategy execution value scores are iteratively updated according to the preset centralized screening step length until the preset number of iterations is met, to obtain K target iterative strategy execution value scores, and the K target iterative strategy execution value scores are used as the centralized values ​​of the K strategy execution value scores.

[0062] In an embodiment of the present application, when K difference values ​​are greater than a preset difference threshold, the score concentration screening module 15 starts an iterative update process, gradually optimizes the score concentration value by dynamically adjusting the score range and neighborhood distribution, and finally obtains K target iterative strategy execution value scores and uses them as K strategy execution value score concentration values.

[0063] Specifically, first, the median values ​​of K strategy execution value scores are randomly adjusted according to the preset centralized screening step size to generate K stage iterative strategy execution value scores. This step is achieved by randomly selecting a new score value for the median value within the preset step size range. For example, if the current score median is 0.75 and the preset step size is 0.1, the adjusted score value may be randomly distributed in the range of [0.75−0.1, 0.75+0.1], such as adjusted to 0.78. This random adjustment operation can introduce dynamic changes in the score data and avoid missing the global distribution characteristics of the score data due to an overly fixed adjustment method. Through this process, K stage iterative strategy execution value scores are obtained.

[0064] Next, the value scoring is performed according to the updated stage iteration strategy to construct the corresponding stage iteration neighborhood. The neighborhood range is determined by the current score value and the centralized screening step size. For example, for a score value of 0.78 after a certain iteration update, when the step size is 0.1, the constructed neighborhood range is [0.68, 0.88]. By counting the score data within the extended range, the upper and lower limits of the neighborhood are defined to ensure that all score data are reasonably distributed within the neighborhood range. Through this operation, K stage iteration neighborhoods are generated.

[0065] Then, the neighborhood density of each stage iteration neighborhood is calculated by the same method as the above calculation. Next, the neighborhood density of the stage iteration neighborhood is compared with the neighborhood density of the corresponding median neighborhood. If the density of the stage iteration neighborhood is greater than or equal to the neighborhood density of the median neighborhood, it is considered that the randomly adjusted score value is effective and can further improve the central tendency of the score data. Therefore, the stage iteration strategy execution value score is continuously randomly adjusted according to the preset step size for the next round. If the stage iteration neighborhood density is less than the median neighborhood density, the iterative update is stopped, and the current neighborhood is determined as the target iteration neighborhood.

[0066] This process is repeated, and the neighborhood density is recalculated after each iteration to determine whether the neighborhood density after expansion is less than the neighborhood density before expansion; the number of iterations reaches a preset limit (for example, a maximum of 3 iterations). If one of the above conditions is met, the update is stopped and the neighborhood obtained in the last iteration is determined as the target iteration neighborhood.

[0067] Finally, through this process, K target iterative strategy execution value scores are obtained for each strategy execution value score set, and they are used as the centralized values ​​of the K strategy execution value scores.

[0068] In the embodiments of the present application, in summary, the embodiments of the present application have at least the following technical effects: The present application interactively deploys a multi-type sensor array on a target new energy heavy truck traveling in a mountainous area, collects the collected data of the multi-type sensor array at the current moment, and obtains a real-time multimodal monitoring data set; uses the Pandas tool to perform standardized structure conversion on the real-time multimodal monitoring data set to obtain a real-time standardized multimodal monitoring data set, and performs text integration on the real-time standardized multimodal monitoring data according to a preset rule template based on a pre-trained large language model to obtain a real-time driving state description text; uses the real-time driving state description text as a matching object, performs semantic matching in a heavy truck mountain driving strategy database, and obtains K driving strategies and K strategy execution feedback information sets, where K is an integer greater than or equal to 1; traverses the K strategy execution feedback information sets to perform a two-dimensional analysis of the strategy execution timeliness and the next state evaluation results, and obtains K strategy execution value score sets of the K driving strategies; performs score concentration screening on the K strategy execution value score sets, determines the K strategy execution value score concentration values, and selects the driving strategy corresponding to the maximum value in the K strategy execution value score concentration values ​​as the multimodal information analysis result. The present invention solves the technical problem that the prior art is difficult to process multimodal data in a complex environment in real time and efficiently generate the optimal driving strategy. Real-time multimodal data is collected through a multi-type sensor array, and the Pandas tool is used to perform standardized structure conversion and text integration of a pre-trained large language model to generate a real-time driving status description text, and semantic matching is performed in a heavy-duty truck mountain driving strategy database. The strategy execution value score is calculated in combination with two-dimensional analysis, and the optimal strategy is selected through centralized screening through the score, so as to realize efficient processing and intelligent analysis of real-time multimodal data, and achieve the technical effect of improving the driving safety and decision-making reliability of heavy-duty trucks in complex mountain environments.

[0069] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. The processes depicted in the accompanying drawings do not necessarily require the specific order and continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0070] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0071] This specification and drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. The multi-modal information intelligent analysis system for new energy heavy trucks in mountainous areas is characterized by: The system comprises: A data acquisition module, wherein the data acquisition module is used to interactively deploy a multi-type sensor array on a target new energy heavy truck traveling in a mountainous area, collect data collected by the multi-type sensor array at a current moment, and obtain a real-time multi-modal monitoring data set; A text integration module, wherein the text integration module is used to perform standardized structure conversion on the real-time multimodal monitoring data set using a Pandas tool to obtain a real-time standardized multimodal monitoring data set, and to perform text integration on the real-time standardized multimodal monitoring data according to a preset rule template based on a pre-trained large language model to obtain a real-time driving status description text; A semantic matching module, wherein the semantic matching module uses the real-time driving status description text as a matching object, performs semantic matching in a heavy truck mountain driving strategy database, and obtains K driving strategies and K strategy execution feedback information sets, wherein K is an integer greater than or equal to 1; A two-dimensional analysis module, the two-dimensional analysis module is used to traverse the K strategy execution feedback information sets to perform a two-dimensional analysis of the strategy execution timeliness and the next state evaluation results, and obtain K strategy execution value score sets of the K driving strategies; The scoring concentration screening module is used to perform scoring concentration screening on the K strategy execution value score sets, determine the K strategy execution value score concentration values, and select the driving strategy corresponding to the maximum value among the K strategy execution value score concentration values ​​as the multimodal information analysis result.

2. The multi-modal information intelligent analysis system for new energy heavy trucks for mountainous areas as claimed in claim 1 is characterized in that: Two-dimensional analysis module for: Using the state transition time as an index, searching the K strategy execution feedback information sets to obtain K state transition time sets; Divide any state transition time of the K state transition time sets by the sum of the corresponding state transition time sets, and use the inverse of the calculation result as the K policy execution time factor sets; Using the next state description text as an index, searching the K strategy execution feedback information sets to obtain K next state description text sets; Analyzing the K next-state description text sets using a state evaluator to obtain K next-state evaluation result sets; A value scoring function is used to perform a two-dimensional analysis on the K strategy execution time factor sets and the K next state evaluation result sets to obtain the K strategy execution value score sets.

3. The multi-modal information intelligent analysis system for new energy heavy trucks for mountainous areas as claimed in claim 2 is characterized in that: The value scoring function is: ； in, is the strategy execution value score output by the value scoring function at the current moment according to the driving strategy, The strategy execution time factor for a strategy execution feedback information, The next state evaluation result of a policy execution feedback information, The weight of the balance strategy execution time factor and the next state evaluation result.

4. The multi-modal information intelligent analysis system for new energy heavy trucks for mountainous areas as claimed in claim 2 is characterized in that: Two-dimensional analysis module for: Obtain multiple sample next state description texts; Using an expert survey method combined with a preset state evaluation index set, the multiple sample next state description texts are evaluated to obtain multiple sample next state evaluation results; The framework built based on the convolutional neural network is supervisedly trained using the multiple sample next-state description texts and the multiple sample next-state evaluation results. During the training, a one-to-one mapping relationship between the next-state description texts and the next-state evaluation results is learned until the training converges to obtain a trained state evaluator.

5. The multi-modal information intelligent analysis system for new energy heavy trucks for mountainous areas as claimed in claim 4 is characterized in that: The preset status evaluation index set includes battery power usage rate, load unit energy consumption, noise and road condition matching degree.

6. The multi-modal information intelligent analysis system for new energy heavy trucks for mountainous areas as claimed in claim 1 is characterized in that Scoring centralized screening module for: Extracting median values ​​from the K strategy execution value score sets respectively to obtain median values ​​of the K strategy execution value scores; According to the preset centralized screening step length, construct the neighborhood of the median values ​​of the K strategy execution value scores to obtain K median value neighborhoods; Calculate the mean of the K median neighborhoods, and perform difference calculations on the calculated results and the median values ​​of the K strategy execution value scores to obtain K differences; When the K differences are less than or equal to a preset difference threshold, edge diffusion is performed on the K median neighborhoods according to a preset centralized screening step size to obtain K target neighborhoods; Calculate the mean of the K target neighborhoods to obtain the concentrated value of the K strategy execution value scores.

7. The multi-modal information intelligent analysis system for new energy heavy trucks for mountainous areas as claimed in claim 6 is characterized in that: Scoring centralized screening module for: Perform edge diffusion on the K median neighborhoods according to a preset centralized screening step size to obtain K diffusion neighborhoods; Determine whether the neighborhood density of the K diffusion neighborhoods is greater than or equal to the neighborhood density of the K median neighborhoods. If not, stop the diffusion and use the K median neighborhoods as K target neighborhoods. If so, edge diffusion is performed on the K diffusion neighborhoods according to a preset centralized screening step length until a preset number of diffusion times is met, and the neighborhood obtained by the last diffusion is used as the K target neighborhoods.

8. The multi-modal information intelligent analysis system for new energy heavy trucks for mountainous areas as claimed in claim 6 is characterized in that: Used for: When the K differences are greater than a preset difference threshold, the median values ​​of the K strategy execution value scores are iteratively updated according to the preset centralized screening step length to obtain K stage iterative strategy execution value scores; According to the preset centralized screening step length, constructing K-stage iterative neighborhoods for executing value scores of the K-stage iterative strategies; Determine whether the neighborhood density of the K stage iteration neighborhoods is greater than or equal to the neighborhood density of the K median neighborhoods. If so, iteratively update the K stage iteration strategy execution value scores according to the preset centralized screening step size until the preset number of iterations is met, obtain K target iteration strategy execution value scores, and use the K target iteration strategy execution value scores as the centralized values ​​of the K strategy execution value scores.

Images