Intelligent cooperative control system and method for semi-trailer based on multimodal perception

Through the application of multimodal perception network and knowledge graph, the problems of poor perception capabilities and poor control effects of semi-trailer control systems are solved, higher perception and control effects are achieved, and driving safety is improved.

CN118651213BActive Publication Date: 2024-12-03JIANGSU HAIFENG TRANSPORTATION EQUIP TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410987797.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-12-03
Estimated Expiration
2044-07-23

AI Technical Summary

Technical Problem

The existing semi-trailer control system relies on a single sensor, making it difficult to ensure stability and safety in complex environments, with poor perception and poor control effect.

Method used

Using multimodal perception intelligent collaborative control system, a multimodal perception network is established through the perception completion module, a multimodal acquisition module acquires information, an information class division module performs clustering and division, an information correlation module builds a knowledge graph, and a fusion control module Perform adaptive perception fusion and collaborative control.

Benefits of technology

It improves the perception and control effect of the semi-trailer and improves the driving safety level.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118651213B_ABST
    Figure CN118651213B_ABST
Patent Text Reader

Abstract

The present invention discloses an intelligent collaborative control system and method for a semi-trailer based on multimodal perception, and relates to the field of vehicle control technology. The method comprises: interacting with a target semi-trailer to establish a multimodal perception network. Activating a multimodal acquisition module to perform perception acquisition and obtain multimodal perception information. Using an information classification module, clustering and dividing the acquired information according to perception domain differences. Using an information association module to construct a trailer control knowledge graph, the clustering results are step-by-step associated and grouped to form a plurality of multimodal perception information groups. Using a fusion control module to perform adaptive perception fusion on each multimodal perception information group, generate fused perception information, and transmit it to a plurality of sub-intelligent control units to achieve collaborative control. Thereby achieving the technical effect of improving the perception capability and control effect of the semi-trailer and improving the driving safety level.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of vehicle control technology, and in particular to an intelligent collaborative control system and method for a semi-trailer based on multimodal perception. Background Art

[0002] With the continuous improvement of modern logistics and transportation needs, semi-trailers are becoming more and more common as an important means of transportation. However, in the actual transportation process, there are problems such as vehicle body stability control, balanced load distribution, and safe driving under complex road conditions.

[0003] Existing trailer control systems mostly rely on single sensors to obtain data, such as GNSS, vehicle speed sensors, etc. These systems are difficult to ensure system stability and safety when facing complex and changing environments. In addition, data acquisition by a single sensor also has certain limitations and cannot fully and accurately reflect the actual operating status of the trailer. There are technical problems such as poor perception and poor control effect. Summary of the invention

[0004] The present invention provides an intelligent collaborative control system and method for a semi-trailer based on multimodal perception, so as to solve the technical problems of poor perception ability and poor control effect in the prior art, and achieve the technical effect of improving the perception ability and control effect of the semi-trailer and improving the driving safety level.

[0005] In a first aspect, the present invention provides a semi-trailer intelligent cooperative control system based on multimodal perception, wherein the system comprises:

[0006] A perception completion module is used to interact with a target trailer and establish a multimodal perception network based on the target control requirements of the target trailer.

[0007] A multimodal acquisition module, which is used to activate the multimodal perception network, perform perception acquisition, and obtain multimodal perception information

[0008] An information classification module is used to cluster the acquired multimodal perception information based on perception domain differences to generate clustering results, wherein the clustering results include neighborhood perception information class, local domain perception information class, and beyond-visual-range perception information class.

[0009] An information association module is used to construct a trailer control knowledge graph, and to perform step-by-step association grouping on the clustering results according to the trailer control knowledge graph to obtain multiple multi-modal perception information groups.

[0010] A fusion control module, the fusion control module is used to perform adaptive perception fusion on the multiple multi-mode perception information groups respectively, generate fused perception information, and transmit the fused perception information to multiple sub-intelligent control units for coordinated control.

[0011] In a second aspect, the present invention further provides a semi-trailer intelligent collaborative control method based on multimodal perception, wherein the method comprises:

[0012] Interactive target trailer, based on the target control requirements of the target trailer, a multimodal perception network is established.

[0013] Activate the multimodal perception network to collect perception information and obtain multimodal perception information

[0014] The acquired multimodal perception information is clustered based on perception domain differences to generate clustering results, wherein the clustering results include a neighborhood perception information class, a local domain perception information class, and a beyond-visual-range perception information class.

[0015] A trailer control knowledge graph is constructed, and the clustering results are grouped step by step according to the trailer control knowledge graph to obtain multiple multi-modal perception information groups.

[0016] Adaptive perception fusion is performed on the plurality of multi-mode perception information groups respectively to generate fused perception information, and the fused perception information is transmitted to the plurality of sub-intelligent control units for coordinated control.

[0017] The present invention discloses an intelligent collaborative control system and method for trailers based on multimodal perception, comprising: a perception completion module interacts with a target trailer, and establishes a multimodal perception network based on the target control requirements of the target trailer; a multimodal acquisition module activates the multimodal perception network, performs perception acquisition, and obtains multimodal perception information; an information classification module clusters the acquired multimodal perception information based on perception domain differences, and generates cluster classification results, wherein the cluster classification results include neighborhood perception information classes, local domain perception information classes, and beyond-visual-range perception information classes; an information association module constructs a trailer control knowledge graph, and performs step-by-step association grouping on the cluster classification results according to the trailer control knowledge graph to obtain multiple multimodal perception information groups; a fusion control module performs adaptive perception fusion on multiple multimodal perception information groups, generates fused perception information, and transmits the fused perception information to multiple sub-intelligent control units for collaborative control. The semi-trailer intelligent collaborative control system and method based on multimodal perception disclosed in the present invention solves the technical problems of poor perception ability and poor control effect, and achieves the technical effect of improving the perception ability and control effect of the semi-trailer and improving the driving safety level. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a structural schematic diagram of the semi-trailer intelligent collaborative control system based on multimodal perception of the present invention.

[0019] Figure 2 It is a flow chart of the intelligent cooperative control method of a semi-trailer based on multimodal perception of the present invention.

[0020] Explanation of the accompanying drawings: perception completion module 11, multimodal acquisition module 12, information classification module 13, information association module 14, fusion control module 15. DETAILED DESCRIPTION

[0021] The technical solution provided in the embodiments of the present invention is to solve the technical problems of poor perception ability and poor control effect in the prior art. The overall idea adopted is as follows:

[0022] Firstly, the perception completion module is used to interact with the target trailer, and a multimodal perception network is established based on the target control requirements of the target trailer.

[0023] Next, the multimodal acquisition module activates the multimodal perception network, performs perception acquisition, and obtains multimodal perception information.

[0024] Then, the information classification module clusters the acquired multimodal perception information based on the perception domain differences and generates clustering results, which include neighborhood perception information class, local perception information class, and beyond-horizon perception information class.

[0025] The information association module constructs a trailer control knowledge graph, and performs step-by-step association grouping on the clustering results according to the trailer control knowledge graph to obtain multiple multi-modal perception information groups.

[0026] Finally, the fusion control module performs adaptive perception fusion on multiple multi-mode perception information groups, generates fused perception information, and transmits the fused perception information to multiple sub-intelligent control units for coordinated control.

[0027] The above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods of the specification to better understand the above technical solution. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments of the present invention. It should be understood that the present invention is not limited to the example embodiments used only to explain the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, it should be noted that, for the convenience of description, only the parts related to the present invention are shown in the drawings, rather than all of them.

[0028] Embodiment 1

[0029] Figure 1The structure diagram of the semi-trailer intelligent cooperative control system based on multimodal perception of the present invention is shown in FIG. 1 , wherein the system includes:

[0030] The perception completion module 11 is used to interact with the target trailer and establish a multimodal perception network based on the target control requirements of the target trailer.

[0031] Specifically, the target control requirements of the target semi-trailer are based on the transportation tasks it performs. Different transportation tasks require different towed trailers, and thus have different control requirements. For semi-trailers with different tasks, it is necessary to first obtain the corresponding control requirements. For example, the types of transportation tasks for semi-trailers include: general cargo transportation, cold chain transportation, livestock and poultry transportation, dangerous goods transportation, large equipment transportation, etc. The above tasks are performed by corresponding types of trailers, including ordinary container trailers, refrigerated trailers, tank trailers, flatbed trailers, and plate trailers.

[0032] Specifically, the multimodal perception network needs to integrate multiple sensors and data sources to meet the needs of different tasks. According to different control requirements, appropriate sensors are selected, including: GPS for positioning and path planning; IMU (inertial measurement unit) for posture detection and stability control; temperature sensor for cold chain transportation temperature monitoring. Gas sensor for gas leak detection in dangerous goods transportation. Cameras and radars for environmental perception and obstacle detection. Weight sensor for cargo weight monitoring.

[0033] Optionally, a multimodal perception network is constructed based on existing sensors of the target trailer, including re-dividing the existing sensor network, replacing sensors, adding sensors, etc., so as to make full use of existing resources, reduce costs, and improve the compatibility and scalability of the system.

[0034] Optionally, the multimodal perception network can be divided into multiple sub-networks, and the sub-network division dimensions include division based on perception area, division based on perception object, and division based on perception form. Exemplarily, based on the division of perception area, the perception sensors are divided into the semi-trailer internal perception network (used to monitor the internal environment and status of the semi-trailer, including temperature sensors, humidity sensors, weight sensors, pressure sensors, accelerometers and gyroscopes, etc.), the surrounding situational awareness network (used to monitor the surrounding environment of the semi-trailer to ensure driving safety, including cameras, ultrasonic radars, laser radars, etc.) and the online perception network (obtaining real-time traffic information and remote monitoring data through network connection, including GNSS components, Internet of Vehicles components, Internet of Things components, etc.).

[0035] The multimodal acquisition module 12 is used to activate the multimodal perception network, perform perception acquisition, and obtain multimodal perception information.

[0036] Specifically, first, start the multimodal perception network and check that each perception node and sensor is in working condition. Then, initialize each perception node, including sensor calibration, network connection test, etc. And use standard communication protocols (such as MQTT, CoAP, etc.) to configure the connection between the perception node and the perception data management component. Then, according to the preset perception plan and collection strategy, collect various types of perception data, transmit them to the perception data management component and store them as multimodal perception information.

[0037] Exemplarily, the collected multimodal sensing information includes temperature information (brakes, oil, inside the trailer, etc.), humidity information, pressure information (tire pressure, oil pressure, etc.), acceleration information, image information, sound information, distance information, road traffic information, etc. at multiple locations.

[0038] Optionally, obtaining multimodal perception information also includes preliminary data preprocessing of the raw data collected by various sensors, such as filtering, denoising, correction and other preprocessing operations, to ensure the accuracy and availability of the data.

[0039] Through the above steps and technical means, the activation and use of the multimodal perception network can be realized, rich perception information can be obtained, and strong support can be provided for the intelligent control and management of trailers.

[0040] The information classification module 13 is used to cluster the acquired multimodal perception information based on perception domain differences to generate clustering results, wherein the clustering results include neighborhood perception information class, local domain perception information class, and beyond-visual-range perception information class.

[0041] Specifically, the perception domain refers to the perceived area or the area where the perceived object is located. Exemplarily, according to the distance relationship of the perception range, the multimodal perception information is divided into a neighborhood perception information class, a local domain perception information class and a beyond-visual-range perception information class.

[0042] Specifically, the neighborhood perception information category includes the perception information around the target semi-trailer, which is used to monitor the environment and situation around the semi-trailer. For example, it includes the surrounding image and video information captured by the camera, which is used to identify road conditions, obstacles and traffic signs; the distance and speed of surrounding objects detected by radar, which assists in obstacle avoidance and safe driving; high-precision environment modeling and obstacle detection provided by LiDAR; close-range obstacle information detected by ultrasonic sensors; and surrounding temperature, humidity, light and other environmental information monitored by environmental sensors.

[0043] Specifically, the perception information class of this domain includes the perception information of the target semi-trailer itself, which is used to monitor the internal environment and status of the semi-trailer. For example, it includes the temperature of the cargo monitored by the temperature sensor to ensure the temperature requirements in cold chain transportation; the humidity of the cargo monitored by the humidity sensor to prevent the cargo from getting damp; the weight and distribution of the cargo monitored by the weight sensor to ensure uniform and safe load; the vibration and tilt of the trailer monitored by the accelerometer and gyroscope to prevent damage to the cargo; the concentration of harmful gases monitored by the gas sensor; the status of the vehicle monitored by the body sensor, including tire pressure, fuel consumption, engine status, etc.

[0044] Specifically, beyond-line-of-sight perception information refers to perception information received from the cloud, vehicle network or other data sources, which is used to expand the perception range and obtain more extensive environmental information. Examples include GNSS positioning information; real-time traffic information obtained by vehicle-to-everything (V2X) components communicating with other vehicles and infrastructure; real-time weather data obtained from meteorological service sources; real-time traffic conditions, road congestion information and accident reports obtained from traffic management systems, etc.

[0045] The information association module 14 is used to construct a trailer control knowledge graph, and to perform step-by-step association grouping on the clustering results according to the trailer control knowledge graph to obtain multiple multi-modal perception information groups.

[0046] The trailer control knowledge graph contains all the knowledge and information about trailer control, including the names, functions and operation methods of various parts, such as wheels, brake systems, suspension systems, trailer connectors, cargo securing equipment, etc.; various systems and equipment used to control vehicle movement and operation, such as brake control systems, power control systems, steering control systems, etc. Common maintenance tasks, fault diagnosis and repair methods, and related tools and equipment. Best practices for driving safety, applicable traffic rules and standards, possible safety issues and risks, etc.

[0047] Specifically, clustering results are grouped according to the trailer control knowledge graph to obtain multiple multi-modal perception information groups, which help to better understand the real-time status of the target trailer, and then accurately control the operation of the trailer, improving driving safety and efficiency.

[0048] In some embodiments, the steps of constructing a trailer control knowledge graph include:

[0049] Connect to the trailer control data source to obtain trailer control information.

[0050] The trailer control data is parsed to extract and define key control entities and control relationships.

[0051] A trailer control knowledge graph is constructed with the key control entity graph nodes and control relationships as relationship edges.

[0052] Specifically, first, determine the data sources related to trailer control, including but not limited to trailer design drawings, operating manuals, sensor data, historical operation records, fault reports, etc. Then, based on data acquisition tools and corresponding data interfaces, APIs, etc., connect to these data sources to obtain trailer control information.

[0053] Furthermore, the trailer control data is parsed to identify the key control information, and the key control entities are extracted and defined through named entity recognition (NER) technology, text mining algorithms, relationship extraction algorithms, etc., such as trailer components (brake system, power system, hydraulic system, lighting system, etc.), control parameters (temperature, pressure, speed, etc.), operation steps, etc. Then, the relationship between the above key control entities is defined, such as the connection relationship between components, the influence of control parameters on components, and the order relationship of operation steps.

[0054] Optionally, the extracted entities and relations are verified and supplemented in combination with the knowledge of domain experts, which helps to improve the accuracy of the generated trailer control knowledge graph.

[0055] Specifically, the extracted and defined key control entities are used as graph nodes in the knowledge graph. The defined control relationships are used as relationship edges between graph nodes. A graph database or a knowledge graph construction tool is used to organize these nodes and edges, and then a trailer control knowledge graph is constructed. Exemplarily, the knowledge graph is stored and managed by Neo4j, and a knowledge graph construction tool such as RDF, OWL, etc. is used to define and construct the knowledge graph.

[0056] In some embodiments, the clustering results are grouped step by step according to the trailer control knowledge graph to obtain multiple multi-modal perception information groups, including:

[0057] The interactive target trailer analyzes multiple sub-intelligent control units respectively, extracts control targets and outputs them as multiple control target clusters.

[0058] Based on the trailer control knowledge graph, a plurality of control target clusters are collected, the clustering results are traversed, and intra-class perception information is grouped for a plurality of information classes respectively.

[0059] Based on the intra-class perception information grouping result, expansion matching is performed to generate a plurality of the multi-mode perception information groups.

[0060] Specifically, first, the target semi-trailer or the manufacturing source of the target semi-trailer is connected through an interactive interface, and multiple sub-intelligent control units on the trailer are parsed to extract the control targets of each sub-intelligent control unit, such as temperature control, brake control, power control, etc. Then, the extracted multiple control targets are output as multiple control target clusters.

[0061] Specifically, the clustering results are traversed to group the intra-class perception information of multiple information classes. This is achieved by matching the associated graph nodes for the above-mentioned multiple control targets based on the trailer control knowledge graph in each information class, and storing this information as multiple groups of intra-class perception information.

[0062] Specifically, based on the result of the intra-class perception information grouping, expansion matching is performed. This is achieved by merging the information groups with the same control objectives corresponding to the multiple groups of intra-class perception information generated by different information classes. The merged multiple information groups are multiple multi-mode perception information groups. Multiple multi-mode perception information groups contain all the perception information required for multiple control objectives.

[0063] Specifically, the multi-mode sensing information contained in the multiple multi-mode sensing information groups can be repeated. In other words, the same sensing information can be used as the control basis for multiple different control targets at the same time. For example, the tire pressure data of the wheels can be used as the control basis information for the driving safety of the target semi-trailer and the trailer weight distribution control at the same time.

[0064] The above method steps realize efficient integration of control information through grouping and expansion matching of knowledge graph and multimodal perception information. At the same time, the step-by-step associative grouping method improves the system's adaptability to semi-trailers in different situations, thereby improving the portability of the system.

[0065] The fusion control module 15 is used to perform adaptive perception fusion on the multiple multi-mode perception information groups respectively, generate fused perception information, and transmit the fused perception information to the multiple sub-intelligent control units for coordinated control.

[0066] In some embodiments, adaptive perception fusion is performed on a plurality of the multi-mode perception information groups to generate fused perception information, and the execution steps include:

[0067] Based on the trailer control knowledge graph, statistical analysis is performed in combination with historical data to obtain the association frequency and correlation between the key control entities and determine the relationship strength set.

[0068] An association relationship is established between the relationship strength set and the relationship edge in the trailer control knowledge graph to generate an enhanced trailer control knowledge graph.

[0069] Based on the enhanced trailer control knowledge graph, adaptive perception fusion of multiple multi-modal perception information groups is performed.

[0070] Specifically, the trailer control knowledge graph is used in combination with historical data for statistical analysis to extract the association frequency and correlation between key control entities. First, historical data is acquired and parsed to extract the interaction records of key control entities. Then, the number of interactions between each pair of key control entities is counted, and the association frequency and correlation indicators (such as the Pearson correlation coefficient) are calculated to form a relationship strength set. The relationship strength set contains the strength information of the relationship between entities, which is used to guide the subsequent perception fusion process.

[0071] Specifically, the frequency and correlation data in the relationship strength set are mapped to the relationship edge of the knowledge graph, and the weight of the relationship edge is adjusted according to the data to form an enhanced knowledge graph. The updated knowledge graph reflects the more accurate association strength between key control entities.

[0072] Furthermore, based on the enhanced trailer control knowledge graph, adaptive perceptual fusion of multiple multimodal perceptual information groups is performed. First, for each multimodal perceptual information group, the enhanced knowledge graph is used to analyze the perceptual information within each group. Then, according to the weights of the relationship edges in the enhanced knowledge graph, the information with high correlation within the group is identified, and information merging and de-redundancy processing are performed. Finally, non-redundant fused perceptual information is generated and output for trailer control.

[0073] Optionally, the adaptive perception fusion process includes multiple levels of adaptive fusion processes, including data-level fusion and feature-level fusion. Exemplarily, through data-level fusion, the raw data of multiple sensors or data sources with the same data format or data category are aligned in time and space, redundant data and noise are removed, and the data format is unified; through feature-level fusion, the key features of the multimodal perception information in the multimodal perception information group are extracted, such as the average, maximum, minimum, variance, etc., and the data features of multiple multimodal perception information are combined to form a comprehensive feature vector, such as combining the data features of the tire pressure sensor and the temperature sensor to generate a feature vector for trailer tire health status assessment.

[0074] In some implementations, the fused perception information is transmitted to multiple sub-intelligent control units for coordinated control, and the execution step further includes:

[0075] The fused perception information is transmitted to a reference control unit, where the reference control unit is a local control unit of the target trailer.

[0076] The benchmark control unit matches and generates benchmark control instructions based on a preset control strategy, and the benchmark control instructions at least include heat dissipation control, brake control, power control, and lighting control.

[0077] The target semi-trailer is subjected to reference control based on the reference control instruction.

[0078] Specifically, the multi-mode perception information after adaptive perception fusion processing is transmitted to the benchmark control unit, where the benchmark control unit is the local control unit of the target semi-trailer, responsible for executing various core control functions, including heat dissipation control, brake control, power control, lighting control, etc.

[0079] Among them, the reference control unit presets a series of control strategies for different control scenarios. After receiving the fused perception information, the reference control unit matches the corresponding control strategy for the fused perception information and generates corresponding control instructions. Exemplarily, the control instructions include but are not limited to: controlling the start and stop and working mode of the cooling system according to the temperature sensor data; controlling the response of the braking system according to the speed and road condition information; adjusting the output power of the engine and transmission system according to the load and road conditions; and controlling the brightness and switch state of the headlights according to the ambient light and driving status.

[0080] In some implementations, the fused perception information is transmitted to multiple sub-intelligent control units for coordinated control, and the execution step further includes:

[0081] The fused perception information is transmitted to an active safety control unit.

[0082] The active safety control unit performs target detection and risk identification based on the fused perception information, and outputs a risk identification result, wherein the risk identification result includes a risk category and a risk level.

[0083] Based on the risk identification result and the active safety response matrix, an active safety control instruction is obtained, and active safety control of the target semi-trailer is performed.

[0084] Specifically, the active safety control unit uses fused perception information as input for risk identification. First, based on the fused perception information, target detection is performed through target detection algorithms (such as YOLOV4, Fast R-CNN, SSD, etc.) to identify potential risk targets around the target semi-trailer, including other vehicles, pedestrians, obstacles, etc. Then, after the target is detected, the type of potential risk target is determined by features such as shape, size, and motion pattern. Then, the distance, approach speed, and direction between the potential risk target and the target semi-trailer are calculated to track the detected target using a multi-target tracking algorithm (such as SORT, DeepSORT) to predict its future position. Then, based on the type information, distance information, approach speed information, and predicted position information of the above potential risk targets, it is evaluated whether there is a safety risk. If there is a risk, the system determines the level of the risk.

[0085] Exemplarily, the risk category and risk level are determined based on the analytic hierarchy process or decision tree model. First, historical risk assessment data is collected, including target type information, distance information, approach speed information, predicted location information, and actual risk level. Then, relevant features are selected as inputs to the decision tree model, and the decision tree model is trained using the training data set, and the performance of the model, such as accuracy and recall rate, is evaluated using the validation data set. Then, the new perception information is input into the decision tree model for risk discrimination, and the risk category and risk level are output.

[0086] The introduction of the active safety control unit uses neural network algorithms for target detection and risk identification, effectively improving the intelligence level and responsiveness of the system, and improving the safety of the semi-trailer by accurately detecting targets and judging risks.

[0087] In some implementations, the fused perception information is transmitted to multiple sub-intelligent control units for coordinated control, and the execution step further includes:

[0088] The fused perception information is transmitted to an intelligent route control unit.

[0089] The intelligent route control unit updates the road condition information with the fused perception information, and performs path replanning according to the updated road condition information.

[0090] The path replanning results are visualized to achieve route control of the target trailer.

[0091] Specifically, the intelligent route control unit updates the road condition information based on the fused perception information, including the synchronous update of information such as traffic flow, road conditions, and obstacle locations, wherein the synchronously updated road condition information has a time stamp. Furthermore, based on the updated road condition information, the intelligent route control unit comprehensively considers the target semi-trailer location, the trailer's special requirements (such as weight, size, etc.), and other factors that may affect driving to replan the route, including calculating the new route, determining the steering angle and speed, etc.

[0092] Optionally, the intelligent route control unit generates the initial path for path replanning based on the A* algorithm (A-star algorithm), Bellman-Ford algorithm, Floyd-Warshall algorithm, etc., and then builds a multi-objective optimization model, defines the optimization objective function considering multiple optimization objectives (such as the shortest path, the least time, the lowest cost, the lowest risk, etc.), and sets optimization constraints, including the maximum acceptable driving time, the maximum fuel consumption, the longest time, the vehicle width limit, the height limit, the weight limit, etc. Finally, the heuristic algorithm (such as the genetic algorithm, the particle swarm optimization algorithm, etc.) is applied to optimize on the basis of the initial path, and the advantages and disadvantages of each candidate path are evaluated according to the optimization objective function, and the optimal path is selected as the final path replanning result.

[0093] Specifically, the path replanning results are visualized, including: displaying the final path on the map to ensure that the driver can intuitively see the planned driving route; providing detailed navigation guidance through voice, including information such as turns and lane changes. By using an intelligent route control unit, the target semi-trailer can perform effective path planning and route control based on real-time road conditions, thereby improving the driving efficiency and safety of the semi-trailer.

[0094] Furthermore, the execution steps of the system also include:

[0095] The fused perception information and collaborative control records are synchronized to a cloud collaborative center.

[0096] The cloud collaboration center updates the cloud perception information according to the synchronously acquired fusion perception information and the collaborative control record.

[0097] Based on the updated cloud perception information, cloud-end shared collaborative control is performed on the multiple semi-trailer individuals under the cloud collaborative center.

[0098] Optionally, based on a preset synchronization frequency, the fused perception information and collaborative control records are transmitted to the cloud collaborative center in real time, and the cloud collaborative center updates the cloud perception information based on the synchronously acquired fused perception information and collaborative control records. Exemplarily, it includes the location, speed, direction, road condition and other information of multiple semi-trailers under the jurisdiction of the cloud collaborative center.

[0099] Furthermore, based on the updated cloud perception information, the cloud coordination center conducts cloud-end shared collaborative control of the multiple semi-trailers under its jurisdiction, including avoiding congested sections encountered by other semi-trailers in advance, planning routes based on the real-time road conditions of other vehicles, and providing rescue or assistance to other vehicles.

[0100] Through the cloud collaboration center, multiple semi-trailers can share perception information and control strategies and conduct effective collaborative control, thereby improving the driving efficiency and safety of the entire fleet.

[0101] In summary, the semi-trailer intelligent cooperative control system based on multimodal perception provided by the present invention has the following technical effects:

[0102] Through the perception completion module, the target trailer is interacted with, and a multimodal perception network is established based on the target control requirements of the target trailer; the multimodal acquisition module activates the multimodal perception network, performs perception acquisition, and obtains multimodal perception information; the information classification module clusters the obtained multimodal perception information based on perception domain differences, and generates clustering results, wherein the clustering results include neighborhood perception information class, local domain perception information class, and beyond-visual-range perception information class; the information association module constructs a trailer control knowledge graph, and performs step-by-step association grouping on the clustering results according to the trailer control knowledge graph to obtain multiple multimodal perception information groups; the fusion control module performs adaptive perception fusion on the multiple multimodal perception information groups, generates fused perception information, and transmits the fused perception information to multiple sub-intelligent control units for coordinated control. Thereby achieving the technical effect of improving the perception capability and control effect of the semi-trailer and improving the driving safety level.

[0103] Embodiment 2

[0104] Figure 2 is a flow chart of the intelligent cooperative control method of a semi-trailer based on multimodal perception of the present invention. For example, Figure 1 The structural schematic diagram of the semi-trailer intelligent cooperative control system based on multimodal perception in the present invention is based on Figure 2 The process shown is implemented.

[0105] Based on the same concept as the semi-trailer intelligent cooperative control system based on multimodal perception in the above embodiment, the present invention also provides a semi-trailer intelligent cooperative control method based on multimodal perception, including:

[0106] Interactive target trailer, based on the target control requirements of the target trailer, a multimodal perception network is established.

[0107] Activate the multimodal perception network to collect perception information and obtain multimodal perception information

[0108] The acquired multimodal perception information is clustered based on perception domain differences to generate clustering results, wherein the clustering results include a neighborhood perception information class, a local domain perception information class, and a beyond-visual-range perception information class.

[0109] A trailer control knowledge graph is constructed, and the clustering results are grouped step by step according to the trailer control knowledge graph to obtain multiple multi-modal perception information groups.

[0110] Adaptive perception fusion is performed on the plurality of multi-mode perception information groups respectively to generate fused perception information, and the fused perception information is transmitted to the plurality of sub-intelligent control units for coordinated control.

[0111] In some implementations, building a trailer control knowledge graph includes:

[0112] Connect to the trailer control data source to obtain trailer control data. Parse the trailer control data to extract and define key control entities and control relationships. Use the key control entity graph nodes and control relationships as relationship edges to construct a trailer control knowledge graph.

[0113] In some implementations, the clustering results are grouped step by step according to the trailer control knowledge graph to obtain multiple multi-modal perception information groups, including:

[0114] For the interactive target trailer, the multiple sub-intelligent control units are analyzed respectively, the control targets are extracted and output as multiple control target clusters. Based on the trailer control knowledge graph, multiple control target clusters are collected, the clustering results are traversed, and the multiple information classes are grouped with intra-class perception information. Based on the intra-class perception information grouping results, expansion matching is performed to generate multiple multi-modal perception information groups.

[0115] In some implementations, performing adaptive perception fusion on a plurality of the multi-modal perception information groups to generate fused perception information includes:

[0116] Based on the trailer control knowledge graph, statistical analysis is performed in combination with historical data to obtain the association frequency and correlation between the key control entities and determine a relationship strength set. An association relationship between the relationship strength set and the relationship edge in the trailer control knowledge graph is established to generate an enhanced trailer control knowledge graph. Based on the enhanced trailer control knowledge graph, adaptive perception fusion of multiple multi-modal perception information groups is performed.

[0117] In some implementations, transmitting the fused perception information to multiple sub-intelligent control units for coordinated control also includes:

[0118] The fused perception information is transmitted to a benchmark control unit, which is a local control unit of the target trailer. The benchmark control unit matches and generates benchmark control instructions based on a preset control strategy, and the benchmark control instructions at least include heat dissipation control, brake control, power control, and lighting control. The benchmark control of the target trailer is performed based on the benchmark control instructions.

[0119] In some implementations, transmitting the fused perception information to multiple sub-intelligent control units for coordinated control also includes:

[0120] The fused perception information is transmitted to an active safety control unit.

[0121] The active safety control unit performs target detection and risk identification based on the fused perception information, and outputs a risk identification result, wherein the risk identification result includes a risk category and a risk level.

[0122] Based on the risk identification result and the active safety response matrix, an active safety control instruction is obtained, and active safety control of the target semi-trailer is performed.

[0123] In some implementations, transmitting the fused perception information to multiple sub-intelligent control units for coordinated control also includes:

[0124] The fused perception information is transmitted to an intelligent route control unit.

[0125] The intelligent route control unit updates the road condition information with the fused perception information, and performs path replanning according to the updated road condition information.

[0126] The path replanning results are visualized to achieve route control of the target trailer.

[0127] In some implementations, the method further includes:

[0128] The fused perception information and collaborative control records are synchronized to a cloud collaborative center.

[0129] The cloud collaboration center updates the cloud perception information according to the synchronously acquired fusion perception information and the collaborative control record.

[0130] Based on the updated cloud perception information, cloud-end shared collaborative control is performed on the multiple semi-trailer individuals under the cloud collaborative center.

[0131] It should be understood that the embodiments mentioned in this specification focus on their differences from other embodiments. The specific embodiments in the aforementioned embodiment one are also applicable to the semi-trailer intelligent collaborative control method based on multimodal perception described in embodiment two. For the sake of brevity of the specification, they will not be further elaborated here.

[0132] It should be understood that the embodiments disclosed in the present invention and the above description can enable those skilled in the art to use the present invention to implement the present invention. At the same time, the present invention is not limited to the above-mentioned embodiments. It should be understood that those skilled in the art can still modify the technical solutions recorded in the above-mentioned embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. The intelligent cooperative control system of semi-trailer based on multimodal perception is characterized by: The system comprises: A perception completion module, wherein the perception completion module is used to interact with the target trailer and establish a multimodal perception network based on the target control requirements of the target trailer; A multimodal acquisition module, the multimodal acquisition module is used to activate the multimodal perception network, perform perception acquisition, and obtain multimodal perception information; An information classification module, the information classification module is used to cluster the acquired multimodal perception information based on perception domain differences to generate cluster classification results, wherein the cluster classification results include a neighborhood perception information class, a local perception information class, and a beyond-visual-range perception information class, wherein the local perception information class includes the perception information of the target trailer itself, and is used to monitor the internal environment and status of the trailer; An information association module, the information association module is used to construct a trailer control knowledge graph, and to perform step-by-step association grouping on the clustering division results according to the trailer control knowledge graph to obtain a plurality of multi-modal perception information groups; A fusion control module, the fusion control module is used to perform adaptive perception fusion on the multiple multi-mode perception information groups, generate fused perception information, and transmit the fused perception information to the multiple sub-intelligent control units for coordinated control; Construct a trailer control knowledge graph. The execution steps include: Connect to the trailer control data source to obtain trailer control information; Parsing the trailer control data, extracting and defining key control entities and control relationships; Using the key control entity graph nodes and control relationships as relationship edges, a trailer control knowledge graph is constructed; According to the trailer control knowledge graph, the clustering division results are grouped step by step to obtain multiple multi-modal perception information groups, and the execution steps include: The interactive target semi-trailer analyzes multiple sub-intelligent control units respectively, extracts control targets and outputs them as multiple control target clusters; Based on the trailer control knowledge graph, a plurality of control target clusters are collected, the clustering results are traversed, and intra-class perception information grouping is performed on a plurality of information classes respectively; Based on the intra-class perception information grouping result, expansion matching is performed to generate a plurality of the multi-mode perception information groups.

2. The semi-trailer intelligent cooperative control system based on multimodal perception as claimed in claim 1, characterized in that: Adaptively performing perception fusion on a plurality of the multi-mode perception information groups to generate fused perception information, the execution steps comprising: Based on the trailer control knowledge graph, statistical analysis is performed in combination with historical data to obtain the association frequency and correlation between the key control entities and determine the relationship strength set; Establishing an association relationship between the relationship strength set and the relationship edge in the trailer control knowledge graph to generate an enhanced trailer control knowledge graph; Based on the enhanced trailer control knowledge graph, adaptive perception fusion of multiple multi-modal perception information groups is performed.

3. The semi-trailer intelligent cooperative control system based on multimodal perception as claimed in claim 2, characterized in that: Transmitting the fused perception information to multiple sub-intelligent control units for coordinated control, the execution step also includes: Transmitting the fused perception information to a reference control unit, wherein the reference control unit is a local control unit of the target trailer; The benchmark control unit matches and generates benchmark control instructions based on a preset control strategy, and the benchmark control instructions at least include heat dissipation control, brake control, power control, and lighting control; The target semi-trailer is subjected to reference control based on the reference control instruction.

4. The semi-trailer intelligent cooperative control system based on multimodal perception as claimed in claim 2, characterized in that: Transmitting the fused perception information to multiple sub-intelligent control units for coordinated control, the execution step also includes: Transmitting the fused perception information to an active safety control unit; The active safety control unit performs target detection and risk discrimination based on the fused perception information, and outputs a risk discrimination result, wherein the risk discrimination result includes a risk category and a risk level; Based on the risk identification result and the active safety response matrix, an active safety control instruction is obtained, and active safety control of the target semi-trailer is performed.

5. The semi-trailer intelligent cooperative control system based on multimodal perception as claimed in claim 2, characterized in that: Transmitting the fused perception information to multiple sub-intelligent control units for coordinated control, the execution step also includes: Transmitting the fused perception information to an intelligent route control unit; The intelligent route control unit updates the road condition information with the fused perception information, and performs path replanning according to the updated road condition information; The path replanning results are visualized to achieve route control of the target trailer.

6. The semi-trailer intelligent cooperative control system based on multimodal perception as claimed in claim 1, characterized in that: The execution steps of the system also include: Synchronizing the fusion perception information and collaborative control records to a cloud collaborative center; The cloud collaboration center updates the cloud perception information according to the fusion perception information and the collaborative control record acquired synchronously; Based on the updated cloud perception information, cloud-end shared collaborative control is performed on the multiple semi-trailer individuals under the cloud collaborative center.

7. The intelligent cooperative control method of semi-trailer based on multimodal perception is characterized by: The method is applied to the semi-trailer intelligent cooperative control system based on multimodal perception according to any one of claims 1 to 6, and the method comprises: Interactive target trailer, based on the target control requirements of the target trailer, a multimodal perception network is established; Activate the multimodal perception network to collect perception information and obtain multimodal perception information Clustering the acquired multimodal perception information based on perception domain differences to generate clustering results, wherein the clustering results include a neighborhood perception information class, a local perception information class, and a beyond-visual-range perception information class, wherein the local perception information class includes the perception information of the target trailer itself, which is used to monitor the internal environment and status of the trailer; Constructing a trailer control knowledge graph, and performing step-by-step association grouping on the clustering results according to the trailer control knowledge graph to obtain a plurality of multi-modal perception information groups; Adaptive perception fusion is performed on the plurality of multi-mode perception information groups respectively to generate fused perception information, and the fused perception information is transmitted to the plurality of sub-intelligent control units for coordinated control.

Citation Information

Cited By

  • Trailer intelligent network connection man-machine co-driving cooperative control system based on multi-mode perception

    CN121433067A