Traffic control method and device based on multi-modal perception, equipment and storage medium
By generating fused feature videos through multimodal perception technology, the problem of insufficient recognition of non-traditional vehicles by existing traffic signal control is solved, achieving more accurate traffic signal control and higher traffic efficiency.
Patent Information
- Application Number
- CN202510709614.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-29
- Publication Date
- 2025-10-10
AI Technical Summary
Existing traffic signal control methods have difficulty accurately identifying non-traditional vehicles such as electric bicycles, resulting in inaccurate traffic signal control and reduced traffic efficiency.
It uses multimodal perception technology, combined with visual cameras, lidar and communication equipment to obtain road image data, point cloud data and vehicle driving information, generate fused feature videos, determine scene recognition results through traffic scene recognition models, and control traffic signs based on the results.
The accuracy of traffic signal control is improved, and traffic efficiency is enhanced.
Smart Images
Figure CN120766548A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of traffic control, and in particular to a traffic control method based on multimodal perception, a traffic control device based on multimodal perception, a traffic control equipment based on multimodal perception, and a storage medium. Background Art
[0002] Traffic signal control is an important means of traffic management, and its purpose is to optimize traffic flow and reduce traffic congestion. At present, common traffic signal control generally involves vehicles passing through coils buried under the road surface and collecting changes in the coil's inductance. The traffic flow information is determined based on the changes in inductance, and the traffic lights are adjusted based on the traffic flow data and the number of lanes set on the road. However, with the diversification of people's transportation, this recognition method is more difficult to identify electric bicycles and light motorcycles, resulting in the inability to accurately identify the actual traffic flow information of each road section, which leads to inaccurate traffic signal control and reduced traffic efficiency.
[0003] The above content is only used to assist in understanding the technical solution of the present invention and does not constitute an admission that the above content is prior art. Summary of the Invention
[0004] The main purpose of the present invention is to provide a traffic control method based on multimodal perception, a traffic control device based on multimodal perception, a traffic control equipment based on multimodal perception, and a storage medium based on multimodal perception, aiming to improve traffic efficiency. To achieve the above-mentioned purpose, the present invention provides a traffic control method based on multimodal perception, which includes the following steps:
[0005] Obtain road image data collected by visual cameras, road point cloud data collected by lidar, and vehicle driving information received by communication equipment;
[0006] generating a fusion feature video according to the road image data, the road point cloud data, and the vehicle driving information;
[0007] Determining a scene recognition result based on the fused feature video and the traffic scene recognition model;
[0008] The traffic indication signal is controlled according to the scene recognition result.
[0009] Optionally, the step of generating a fusion feature video based on the road image data, the road point cloud data, and the vehicle driving information includes:
[0010] generating a fused feature image according to the road image data and the road point cloud data;
[0011] Marking the vehicle object of the fused feature image according to the vehicle driving information and the road point cloud data, so that the vehicle object of the fused feature image includes: driving state information;
[0012] A fused feature video is determined according to the driving state information of the vehicle object in the fused feature image.
[0013] Optionally, the step of generating a fused feature image based on the road image data and the road point cloud data includes:
[0014] determining point cloud feature points based on the road point cloud data, and determining image feature points based on the road image data;
[0015] Matching the point cloud feature points with the image feature points to obtain a matching result;
[0016] The road point cloud data and the road image data are combined to generate a fused feature image according to the matching result, wherein each pixel of the fused feature image includes corresponding color information and distance information.
[0017] Optionally, the step of receiving vehicle driving information by the communication device includes:
[0018] The vehicle driving information is received at a first preset frequency.
[0019] The step of marking the vehicle object of the fused feature image according to the vehicle driving information and the road point cloud data comprises:
[0020] When the vehicle object in the fused feature image does not include vehicle driving information, marking the vehicle object in the fused feature image according to the vehicle driving information;
[0021] When the vehicle object of the fused feature image includes vehicle driving information, the mark is updated at a first preset frequency so that the driving state information is updated at the first preset frequency.
[0022] Optionally, the driving state information includes: driving speed, driving direction, and driving acceleration, and the step of determining the fused feature video according to the driving state information of the vehicle object in the fused feature image includes:
[0023] Determining a virtual position corresponding to the vehicle object in a plurality of simulated fused feature images based on the vehicle object's speed, direction, and acceleration in the fused feature image, wherein the simulated fused feature image is a virtual image generated based on the fused feature image after a current moment;
[0024] The fused feature video is generated according to the fused feature image and the multiple simulated fused feature images.
[0025] Optionally, the step of determining a scene recognition result based on the fused feature video and the traffic scene recognition model includes:
[0026] Identifying object feature data of each target object in the fused feature video to obtain a plurality of object feature data;
[0027] The plurality of object feature data are input into the traffic scene recognition model, a road scene index is output, and a scene recognition result is determined according to the road scene index.
[0028] Optionally, the step of controlling the traffic indication signal according to the scene recognition result includes:
[0029] determining traffic control data according to the scene recognition result and a first corresponding relationship, wherein the first corresponding relationship is a corresponding relationship between the scene recognition result and the traffic control data;
[0030] The traffic indication signal is controlled according to the traffic control data.
[0031] In addition, to achieve the above-mentioned purpose, the present invention further provides a traffic control device based on multimodal perception, the traffic control device based on multimodal perception comprising:
[0032] An acquisition module is used to obtain road image data collected by a visual camera, road point cloud data collected by a lidar, and vehicle driving information received by a communication device;
[0033] A fusion module, configured to generate a fusion feature video based on the road image data, the road point cloud data, and the vehicle driving information;
[0034] A recognition module, configured to determine a scene recognition result based on the fused feature video and the traffic scene recognition model;
[0035] A control module is used to control the traffic indication signal according to the scene recognition result.
[0036] In addition, to achieve the above-mentioned purpose, the present invention also provides a traffic control device based on multimodal perception, characterized in that the device includes: a memory, a processor, and a traffic control program based on multimodal perception stored on the memory and executable on the processor, the traffic control program based on multimodal perception being configured to implement the steps of any of the above-mentioned traffic control methods based on multimodal perception.
[0037] In addition, to achieve the above-mentioned purpose, the present invention also provides a storage medium, on which a traffic control program based on multimodal perception is stored. When the traffic control program based on multimodal perception is executed by a processor, the steps of the traffic control method based on multimodal perception described above are implemented.
[0038] The present invention proposes a traffic control method based on multimodal perception. The method generates a fused feature video through the road image data, the road point cloud data and the vehicle driving information. Compared with traditional coils buried under the road surface, the method can more accurately determine the vehicle driving conditions on the road, and determine the scene recognition result based on the fused feature video and the traffic scene recognition model; the scene occurring in the traffic scene is identified in real time, and the traffic indication signal is controlled according to the scene recognition result, thereby improving the accuracy of traffic signal control and further improving traffic efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 Schematic diagram of the structure of a traffic control device based on multimodal perception in a hardware operating environment according to an embodiment of the present invention;
[0040] Figure 2 1 is a flow chart of a first embodiment of a traffic control method based on multimodal perception according to the present invention;
[0041] Figure 3 1 is a flow chart of a second embodiment of a traffic control method based on multimodal perception according to the present invention;
[0042] Figure 4 3 is a schematic diagram of a specific flow of step S21 of the second embodiment of the traffic control method based on multimodal perception of the present invention.
[0043] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION
[0044] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0045] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a traffic control device based on multimodal perception in the hardware operating environment involved in an embodiment of the present invention.
[0046] like Figure 1As shown, the traffic control device based on multi-modal perception can include a processor 1001, for example, a central processing unit (CPU), a communication bus 1002, an interactive device 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection and communication between the components. The interactive device 1003 can include a display, an input unit such as a keyboard, and can also be connected to the communication bus through a standard wired interface or a wireless interface. The network interface 1004 can optionally include a standard wired interface or a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 1005 can be a high-speed random access memory (RAM) memory or a stable non-volatile memory (Non-Volatile Memory, NVM) such as a disk memory. The memory 1005 can also be a storage device independent of the aforementioned processor 1001.
[0047] Those skilled in the art can understand that Figure 1 The structure shown in the figure does not constitute a limitation on the traffic control device based on multi-modal perception, and can include more or fewer components than the figure, or combine certain components, or different component arrangements.
[0048] As Figure 1 As shown, the memory 1005 as a storage medium can include an operating system, a data storage module, a network communication module, a user interface module, and a traffic control program based on multi-modal perception.
[0049] In Figure 1 The traffic control device based on multi-modal perception shown in the figure, the network interface 1004 is mainly used for data communication with other devices; the interactive device 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the traffic control device based on multi-modal perception in the application can be arranged in the traffic control device based on multi-modal perception, and the traffic control device based on multi-modal perception calls the traffic control program based on multi-modal perception stored in the memory 1005 through the processor 1001, and executes the traffic control method based on multi-modal perception provided by the embodiment of the application.
[0050] The embodiment of the application provides a traffic control method based on multi-modal perception, referring to Figure 2 , Figure 2 is a flowchart of a traffic control method based on multi-modal perception according to the first embodiment of the application.
[0051] In this embodiment, the traffic control method based on multimodal perception includes:
[0052] Step S1, obtaining road image data collected by a visual camera, road point cloud data collected by a lidar, and vehicle driving information received by a communication device;
[0053] In this embodiment, the visual camera, the laser radar, and the communication equipment are preferably installed at traffic intersections or in areas with high traffic volume. The road image data and road point cloud data here generally need to be installed in the same area to monitor the same roads and lanes. The vehicle driving information is realized through technologies such as 5G communication, dedicated short-range communication (DSRC), and cellular vehicle networking, achieving real-time high-frequency interconnection between vehicles and infrastructure, thereby realizing real-time vehicle driving information.
[0054] Step S2, generating a fusion feature video based on the road image data, the road point cloud data and the vehicle driving information;
[0055] It should be noted that although the road image data and road point cloud data are both collected for the same road, due to differences in sensor placement, the road image data and road point cloud data need to be matched to calibrate the two data sets and improve the accuracy of the subsequent fused feature video generation. At an intersection, two sets of sensors can be set up, each including road image data collected by a corresponding visual camera, road point cloud data collected by a lidar, and vehicle driving information received by a communication device. The two sets of sensors collect data in the direction of the corresponding road, and the roads collected by the two sets of sensors are perpendicular to each other.
[0056] Step S3, determining a scene recognition result based on the fused feature video and the traffic scene recognition model;
[0057] Specifically, the generated fusion feature video is input into the traffic scene recognition model to determine the road scene index of the current road, and the scene recognition result is determined based on the road scene index. The road scene index here includes the number of vehicles on the road, the road speed, the lane speed difference, etc.
[0058] Step S4: Control the traffic signal according to the scene recognition result.
[0059] The traffic signal corresponding to the current intersection is determined based on the scene recognition result. Specifically, preferably, because sensors collecting information from different roads are installed at an intersection, the number of scene recognition results here is more than one, so that the traffic signal can be controlled based on the recognition results. For example, the time ratio of the traffic lights in each direction corresponding to the intersection can be adjusted.
[0060] In this embodiment, a fused feature video is generated by the road image data, the road point cloud data and the vehicle driving information. Compared with the traditional coil buried under the road surface, it can more accurately determine the vehicle driving conditions on the road, and determine the scene recognition result based on the fused feature video and the traffic scene recognition model; the scene occurring in the traffic scene is identified in real time, and the traffic indication signal is controlled according to the scene recognition result, thereby improving the accuracy of traffic signal control and further improving traffic efficiency.
[0061] Furthermore, based on the first embodiment, a second embodiment of the traffic control method based on multimodal perception of the present invention is proposed. In this embodiment, referring to Figure 3 The step of generating a fusion feature video based on the road image data, the road point cloud data and the vehicle driving information includes:
[0062] Step S21, generating a fused feature image based on the road image data and the road point cloud data;
[0063] By respectively extracting the image feature points corresponding to the road image data and the point cloud feature points of the road point cloud data, and then aligning them based on the point cloud feature points and the image feature points, optionally, by quantifying the geometric properties of the local point cloud, such as curvature, plane normal, point cloud density, morphology, etc., a feature vector with distinguishing properties in the road point cloud data is identified. Specifically, the local plane of the road point cloud data is determined according to the principal component analysis method, the local point cloud data is translated to a coordinate system with the mean point as the origin, and the covariance matrix of the point cloud is calculated. The covariance matrix is subjected to eigenvalue decomposition to obtain three eigenvalues and corresponding eigenvectors. The direction corresponding to the eigenvector with the smallest eigenvalue is the direction of minimum change in the point cloud data, that is, the normal. A histogram is generated by calculating the relative differences between the normals in the neighborhood, and the point cloud feature points are determined based on the histogram corresponding to the point cloud data. Furthermore, image feature points are determined by calculating a histogram of the neighborhood of pixels in the road image data. Preferably, a feature matching algorithm and the feature points are used to calculate the rotation matrix and translation vector of the visual camera relative to the lidar. Coordinate transformation is then performed based on the rotation matrix and translation vector to achieve data alignment. The feature matching algorithm may include a direct linear transformation method, an optimal PnP algorithm, or the like.
[0064] Step S22, marking the vehicle object of the fused feature image according to the vehicle driving information and the road point cloud data, so that the vehicle object of the fused feature image includes: driving state information;
[0065] Specifically, a mapping relationship is determined based on the changes in the vehicle driving information and the point cloud in the road point cloud data over time. Since the road point cloud data is already fused with feature images, the vehicle object can be marked according to the mapping relationship, and the marked content can be complete vehicle driving information or partial vehicle driving information.
[0066] Step S23: determining a fused feature video according to the driving state information of the vehicle object in the fused feature image.
[0067] In this embodiment, a plurality of images that change with time are generated according to the driving state information of the vehicle object in the fused feature image, and a fused feature video is determined based on the plurality of images that change with time.
[0068] In this embodiment, the accuracy of the detected road data can be improved through detection data in multiple dimensions.
[0069] Further, refer to Figure 4 The step of generating a fused feature image based on the road image data and the road point cloud data includes:
[0070] Step S211, determining point cloud feature points based on the road point cloud data, and determining image feature points based on the road image data;
[0071] Generally speaking, feature points need to be extracted from both the road point cloud data and the road image data. For road image data, the types of features that can be extracted include corner features, edge features, and texture features. For road point cloud data, the types of features that can be extracted include local geometric features, global geometric features, curvature, and normal features.
[0072] Step S212, matching the point cloud feature points with the image feature points to obtain a matching result;
[0073] The matching result here specifically refers to the rotation matrix and translation vector of the visual camera relative to the lidar.
[0074] Step S213 : generating a fused feature image from the road point cloud data and the road image data according to the matching result, wherein each pixel of the fused feature image includes corresponding color information and distance information.
[0075] The road point cloud data can be projected onto the road image data according to the rotation matrix and translation vector, thereby obtaining a fused feature image. The color information here means that each pixel includes three RGB color channels. In addition, each pixel also includes a distance channel for storing the distance information.
[0076] In this embodiment, point cloud feature points are determined based on the road point cloud data, and image feature points are determined based on the road image data. Since the matching result here is specifically the rotation matrix and translation vector of the visual camera relative to the lidar, the road point cloud data can be projected into the road image data based on the rotation matrix and translation vector, thereby obtaining an accurate fused feature image and improving the information content of the monitored data.
[0077] Furthermore, based on any of the above embodiments, a third embodiment of a traffic control method based on multimodal perception is proposed. In this embodiment, the step of receiving vehicle driving information by the communication device includes:
[0078] The vehicle driving information is received at a first preset frequency.
[0079] The step of marking the vehicle object of the fused feature image according to the vehicle driving information and the road point cloud data comprises:
[0080] When the vehicle object in the fused feature image does not include vehicle driving information, marking the vehicle object in the fused feature image according to the vehicle driving information;
[0081] When the vehicle object of the fused feature image includes vehicle driving information, the mark is updated at a first preset frequency so that the driving state information is updated at the first preset frequency.
[0082] Since the amount of measured driving information is relatively low, the bandwidth required during the transmission process is lower than the bandwidth for transmitting image data. Therefore, different acquisition frequencies can be set for different data. In this embodiment, the vehicle driving information is collected according to a first preset frequency, and the frequency of collection of the visual camera and the lidar can be a second preset frequency, and the first preset frequency is greater than the second preset frequency. It is also determined whether the vehicle object of the current fused feature image includes vehicle driving information. For vehicle objects that already have vehicle driving information, the vehicle driving information is updated based on the collected data. It should be noted that since it is necessary to use the fused feature image to generate a fused feature video, the vehicle driving information in the virtual image generated based on the fused feature image is actually updated during the update process.
[0083] In this embodiment, the vehicle driving information is received at a first preset frequency. When the vehicle object in the fused feature image does not include vehicle driving information, the vehicle object in the fused feature image is marked according to the vehicle driving information. When the vehicle object in the fused feature image includes vehicle driving information, the mark is updated at a first preset frequency, so that the driving state information is updated at the first preset frequency, thereby ensuring the accuracy of the subsequently generated fused feature video.
[0084] Furthermore, the driving state information includes driving speed, driving direction, and driving acceleration, and the step of determining the fused feature video based on the driving state information of the vehicle object in the fused feature image includes:
[0085] Determining a virtual position corresponding to the vehicle object in a plurality of simulated fused feature images based on the vehicle object's speed, direction, and acceleration in the fused feature image, wherein the simulated fused feature image is a virtual image generated based on the fused feature image after a current moment;
[0086] The fused feature video is generated according to the fused feature image and the multiple simulated fused feature images.
[0087] In this embodiment, the first moment corresponding to the generated virtual image is determined, an interval duration is determined based on the current moment and the first moment, and the position of the vehicle object in the virtual image corresponding to the first moment is calculated based on the interval duration, the vehicle object's driving speed, driving direction, driving acceleration, and the conversion coefficient between the image and the actual vehicle position. When the driving state information is updated, the new driving state information is marked in the corresponding virtual image, and the fused feature video is generated based on the fused feature image and the multiple simulated fused feature images. Preferably, when road image data and road point cloud data are collected at a second preset frequency, a new fused feature image is generated to replace the virtual image. The generated fused feature video thus includes the fused feature images and virtual images corresponding to each moment, and at least one virtual image is included between the fused feature images at each moment.
[0088] Furthermore, based on any of the above embodiments, a fourth embodiment of the traffic control method based on multimodal perception of the present invention is proposed. In this embodiment, the step of determining the scene recognition result based on the fused feature video and the traffic scene recognition model includes:
[0089] Identifying object feature data of each target object in the fused feature video to obtain a plurality of object feature data;
[0090] The plurality of object feature data are input into the traffic scene recognition model, a road scene index is output, and a scene recognition result is determined according to the road scene index.
[0091] In this embodiment, before the step of inputting the multiple object feature data into the traffic scene recognition model, the step also includes: obtaining historical road data collected before the current moment and corresponding marked scene indicators, and extracting historical feature data corresponding to the traffic scene recognition model based on the historical road data, training a deep recognition model based on the historical feature data and scene indicators to obtain the traffic scene recognition model.
[0092] Furthermore, the step of controlling the traffic indication signal according to the scene recognition result includes:
[0093] determining traffic control data according to the scene recognition result and a first corresponding relationship, wherein the first corresponding relationship is a corresponding relationship between the scene recognition result and the traffic control data;
[0094] The traffic indication signal is controlled according to the traffic control data.
[0095] In this embodiment, the scene recognition results include but are not limited to standard scene modes such as commuting peak, low traffic at night, bad weather, and emergencies. According to the scene recognition results, a corresponding response plan is selected and the traffic signal is controlled according to the response plan.
[0096] In addition, an embodiment of the present invention further provides a traffic control device based on multimodal perception, wherein the traffic control device based on multimodal perception includes:
[0097] An acquisition module is used to obtain road image data collected by a visual camera, road point cloud data collected by a lidar, and vehicle driving information received by a communication device;
[0098] A fusion module, configured to generate a fusion feature video based on the road image data, the road point cloud data, and the vehicle driving information;
[0099] A recognition module, configured to determine a scene recognition result based on the fused feature video and the traffic scene recognition model;
[0100] A control module is used to control the traffic indication signal according to the scene recognition result.
[0101] Furthermore, the traffic control device based on multimodal perception can implement the steps of any embodiment of the above-mentioned traffic control method based on multimodal perception.
[0102] In addition, an embodiment of the present invention also proposes a traffic control device based on multimodal perception, which includes: a memory, a processor, and a traffic control program based on multimodal perception stored on the memory and executable on the processor, wherein the traffic control program based on multimodal perception is configured to implement the steps of the traffic control method based on multimodal perception as described in any one of the above items.
[0103] In addition, an embodiment of the present invention also proposes a storage medium, on which a traffic control program based on multimodal perception is stored. When the traffic control program based on multimodal perception is executed by a processor, the steps of any of the above-mentioned traffic control methods based on multimodal perception are implemented.
[0104] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.
[0105] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.
[0106] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.
[0107] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A traffic control method based on multimodal perception, characterized in that: The traffic control method based on multimodal perception includes the following steps: Obtain road image data collected by visual cameras, road point cloud data collected by lidar, and vehicle driving information received by communication equipment; generating a fusion feature video according to the road image data, the road point cloud data, and the vehicle driving information; Determining a scene recognition result based on the fused feature video and the traffic scene recognition model; The traffic indication signal is controlled according to the scene recognition result.
2. The traffic control method based on multimodal perception according to claim 1, characterized in that: The step of generating a fusion feature video based on the road image data, the road point cloud data and the vehicle driving information includes: generating a fused feature image according to the road image data and the road point cloud data; Marking the vehicle object of the fused feature image according to the vehicle driving information and the road point cloud data, so that the vehicle object of the fused feature image includes: driving state information; A fused feature video is determined according to the driving state information of the vehicle object in the fused feature image.
3. The traffic control method based on multimodal perception according to claim 2, characterized in that: The step of generating a fused feature image based on the road image data and the road point cloud data comprises: determining point cloud feature points based on the road point cloud data, and determining image feature points based on the road image data; Matching the point cloud feature points with the image feature points to obtain a matching result; The road point cloud data and the road image data are combined to generate a fused feature image according to the matching result, wherein each pixel of the fused feature image includes corresponding color information and distance information.
4. The traffic control method based on multimodal perception as described in claim 2 is characterized in that: The step of the communication device receiving the vehicle driving information includes: The vehicle driving information is received at a first preset frequency. The step of marking the vehicle object of the fused feature image according to the vehicle driving information and the road point cloud data comprises: When the vehicle object in the fused feature image does not include vehicle driving information, marking the vehicle object in the fused feature image according to the vehicle driving information; When the vehicle object of the fused feature image includes vehicle driving information, the mark is updated at a first preset frequency so that the driving state information is updated at the first preset frequency.
5. The traffic control method based on multimodal perception according to claim 2, characterized in that: The driving state information includes: driving speed, driving direction and driving acceleration. The step of determining the fused feature video according to the driving state information of the vehicle object in the fused feature image includes: Determining a virtual position corresponding to the vehicle object in a plurality of simulated fused feature images based on the vehicle object's speed, direction, and acceleration in the fused feature image, wherein the simulated fused feature image is a virtual image generated based on the fused feature image after a current moment; The fused feature video is generated according to the fused feature image and the multiple simulated fused feature images.
6. The traffic control method based on multimodal perception according to any one of claims 1 to 5, characterized in that: The step of determining the scene recognition result according to the fusion feature video and the traffic scene recognition model comprises: Identifying object feature data of each target object in the fused feature video to obtain a plurality of object feature data; The plurality of object feature data are input into the traffic scene recognition model, a road scene index is output, and a scene recognition result is determined according to the road scene index.
7. The traffic control method based on multimodal perception according to any one of claims 1 to 5, characterized in that: The step of controlling the traffic indication signal according to the scene recognition result includes: determining traffic control data according to the scene recognition result and a first corresponding relationship, wherein the first corresponding relationship is a corresponding relationship between the scene recognition result and the traffic control data; The traffic indication signal is controlled according to the traffic control data.
8. A traffic control device based on multimodal perception, characterized in that: The traffic control device based on multimodal perception includes: An acquisition module is used to obtain road image data collected by a visual camera, road point cloud data collected by a lidar, and vehicle driving information received by a communication device; A fusion module, configured to generate a fusion feature video based on the road image data, the road point cloud data, and the vehicle driving information; A recognition module, configured to determine a scene recognition result based on the fused feature video and the traffic scene recognition model; A control module is used to control the traffic indication signal according to the scene recognition result.
9. A traffic control device based on multimodal perception, characterized in that: The device includes: a memory, a processor, and a traffic control program based on multimodal perception stored in the memory and executable on the processor, wherein the traffic control program based on multimodal perception is configured to implement the steps of the traffic control method based on multimodal perception as described in any one of claims 1 to 7.
10. A storage medium, characterized in that: The storage medium stores a traffic control program based on multimodal perception, and when the traffic control program based on multimodal perception is executed by the processor, the steps of the traffic control method based on multimodal perception as described in any one of claims 1 to 7 are implemented.