A roadside parking scene analysis method and device based on multimodal perception
By building a semantic segmentation model based on SegFormer, combining the multi-directional feature fusion module and information exchange module, the problem of poor flexibility in the use of multimodal data in the existing technology is solved, and efficient analysis and traffic management support for roadside parking scenarios are realized.
Patent Information
- Application Number
- CN202411785786.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-06
- Publication Date
- 2025-04-08
AI Technical Summary
In the prior art, the flexibility of using multimodal data is poor, making it difficult to achieve large-scale application in actual roadside scenarios.
By acquiring multimodal data (including RGB images, depth maps, thermal imaging images) and pre-processing, a semantic segmentation model based on SegFormer is built, combining multi-directional feature fusion module and information exchange module to realize feature integration and cross-modal information exchange of different modal data.
It realizes flexible fusion of multimodal features, adapts to the combination of different modal inputs, improves the accuracy and efficiency of road-side parking scenario analysis, and supports traffic management.
Smart Images
Figure CN119785023A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a roadside parking scene analysis method and device based on multimodal perception. Background Art
[0002] In recent years, with the development of social economy, people's demand for convenience in daily travel has continued to increase, and the number of cars has continued to increase. However, the increase in the number of parking spaces is far less than the increase in the number of vehicles, resulting in parking difficulties and parking chaos, which has become one of the main problems that need to be solved in the development of urban transportation in my country. At present, intelligent parking charges and management are mainly achieved by installing high-position video cameras on the roadside for data collection, and then using visual algorithms for license plate detection, license plate recognition, target detection, semantic segmentation and other algorithms. In the prior art, for the use of multimodal data, a feature fusion module of specific modal data is usually constructed, which limits the flexibility of different modalities as inputs and increases the model training parameters. It is difficult to flexibly and conveniently realize large-scale applications in actual roadside scenarios. Summary of the invention
[0003] In view of this, an embodiment of the present invention provides a roadside parking scene analysis method and device based on multimodal perception, which solves the problems of poor flexibility and difficult application in the prior art.
[0004] According to a first aspect, an embodiment of the present invention provides a roadside parking scene analysis method based on multimodal perception, comprising:
[0005] Obtain multimodal data and historical data of the object to be detected;
[0006] Preprocessing the historical data to determine the preprocessed target data;
[0007] Constructing a parsing model, training the parsing model using the target data, and determining a target parsing model;
[0008] The multimodal data is recognized using the target parsing model to determine a recognition result.
[0009] In combination with the first aspect, in a first implementation of the first aspect, the acquiring of multimodal data and historical data of the object to be detected includes:
[0010] Installing data acquisition equipment at various traffic locations, the data acquisition equipment including RGB-D cameras and thermal imagers;
[0011] The data acquisition device is used to acquire the multimodal data and the historical data of different time periods and different scenes, and the multimodal data and the historical data include RGB images, depth maps, and thermal imaging images.
[0012] In combination with the first implementation of the first aspect, in the second implementation of the first aspect, preprocessing the historical data to determine the preprocessed target data includes:
[0013] Performing data alignment on the RGB image, the depth map, and the thermal image corresponding to the historical data to determine alignment data, wherein the data alignment includes time dimension alignment and space dimension alignment;
[0014] The aligned data are semantically segmented and annotated using a polygon tool to determine target data, wherein the categories of the semantic segmentation and annotation include: pedestrians, vehicles, lane lines, traffic signs, and green plants.
[0015] In combination with the first aspect, in a third implementation of the first aspect, constructing the analytical model includes:
[0016] Constructing the feature extraction module and encoding module of the parsing model based on the semantic segmentation model of SegFormer;
[0017] Construct a multi-directional feature fusion module and an information exchange module of the analytical model.
[0018] In combination with the third implementation manner of the first aspect, in the fourth implementation manner of the first aspect, the SegFormer-based semantic segmentation model constructs a feature extraction module and an encoding module of the parsing model, including:
[0019] The feature extraction module is determined based on the encoder of the hierarchical transformer to generate high-resolution coarse features and low-resolution fine features;
[0020] The encoding module is determined based on a multi-layer perceptron decoder, and decoder features at different levels are fused to generate a final semantic segmentation mask.
[0021] In combination with the third implementation manner of the first aspect, in the fifth implementation manner of the first aspect, the multi-directional feature fusion module and the information exchange module for constructing the analytical model include:
[0022] The multi-directional feature fusion module includes: downsampling, nonlinear activation, and upsampling;
[0023] The downsampling is expressed by the following formula:
[0024] X down =W down x+b down
[0025] Where x represents the input feature vector, x∈R d , W down represents the weight matrix, Wdown ∈R dxr , b down represents the bias vector, b down ∈R dxr , r represents the downsampling parameter;
[0026] Specifically, the nonlinear activation part is specifically expressed as:
[0027] X mid = Dropout(GELU(W mid x down +b mid ))
[0028] Among them, W mid represents the weight matrix, W mid ∈R rxr , b mid represents the bias vector, b mid ∈R rxr ; Dropout represents the Dropout() function, which is used to randomly drop neurons to prevent overfitting of the model, and GELU represents the GELU() activation function;
[0029] Specifically, the upsampling part restores the features to their original size, which is specifically expressed as:
[0030] X up =W up x mid +b up
[0031] Among them, W up represents the weight matrix, W up ∈R rxd , b up represents the bias vector, b up ∈R rxd ;
[0032] The information exchange module exchanges the analytical models of different modal data for information exchange between cross-modal features.
[0033] In combination with the first aspect, in a sixth implementation of the first aspect, the constructing of the parsing model, training the parsing model using the target data, and determining the target parsing model includes:
[0034] Construct loss function;
[0035] The loss function and the target data are used to update the parameters of the analytical model to generate a target analytical model.
[0036] In combination with the sixth implementation of the first aspect, in the seventh implementation of the first aspect, the loss function is:
[0037]
[0038] Among them, C represents the number of target segmentation categories, p c Indicates the probability that the predicted sample belongs to class c, y c Represents the true label category distribution.
[0039] The roadside parking scene analysis method based on multimodal perception provided by an embodiment of the present invention uses a pre-trained large model as an encoder and a feature fuser, which is conducive to multimodal feature fusion and can adapt to the combination of different modal inputs. In the feature encoding process, by utilizing a multi-directional feature fusion module, the information exchange between cross-modal features is enhanced to achieve feature integration of different modal data; visible light modal data, RGB-D depth camera modal data, and thermal infrared sensor data are used to achieve data fusion between different modal data. For roadside parking scene analysis, semantic segmentation is used to achieve it. Considering the requirements for model efficiency, robustness, accuracy, etc. in roadside scenes, a semantic segmentation model structure based on SegFormer is used to achieve scene analysis tasks based on multimodal perception data, and the category and position output of different targets in roadside scenes is achieved, which is helpful for traffic management of roadside parking scenes.
[0040] According to the second aspect, the roadside parking scene analysis device based on multimodal perception provided by an embodiment of the present invention includes:
[0041] A first processing module, used to obtain multimodal data and historical data of the object to be detected;
[0042] A second processing module is used to preprocess the historical data and determine the preprocessed target data;
[0043] A third processing module is used to construct a parsing model, train the parsing model using the target data, and determine a target parsing model;
[0044] The fourth processing module is used to use the target parsing model to identify the multimodal data and determine the recognition result.
[0045] According to the third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the roadside parking scene analysis method based on multimodal perception described in the first aspect or any one embodiment of the first aspect by executing the computer instructions.
[0046] According to the fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the roadside parking scene analysis method based on multimodal perception described in the first aspect or any one embodiment of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0048] Figure 1 is a flow chart of a roadside parking scene analysis method based on multimodal perception according to an embodiment of the present invention;
[0049] Figure 2 is a schematic diagram of a roadside parking scene analysis device based on multimodal perception according to an embodiment of the present invention;
[0050] Figure 3 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0052] In recent years, with the development of social economy, people's demand for convenience in daily travel has continued to increase, and the number of cars has continued to increase. However, the increase in the number of parking spaces is far less than the increase in the number of vehicles, resulting in parking difficulties and disorderly parking, which has become one of the main problems that need to be solved in the development of urban transportation in my country. At present, traffic control departments in various places are also actively carrying out and promoting solutions to the problems of parking difficulties and disorderly parking for the majority of car owners. Especially in recent years, with the development of intelligent technology, in order to effectively alleviate this problem, traffic control departments in various places have tried and built various intelligent parking management technologies. Among them, by setting up parking spaces on the roadside, using intelligent sensing equipment to collect data such as video images, and processing and analyzing the data through computer vision algorithms, intelligent parking management is currently a mainstream solution. At present, it is mainly achieved by installing high-position video cameras on the roadside for data collection, and then using visual algorithms for license plate detection, license plate recognition, target detection, semantic segmentation and other algorithms to realize intelligent parking charges and management.
[0053] However, compared with other spectra, the visible light imaging range is significantly narrower, and it is only effective in environments with good lighting and high visibility. It may fail at night, in special weather such as rain and fog, or in environments with obstructions. As a result, the understanding and analysis of traffic scenes in special environments cannot be achieved. Therefore, many schemes currently propose to use multi-modal data of different modes of multiple sensor devices including visible light cameras, infrared cameras, lidar, millimeter-wave radar, etc. to analyze roadside parking scenes. Data of different modes can play different advantages in different environments and weather conditions. The fusion of multiple modal data can provide more data information support for the analysis of complex scenes, which plays an important role in improving the accuracy of traffic scene analysis. However, the current use of multi-modal data usually constructs a feature fusion module of specific modal data, which limits the flexibility of different modalities as input and increases model training parameters. It is difficult to flexibly and conveniently realize large-scale applications in actual roadside scenes.
[0054] In this embodiment, a roadside parking scene analysis method based on multimodal perception is provided, which can be used in electronic devices such as computers, mobile phones, tablet computers, etc. Figure 1 is a flow chart of a roadside parking scene analysis method based on multimodal perception according to an embodiment of the present invention, such as Figure 1 As shown, the process includes the following steps:
[0055] S11, obtaining multimodal data and historical data of the object to be detected. After determining the location to be detected, such as a roadside parking lot, different intersections, different locations, etc., the multimodal data and historical data of the object to be detected are obtained using existing technology.
[0056] S12, preprocessing the historical data and determining the preprocessed target data.
[0057] Specifically, in order to improve data processability and data accuracy, it is necessary to align the data in the time dimension of perception devices of different spectra to ensure that the acquired data of multiple modal devices are perception data recorded in the same scene at the same time, and determine the target data.
[0058] S13, constructing a parsing model, training the parsing model using target data, and determining a target parsing model.
[0059] S14, using the target parsing model to identify the multimodal data and determine the identification result.
[0060] The roadside parking scene analysis method based on multimodal perception provided in this embodiment uses the pre-trained large model as an encoder and feature fuser, which is conducive to multimodal feature fusion and can adapt to the combination of different modal inputs. In the feature encoding process, by utilizing a multi-directional feature fusion module, the information exchange between cross-modal features is enhanced to achieve feature integration of different modal data; visible light modal data, RGB-D depth camera modal data, and thermal infrared sensor data are used to achieve data fusion between different modal data. For roadside parking scene analysis, semantic segmentation is used to achieve it. Considering the requirements for model efficiency, robustness, accuracy, etc. in roadside scenes, a semantic segmentation model structure based on SegFormer is used to achieve scene analysis tasks based on multimodal perception data, and to achieve the output of categories and positions of different targets in roadside scenes, which is helpful for traffic management of roadside parking scenes.
[0061] In another embodiment, a roadside parking scene analysis method based on multimodal perception is also provided, comprising the following steps:
[0062] S21, obtaining multimodal data and historical data of the object to be detected.
[0063] Specifically, the above step S21 also includes the following steps:
[0064] S211, installing data acquisition equipment at various traffic locations, the data acquisition equipment including RGB-D cameras and thermal imagers; specifically, installing multimodal perception data acquisition equipment at different traffic locations, including RGB-D cameras and thermal imagers, etc., to obtain multimodal perception data.
[0065] S212, using data acquisition equipment to obtain multimodal data and historical data of different time periods and different scenes, the multimodal data and historical data including RGB images, depth maps, and thermal images.
[0066] In this implementation, different traffic locations include major urban roads, intersections, roadside parking areas, and parking lots at the entrances of key areas such as schools and hospitals, etc., covering the dynamic change data of vehicles, pedestrians and other targets in the above-mentioned different scenarios in different time periods; specifically, because different sensors may have different perception ranges, resolutions and sensing capabilities, etc., therefore, in order to subsequently fuse the multimodal data, it is necessary to first align the acquired multimodal perception data including RGB images, depth maps, and thermal images.
[0067] S22, preprocessing the historical data to determine the preprocessed target data.
[0068] In this embodiment, the above step S22 specifically further includes:
[0069] S221, align the RGB images, depth maps and thermal images corresponding to the historical data to determine the aligned data. The data alignment includes time dimension alignment and space dimension alignment. The spatial dimension data alignment is to unify the images of data acquisition of different spectral perceptions into the same coordinate system; the time dimension data alignment is that the data acquisition frequencies of sensors of different spectra may be different, so it is necessary to align the time dimension data of perception devices of different spectra to ensure that the acquisition data of multiple modal devices are the perception data recorded in the same scene at the same time.
[0070] S222, using polygon tools to semantically segment and annotate the aligned data, and determine the target data. The semantic segmentation and annotation categories include: pedestrians, vehicles, lane lines, traffic signs, and green plants. The annotation categories include, but are not limited to, target annotations of categories such as pedestrians, vehicles, non-motor vehicles, lane lines, traffic signs, and green plants. Vehicles specifically include various categories such as cars, buses, and taxis.
[0071] S23, constructing a parsing model, training the parsing model using target data, and determining a target parsing model.
[0072] In this embodiment, the above step S23 specifically further includes:
[0073] S231, based on the semantic segmentation model of SegFormer, builds the feature extraction module and encoding module of the parsing model; combines lightweight multi-layer perception to achieve simple and efficient semantic segmentation capabilities; determines the feature extraction module based on the encoder of the hierarchical transformer to generate high-resolution coarse features and low-resolution fine features; determines the encoding module based on the multi-layer perceptron decoder, fuses the decoder features at different levels, and generates the final semantic segmentation mask.
[0074] Specifically, the basic process of the SegFormer model is to first divide the HxWx3 image into 4*4 image blocks. Using smaller image blocks is beneficial for dense prediction tasks such as semantic segmentation. Then the image blocks are fed into the hierarchical transformer encoder as input to obtain multi-level features with input image resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32. Finally, the multi-level features are fed into the multi-layer perceptron module to output semantic segmentation masks at the feature map size of H / 4xW / 4xN, where N represents the number of semantic segmentation categories. Based on the advantages of the SegFormer model, the feature extraction and encoding model part of the SegFormer pre-trained on large-scale multimodal data is used as the feature encoding and fusion module of the multimodal scene parsing model to achieve feature extraction and encoding of each modality data.
[0075] S232, constructing a multi-directional feature fusion module and an information exchange module of the parsing model;
[0076] The multi-directional feature fusion module includes: downsampling, nonlinear activation, and upsampling;
[0077] Downsampling is expressed by the following formula:
[0078] X down =W down x+b down
[0079] Where x represents the input feature vector, x∈R d , W down represents the weight matrix, W down ∈R dxr , b down represents the bias vector, b down ∈R dxr , r represents the downsampling parameter;
[0080] Specifically, the nonlinear activation part is expressed as:
[0081] X mid = Dropout(GELU(W mid x down +b mid ))
[0082] Among them, W mid represents the weight matrix, W mid ∈R rxr , b mid represents the bias vector, b mid ∈R rxr; Dropout represents the Dropout() function, which is used to randomly drop neurons to prevent overfitting of the model, and GELU represents the GELU() activation function;
[0083] Specifically, the upsampling part restores the features to their original size, which is specifically expressed as:
[0084] X up =W up x mid +b up
[0085] Among them, W up represents the weight matrix, W up ∈R rxd , b up represents the bias vector, b up ∈R rxd ;
[0086] The information exchange module exchanges the analytical models of different modal data for information exchange between cross-modal features.
[0087] S233, construct loss function;
[0088]
[0089] Among them, C represents the number of target segmentation categories, p c Indicates the probability that the predicted sample belongs to class c, y c Represents the true label category distribution.
[0090] S234, using the loss function and the target data, update the parameters of the parsing model to generate a target parsing model.
[0091] S24, using the target parsing model to identify the multimodal data and determine the recognition result. Based on the multimodal perception model, different modal data are used to predict the semantic segmentation task and realize roadside scene analysis; specifically, different modal data include visible light data, RGB-D data, thermal infrared data, etc., semantic segmentation means to classify the target in the image pixel by pixel to obtain the mask information and category information of the target; multiple categories mainly include green plants, parking lines, lane lines, manhole covers, zebra crossings, iron grates and other categories. By using the semantic segmentation algorithm, the positions of targets such as green plants and parking lines can be obtained more accurately, realizing the roadside scene analysis task.
[0092] A roadside parking scene analysis method based on multimodal perception is provided in this embodiment. The pre-trained large model is used as an encoder and feature fuser, which is conducive to multimodal feature fusion and can adapt to the combination of different modal inputs. In the feature encoding process, the information exchange between cross-modal features is enhanced by utilizing a multi-directional feature fusion module to achieve feature integration of different modal data. Visible light modal data, RGB-D depth camera modal data, and thermal infrared sensor data are used to achieve data fusion between different modal data. For roadside parking scene analysis, semantic segmentation is used. Considering the requirements for model efficiency, robustness, accuracy, etc. in roadside scenes, a semantic segmentation model structure based on SegFormer is used to implement scene analysis tasks based on multimodal perception data, and to achieve the output of categories and positions of different targets in roadside scenes, which is helpful for traffic management of roadside parking scenes.
[0093] In this embodiment, a roadside parking scene analysis device based on multimodal perception is also provided. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0094] The present invention discloses a roadside parking scene analysis device based on multimodal perception, such as Figure 2 As shown, including:
[0095] The first processing module 01 is used to obtain multimodal data and historical data of the object to be detected; the detailed operation process is detailed in the corresponding steps in the above method embodiment.
[0096] The second processing module 02 is used to pre-process the historical data and determine the pre-processed target data; the detailed operation process is detailed in the corresponding steps in the above method embodiment.
[0097] The third processing module 03 is used to construct a parsing model, train the parsing model using target data, and determine a target parsing model; the detailed operation process is detailed in the corresponding steps in the above method embodiment.
[0098] The fourth processing module 04 is used to use the target parsing model to identify the multimodal data and determine the identification result; the detailed operation process is detailed in the corresponding steps in the above method embodiment.
[0099] The present invention also provides an electronic device. Figure 3 , Figure 3 is a schematic diagram of the structure of an electronic device provided by an optional embodiment of the present invention, such as Figure 3As shown, the electronic device may include: at least one processor 601, such as a CPU (Central Processing Unit), at least one communication interface 603, a memory 604, and at least one communication bus 602. The communication bus 602 is used to realize the connection and communication between these components. The communication interface 603 may include a display screen (Display) and a keyboard (Keyboard), and the optional communication interface 603 may also include a standard wired interface and a wireless interface. The memory 604 may be a high-speed RAM memory (Random Access Memory) or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 604 may optionally be at least one storage device located away from the aforementioned processor 601. The processor 601 may be combined with Figure 3 In the described device, the memory 604 stores an application program, and the processor 601 calls the program code stored in the memory 604 to execute any of the above method steps.
[0100] The communication bus 602 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The communication bus 602 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0101] Among them, the memory 604 may include a volatile memory (English: volatile memory), such as a random access memory (English: random-access memory, abbreviated: RAM); the memory may also include a non-volatile memory (English: non-volatile memory), such as a flash memory (English: flash memory), a hard disk drive (English: hard disk drive, abbreviated: HDD) or a solid-state drive (English: solid-state drive, abbreviated: SSD); the memory 604 may also include a combination of the above types of memory.
[0102] The processor 601 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and a NP.
[0103] The processor 601 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.
[0104] Optionally, the memory 604 is also used to store program instructions. The processor 601 can call the program instructions to implement the method shown in the embodiment of the present application.
[0105] The embodiment of the present invention further provides a non-transitory computer storage medium, which stores computer executable instructions, and the computer executable instructions can execute the method in any of the above method embodiments. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.
[0106] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A roadside parking scene analysis method based on multimodal perception, characterized in that: include: Obtain multimodal data and historical data of the object to be detected; Preprocessing the historical data to determine the preprocessed target data; Constructing a parsing model, training the parsing model using the target data, and determining a target parsing model; The multimodal data is recognized using the target parsing model to determine a recognition result.
2. The method according to claim 1, characterized in that The step of obtaining multimodal data and historical data of the object to be detected includes: Installing data acquisition equipment at various traffic locations, the data acquisition equipment including RGB-D cameras and thermal imagers; The data acquisition device is used to acquire the multimodal data and the historical data of different time periods and different scenes, and the multimodal data and the historical data include RGB images, depth maps, and thermal imaging images.
3. The method according to claim 2, characterized in that The preprocessing of the historical data to determine the preprocessed target data includes: Performing data alignment on the RGB image, the depth map, and the thermal image corresponding to the historical data to determine alignment data, wherein the data alignment includes time dimension alignment and space dimension alignment; The aligned data are semantically segmented and annotated using a polygon tool to determine target data, wherein the categories of the semantic segmentation and annotation include: pedestrians, vehicles, lane lines, traffic signs, and green plants.
4. The method according to claim 1, characterized in that: The constructing of the parsing model comprises: Constructing the feature extraction module and encoding module of the parsing model based on the semantic segmentation model of SegFormer; Construct a multi-directional feature fusion module and an information exchange module of the analytical model.
5. The method according to claim 4, characterized in that The SegFormer-based semantic segmentation model constructs a feature extraction module and an encoding module of the parsing model, including: The feature extraction module is determined based on the encoder of the hierarchical transformer to generate high-resolution coarse features and low-resolution fine features; The encoding module is determined based on a multi-layer perceptron decoder, and decoder features at different levels are fused to generate a final semantic segmentation mask.
6. The method according to claim 4, characterized in that The multi-directional feature fusion module and information exchange module for constructing the analytical model include: The multi-directional feature fusion module includes: downsampling, nonlinear activation, and upsampling; The downsampling is expressed by the following formula: X down =W down x+b down Where x represents the input feature vector, x∈R d , W down represents the weight matrix, W down ∈R dxr , b down represents the bias vector, b down ∈R dxr , r represents the downsampling parameter; Specifically, the nonlinear activation part is specifically expressed as: X mid =Dropout(GELU(W mid x down+ b mid )) Among them, W mid represents the weight matrix, W mid ∈R rxr , b mid represents the bias vector, b mid ∈R rxr ; Dropout represents the Dropout() function, which is used to randomly drop neurons to prevent overfitting of the model, and GELU represents the GELU() activation function; Specifically, the upsampling part restores the features to their original size, which is specifically expressed as: X up =W up x mid +b up Among them, W up represents the weight matrix, W up ∈R rxd , b up represents the bias vector, b up ∈R rxd ; The information exchange module exchanges information of the analytical models of different modal data for information exchange between cross-modal features.
7. The method according to claim 1, characterized in that The constructing of the parsing model, training the parsing model using the target data, and determining the target parsing model includes: Construct loss function; The loss function and the target data are used to update the parameters of the analytical model to generate a target analytical model.
8. The method according to claim 7, characterized in that The loss function is: Among them, C represents the number of target segmentation categories, p c Indicates the probability that the predicted sample belongs to class c, y c Represents the true label category distribution.
9. A roadside parking scene analysis device based on multimodal perception, characterized in that: include: A first processing module, used to obtain multimodal data and historical data of the object to be detected; A second processing module is used to preprocess the historical data and determine the preprocessed target data; A third processing module is used to construct a parsing model, train the parsing model using the target data, and determine a target parsing model; The fourth processing module is used to use the target parsing model to identify the multimodal data and determine the recognition result.