An AI vision-based vehicle illegal passenger detection and identification method

Abstract

This invention discloses a method for detecting and identifying illegal passenger transport in vehicles based on AI vision, comprising the following steps: Step 1: acquiring raw visual data and related information of vehicles traveling within a target road area; Step 2: performing visual preprocessing and cross-view alignment; Step 3: performing vehicle subject detection and occupant perception through an AI vision joint perception network; Step 4: performing passenger-carrying semantic state modeling through an improved Vision Mamba model to obtain a passenger-carrying determination vector for the target vehicle; Step 5: performing violation rule constraints and visual reasoning fusion on the passenger-carrying determination vector for the target vehicle to obtain the illegal passenger transport identification result; Step 6: performing multi-evidence consistency verification based on the illegal passenger transport identification result to generate a standard violation evidence collection result set. This invention improves the accuracy and reliability of detecting and identifying illegal passenger transport in vehicles by using an AI vision joint perception network and an improved Vision Mamba model.

Description

Technical Field

[0001] This invention relates to the field of visual recognition and intelligent traffic violation analysis technology, and in particular to a method for detecting and identifying illegal passenger transport by vehicles based on AI vision. Background Technology

[0002] With the increasing demand for intelligent road traffic monitoring and automatic identification of violations based on video surveillance, visual perception of passenger carrying status in passing vehicles and technologies for determining illegal passenger carrying have received widespread attention. Existing road monitoring or traffic enforcement systems mainly rely on single-view vehicle image analysis, manual visual interpretation, or simple target detection methods for identifying illegal passenger carrying. However, these methods generally suffer from the following problems in practical applications:

[0003] The acquired images of passing vehicles are susceptible to changes in lighting, occlusion, window reflections, shooting angle shifts, and vehicle motion blur, resulting in poor visibility of occupants, weak window area features, and unstable occupant counts, making it difficult to accurately obtain the true passenger load status inside the vehicle. Differences in acquisition time, location, and equipment source between multi-view vehicle images often prevent existing cross-view association and alignment methods from achieving accurate matching and unified representation of the same target vehicle from different perspectives, leading to fragmented information about the vehicle's interior and exterior, and an inability to effectively integrate vehicle structure information with occupant distribution information. Given the characteristics of illegal passenger transport in complex traffic scenarios, such as irregular occupant distribution, severe occlusion, complex spatial changes, and diverse violation judgment rules, traditional vehicle detection, occupant counting, or rule threshold judgment methods suffer from insufficient semantic modeling capabilities, limited visual reasoning abilities, and inadequate evidence consistency verification. This results in low accuracy and high false positive rates in identifying illegal passenger transport, failing to meet the requirements of accuracy, result stability, and evidence reliability in actual traffic enforcement scenarios.

[0004] Therefore, how to provide a method for detecting and identifying illegal passenger transport by vehicles based on AI vision is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a method for detecting and identifying illegal passenger transport in vehicles based on AI vision. This invention fully utilizes multi-view visual perception of vehicles, AI vision joint detection, passenger transport semantic state modeling, and illegal rule constraint reasoning technology. It describes in detail the joint perception of the inside and outside of the vehicle, passenger transport status determination, illegal identification, and evidence output of target vehicles in traffic scenarios. It has the advantages of high recognition accuracy, strong adaptability to complex scenarios, and high reliability of evidence results.

[0006] According to an embodiment of the present invention, a method for detecting and identifying illegal passenger transport in vehicles based on AI vision includes the following steps:

[0007] Step 1: In a traffic scenario, acquire the original visual data and related information of vehicles traveling within the target road area to generate the original vehicle visual dataset;

[0008] Step 2: Perform visual preprocessing and cross-view alignment on the original vehicle visual dataset to obtain a standard vehicle visual feature set;

[0009] Step 3: The standard vehicle visual feature set is used to perform vehicle subject detection and occupant perception through an AI vision joint perception network, generating a joint representation result of the vehicle interior and exterior. The AI ​​vision joint perception network includes a backbone feature extraction module, an efficient hybrid encoding module, a target query and filtering module, a Transformer decoding module, and a joint detection head for the vehicle interior and exterior. Among them, the efficient hybrid encoding module guides interaction based on the window area; the Transformer decoding module decodes occupants based on dense perception of occupants inside the vehicle.

[0010] Step 4: Based on the joint representation results inside and outside the vehicle, perform passenger-carrying semantic state modeling through the improved Vision Mamba model to obtain the passenger-carrying determination vector of the target vehicle; the improved Vision Mamba model includes an embedding mapping module, a position encoding module, a bidirectional state space modeling module, a semantic compression and aggregation module, and a determination feature generation module; among which, the bidirectional state space modeling module performs bidirectional state transfer based on the semantic segmentation inside and outside the vehicle and the modulation of the region boundary;

[0011] Step 5: Apply illegal rule constraints and visual reasoning to the passenger-carrying determination vector of the target vehicle to obtain the illegal passenger-carrying identification result;

[0012] Step Six: Based on the results of illegal passenger transport identification, perform multi-evidence consistency verification to generate a standard illegal evidence collection result set.

[0013] Optionally, step one specifically includes:

[0014] In traffic scenarios covered by road monitoring equipment, checkpoint cameras, or mobile law enforcement terminals, collect raw visual data of vehicles passing through the target road area;

[0015] The raw visual data includes external images of the target vehicle, side images of the target vehicle, front and rear images of the target vehicle, and images of local window areas of the target vehicle;

[0016] Obtain relevant information, including collection time, collection location identifier, and device identifier, and perform association matching on the original visual data based on the relevant information to generate the original vehicle visual dataset.

[0017] Optionally, the visual preprocessing and cross-view alignment include image size normalization, brightness compensation, occlusion suppression, window region enhancement, view distortion correction, vehicle outline position alignment, window boundary position alignment, and vehicle body structure key point mapping.

[0018] Optionally, step three specifically includes:

[0019] In the backbone feature extraction module, the standard vehicle visual feature set is processed by multi-layer convolutional extraction units and staged downsampling to generate shallow vehicle visual feature maps, mid-layer vehicle visual feature maps, and deep vehicle visual feature maps; among them, the convolutional extraction unit consists of a 3×3 convolutional layer, a batch normalization layer, and a SiLU activation function layer.

[0020] In the efficient hybrid coding module, shallow vehicle visual feature maps, mid-level vehicle visual feature maps, and deep vehicle visual feature maps are used to guide interaction based on the window area to obtain cross-scale fused vehicle feature maps.

[0021] In the target query filtering module, target response sorting and target query filtering are performed based on cross-scale fused vehicle feature maps to obtain an initial target query vector group;

[0022] The Transformer decoding module includes several layers of Transformer decoding, wherein each layer of Transformer decoding includes a query grouping reconstruction unit, an in-vehicle occupant dense perception decoding unit, and a self-attention refinement unit;

[0023] In the query grouping reconstruction unit, the initial target query vector group is divided into query vector groups to obtain the vehicle body query vector group, the window area query vector group, and the occupant area query vector group.

[0024] In the dense occupant perception decoding unit, based on the cross-scale fusion of vehicle feature maps, the visible area feature map inside the vehicle is extracted, and the visible area feature map inside the vehicle is generated into an occupant dense perception matrix through linear mapping.

[0025] Based on the occupant dense perception matrix, cross-attention operation is performed between the in-vehicle visible area feature map and the occupant dense perception matrix to obtain the occupant intermediate query vector group.

[0026] Based on the passenger intermediate query vector group, the neighboring target difference constraint is executed to obtain the decoupled passenger query vector group;

[0027] The decoupled occupant query vector group and the window area query vector group are subjected to gating fusion to obtain the occupant enhanced query vector group.

[0028] In the self-attention refinement unit, the vehicle main query vector group, the window area query vector group, and the occupant enhancement query vector group are concatenated to obtain a joint query vector group, and self-attention operation is performed on the joint query vector group to obtain a refined query vector group.

[0029] Perform feedforward update on the refined query vector group to obtain the decoded output query vector group, and use the decoded output query vector group generated in the last layer as the target-level detection feature vector group.

[0030] In the combined in-vehicle and out-of-vehicle detection head, in-vehicle and out-of-vehicle detection features are extracted based on the target-level detection feature vector group;

[0031] Based on the detection features inside and outside the vehicle, a spatial distribution map of the occupants inside the vehicle is generated through location regression and region mapping. The number of occupants is generated through target counting. The occupant congestion level inside the vehicle is generated by merging the spatial distribution density of the occupants with the occupant area occupancy rate. These results form a joint representation of the inside and outside of the vehicle.

[0032] Optionally, in the efficient hybrid encoding module, the shallow vehicle visual feature map, the mid-level vehicle visual feature map, and the deep vehicle visual feature map are used to guide interaction based on the window area to obtain a cross-scale fused vehicle feature map, specifically including:

[0033] The high-efficiency hybrid coding module includes a window area guidance interaction unit, an intra-scale interaction unit, and a cross-scale fusion output unit;

[0034] In the window area, the interactive unit performs edge response convolution and strip region convolution on the scale vehicle visual feature map to obtain the window edge response feature map and the window strip region response feature map.

[0035] The response feature map of the window edge and the response feature map of the window strip area are concatenated by channels, and the window area guiding weight matrix is ​​generated by convolution of 1×1 convolution extraction unit and mapping by Sigmoid function.

[0036] Among them, the scale vehicle visual feature map includes shallow vehicle visual feature map, medium vehicle visual feature map and deep vehicle visual feature map.

[0037] Among them, the edge response convolution consists of a 3×3 convolutional layer and a SiLU activation function layer; the strip region convolution consists of a 1×5 horizontal strip convolutional layer and a 5×1 vertical strip convolutional layer arranged in parallel, and the convolution results are added and fused before being input into the SiLU activation function layer.

[0038] In the scale-interactive unit, the scale vehicle visual feature map is obtained by three sets of linear mappings to obtain the scale vehicle query matrix, scale vehicle key matrix and scale vehicle value matrix.

[0039] The window area guidance weight matrix is ​​extended to the spatial dimension of the scale vehicle query matrix, and modulated by the area guidance modulation coefficient to obtain the area guidance matrix;

[0040] Based on the region-guided matrix, region-guided attention operations are performed on the scale vehicle query matrix, scale vehicle key matrix, and scale vehicle value matrix to obtain a scale-enhanced feature map;

[0041] In the cross-scale fusion output unit, scale alignment, cross-scale fusion, channel mapping, and feature compression are performed on the enhanced feature maps of the three scales to obtain cross-scale fused vehicle feature maps.

[0042] Optionally, the neighboring target difference constraint specifically includes:

[0043] The cosine similarity between the intermediate query vector of the i-th occupant and the intermediate query vector of the j-th occupant is calculated, and the neighboring target suppression coefficient is obtained based on the difference between 1 and the cosine similarity.

[0044] Based on the neighbor target suppression coefficient, the vector difference between the intermediate query vector of the i-th occupant and the intermediate query vector of the j-th occupant is scaled to obtain the neighbor difference enhancement vector;

[0045] The nearest difference enhancement vector is added to the i-th passenger intermediate query vector to obtain the i-th decoupled passenger query vector; where i and j are the indices of the passenger intermediate query vector group.

[0046] Optionally, step four specifically includes:

[0047] In the embedded mapping module, the spatial location distribution map of the occupants in the vehicle is divided into several spatial regions, and the position response values ​​of all occupants in each spatial region are aggregated and feature-encoded to obtain regional feature vectors; numerical normalization and feature concatenation are performed on the number of occupants and the degree of crowding of occupants in the vehicle to obtain statistical state feature vectors.

[0048] The statistical state feature vector is concatenated with the feature vectors of each region to obtain the passenger-carrying state feature vector. The passenger-carrying state feature vectors of several spatial regions are then combined to form a passenger-carrying state feature vector sequence. The passenger-carrying state feature vector sequence is then converted into a passenger-carrying semantic embedding vector sequence through linear mapping.

[0049] In the location encoding module, the passenger semantic embedding vector sequence is position-encoded using a learnable location encoding method to obtain a location-enhanced passenger semantic vector sequence.

[0050] In the bidirectional state space modeling module, the location-enhanced passenger semantic vector sequence is divided into in-vehicle region features and out-of-vehicle region features according to the spatial category to which the spatial region belongs, and forward state scanning and backward state scanning are performed respectively.

[0051] In the forward state scan, the vector difference between the region feature vector of the current spatial region and the region feature vector of the previous spatial region is used to generate a forward region boundary modulation vector through linear mapping and the Sigmoid function; the forward state transfer term from the previous spatial region to the current spatial region is modulated using the forward region boundary modulation vector to generate a forward state feature vector sequence.

[0052] In the backward state scan, the vector difference between the regional feature vector of the current spatial region and the regional feature vector of the next spatial region is used to generate the backward region boundary modulation vector through linear mapping and the Sigmoid function; the backward region boundary modulation vector is used to modulate the backward state transfer term from the next spatial region to the current spatial region to generate the backward state feature vector sequence.

[0053] The in-vehicle region features are fused with the forward and backward state feature vector sequences of the in-vehicle region features to obtain the in-vehicle bidirectional state feature vector sequence and the out-of-vehicle bidirectional state feature vector sequence.

[0054] The in-vehicle bidirectional state feature vector sequence and the out-of-vehicle bidirectional state feature vector sequence are fused across sequences to obtain a joint state feature vector sequence.

[0055] In the semantic compression and aggregation module, the joint state feature vector sequence is compressed by one-dimensional convolution, and semantic aggregation is performed by global average pooling to obtain the passenger-carrying semantic state feature vector.

[0056] In the feature generation module, the passenger-carrying semantic state feature vector is linearly mapped to generate the target vehicle passenger-carrying determination vector.

[0057] Optionally, the fusion of the violation rule constraints and visual reasoning specifically includes:

[0058] Based on the target vehicle passenger determination vector, the vehicle category determination vector, passenger number determination vector, passenger space distribution determination vector and passenger crowding determination vector are generated through four parallel linear mapping branches.

[0059] Construct rules to constrain violations, including vehicle category matching rules, passenger capacity constraint rules, in-vehicle space distribution constraint rules, and scenario legality constraint rules;

[0060] The vehicle category determination vector is matched with the vehicle category adaptation rule to obtain the vehicle category constraint vector;

[0061] The passenger number determination vector is matched with the maximum passenger capacity constraint rule by threshold comparison to obtain the passenger capacity constraint vector;

[0062] The spatial pattern matching between the occupant spatial distribution determination vector and the in-vehicle spatial distribution constraint rules is performed to obtain the spatial distribution constraint vector.

[0063] The occupant crowding determination vector in the vehicle is matched with the scene legality constraint rules to obtain the scene crowding constraint vector.

[0064] The vehicle category constraint vector, the number of people constraint vector, the spatial distribution constraint vector, and the scene congestion constraint vector are concatenated and fused to obtain the rule constraint vector.

[0065] The target vehicle passenger-carrying determination vector is spliced ​​and fused with the rule constraint vector, and then visual reasoning is performed through a multilayer perceptron to obtain the violation determination response vector.

[0066] The probability of each illegal passenger transport category is obtained by normalizing the response vector of the violation judgment through the Softmax function.

[0067] The maximum probability value among the category probability values ​​of each illegal passenger transport category is taken as the confidence level of illegal passenger transport, and the illegal passenger transport category corresponding to the maximum probability value is taken as the illegal passenger transport category label;

[0068] Based on the rule constraint vector, a set of illegal triggering criteria is constructed by mapping the constraint response value threshold to the triggering rule item. The set of illegal triggering criteria includes vehicle category triggering criteria, number of people constraint triggering criteria, spatial distribution triggering criteria, and scene legality triggering criteria.

[0069] The confidence level of illegal passenger transport, the category label of illegal passenger transport, and the set of illegal triggering evidence are used as the results of illegal passenger transport identification.

[0070] Optionally, the multi-evidence consistency verification specifically includes:

[0071] Based on the associated information, the identification results of multiple illegal passenger carrying by the target vehicle within a set time window are associated and merged to obtain a group of illegal passenger carrying results to be verified.

[0072] Based on the group of illegal passenger transport results to be verified, the consistency of each illegal passenger transport category label is compared to obtain the category consistency verification result; the confidence stability of each illegal passenger transport confidence is compared to obtain the confidence consistency verification result; and the consistency of each illegal triggering basis set is compared to obtain the triggering basis consistency verification result.

[0073] Based on the association information of the illegal passenger transport result group to be verified, the collection time is compared for temporal continuity, the collection location identifier is compared for spatial proximity, and the device identifier is compared for device association to obtain the consistency verification result of the association information.

[0074] Among them, the category consistency verification result, confidence consistency verification result, trigger basis consistency verification result, and associated information consistency verification result all include pass and fail status;

[0075] When the consistency verification results of category, confidence level, trigger basis, and related information are all passed, the evidence images, related information, illegal passenger carrying category labels, illegal passenger carrying confidence level, and illegal trigger basis set of the target passing vehicle are extracted to generate a standard illegal evidence collection result set.

[0076] The beneficial effects of this invention are:

[0077] This invention unifies the collection of raw visual data and associated information of vehicles passing through the target road area, and performs visual preprocessing and cross-viewpoint alignment to form a standardized vehicle visual feature set from vehicle images under different devices and viewpoints, reducing the impact of illumination changes, window reflections, occlusion interference, and viewpoint shifts on the recognition results. For the vehicle body detection and occupant perception stages, a guided interaction mechanism based on the window area and a decoding mechanism based on dense occupant perception are introduced into the AI ​​visual joint perception network. This enhances the feature representation ability of window-related areas and the ability to separate and identify dense occupant targets, thereby improving the accuracy of extracting the spatial distribution, number, and crowding level of occupants. For the passenger-carrying semantic state modeling stage, a bidirectional state transfer mechanism based on vehicle-inside / outside semantic segmentation and region boundary modulation is introduced into the improved Vision Mamba model, enhancing the semantic modeling ability of vehicle passenger-carrying status, vehicle-inside / outside spatial relationships, and abnormal passenger-carrying patterns. For the violation determination stage, the target vehicle passenger-carrying determination vector is fused with multiple constraint rules for reasoning, improving the accuracy and interpretability of illegal passenger-carrying identification. For the evidence collection stage, a standard set of illegal evidence collection results is formed through consistency verification, which improves the stability of the identification results and the reliability of the evidence collection. Therefore, it is of great significance to improve the automatic identification capability of vehicles illegally carrying passengers and reduce the false judgment rate and the missed judgment rate. Attached Figure Description

[0078] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0079] Figure 1 This is a schematic diagram of a vehicle illegal passenger carrying detection and identification method based on AI vision proposed in this invention;

[0080] Figure 2 This is a flowchart of the AI ​​vision joint perception network structure in the AI ​​vision-based vehicle illegal passenger carrying detection and identification method proposed in this invention;

[0081] Figure 3 This is a flowchart of the improved Vision Mamba model structure in the AI ​​vision-based method for detecting and recognizing illegal passenger transport in vehicles proposed in this invention. Detailed Implementation

[0082] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0083] refer to Figures 1-3 A method for detecting and identifying illegal passenger transport in vehicles based on AI vision includes the following steps:

[0084] Step 1: In a traffic scenario, acquire the original visual data and related information of vehicles traveling within the target road area to generate the original vehicle visual dataset;

[0085] Step 2: Perform visual preprocessing and cross-view alignment on the original vehicle visual dataset to obtain a standard vehicle visual feature set;

[0086] Step 3: The standard vehicle visual feature set is used to perform vehicle subject detection and occupant perception through an AI vision joint perception network, generating a joint representation result of the vehicle's interior and exterior. The AI ​​vision joint perception network includes a backbone feature extraction module, an efficient hybrid encoding module, a target query and filtering module, a Transformer decoding module, and a joint detection head for the vehicle's interior and exterior. Among them, the efficient hybrid encoding module guides interaction based on the window area; the Transformer decoding module decodes occupants based on dense perception of occupants inside the vehicle.

[0087] Step 4: Based on the joint representation results inside and outside the vehicle, perform passenger-carrying semantic state modeling through the improved Vision Mamba model to obtain the passenger-carrying determination vector of the target vehicle; the improved Vision Mamba model includes an embedding mapping module, a position encoding module, a bidirectional state space modeling module, a semantic compression and aggregation module, and a determination feature generation module; among them, the bidirectional state space modeling module performs bidirectional state transfer based on the semantic segmentation inside and outside the vehicle and the modulation of the region boundary;

[0088] Step 5: Apply illegal rule constraints and visual reasoning to the passenger-carrying determination vector of the target vehicle to obtain the illegal passenger-carrying identification result;

[0089] Step Six: Based on the results of illegal passenger transport identification, perform multi-evidence consistency verification to generate a standard illegal evidence collection result set.

[0090] In this embodiment, step one specifically includes:

[0091] In traffic scenarios covered by road monitoring equipment, checkpoint cameras, or mobile law enforcement terminals, collect raw visual data of vehicles passing through the target road area;

[0092] The raw visual data includes external images of the target vehicle, side images of the target vehicle, front and rear images of the target vehicle, and images of local window areas of the target vehicle;

[0093] Obtain relevant information, including collection time, collection location identifier, and device identifier, and perform association matching on the original visual data based on the relevant information to generate the original vehicle visual dataset.

[0094] In this embodiment, visual preprocessing and cross-view alignment include image size normalization, brightness compensation, occlusion suppression, window region enhancement, view distortion correction, vehicle outline position alignment, window boundary position alignment, and vehicle body structure key point mapping.

[0095] In this invention, image size normalization processing is performed on the target vehicle's external image, side image, front and rear images, and local window region image from the original vehicle visual dataset to obtain a normalized target visual image. Brightness compensation processing is then performed on the normalized target visual image to reduce brightness differences under different acquisition lighting conditions, resulting in a brightness-corrected target visual image. Occlusion suppression processing is then performed on the brightness-corrected target visual image to reduce interference from occluded areas on the vehicle's main body and the visible area inside the vehicle, resulting in an occlusion-suppressed target visual image. Finally, windowing processing is applied to the target vehicle's local window region image within the occlusion-suppressed target visual image. Region enhancement processing is performed to enhance the visual response of the window edges, the visible area inside the window, and the area adjacent to the window, obtaining a window-enhanced target visual image. Viewpoint distortion correction processing is then performed on the window-enhanced target visual image to reduce the differences in vehicle deformation under different acquisition viewpoints, obtaining a viewpoint-corrected target visual image. Based on the viewpoint-corrected target visual image, the target vehicle outline position, window boundary position, and key points of the vehicle body structure are extracted. Then, based on the target vehicle outline position, window boundary position, and key points of the vehicle body structure, cross-viewpoint alignment and region mapping processing are performed on the target visual images under different viewpoints to obtain a standard vehicle visual feature set under a unified vehicle coordinate reference system.

[0096] In this embodiment, step three specifically includes:

[0097] In the backbone feature extraction module, the standard vehicle visual feature set is processed by multi-layer convolutional extraction units and staged downsampling to generate shallow vehicle visual feature maps, mid-layer vehicle visual feature maps, and deep vehicle visual feature maps; among them, the convolutional extraction unit consists of a 3×3 convolutional layer, a batch normalization layer, and a SiLU activation function layer.

[0098] In the efficient hybrid coding module, shallow vehicle visual feature maps, mid-level vehicle visual feature maps, and deep vehicle visual feature maps are used to guide interaction based on the window area to obtain cross-scale fused vehicle feature maps.

[0099] In the target query filtering module, target response sorting and target query filtering are performed based on cross-scale fused vehicle feature maps to obtain an initial target query vector group;

[0100] The Transformer decoding module includes several layers of Transformer decoding, where each layer of Transformer decoding includes a query grouping reconstruction unit, an in-vehicle occupant dense perception decoding unit, and a self-attention refinement unit.

[0101] In the query grouping reconstruction unit, the initial target query vector group is divided into query vector groups to obtain the vehicle body query vector group, the window area query vector group, and the occupant area query vector group.

[0102] In the dense occupant perception decoding unit, based on the cross-scale fusion of vehicle feature maps, the visible area feature map inside the vehicle is extracted, and the visible area feature map inside the vehicle is generated into an occupant dense perception matrix through linear mapping.

[0103] Based on the occupant dense perception matrix, cross-attention operation is performed between the in-vehicle visible area feature map and the occupant dense perception matrix to obtain the occupant intermediate query vector group.

[0104] Based on the passenger intermediate query vector group, the neighboring target difference constraint is executed to obtain the decoupled passenger query vector group;

[0105] The decoupled occupant query vector group and the window area query vector group are subjected to gating fusion to obtain the occupant enhanced query vector group.

[0106] In the self-attention refinement unit, the vehicle main query vector group, the window area query vector group, and the occupant enhancement query vector group are concatenated to obtain a joint query vector group, and self-attention operation is performed on the joint query vector group to obtain a refined query vector group.

[0107] Perform feedforward update on the refined query vector group to obtain the decoded output query vector group, and use the decoded output query vector group generated in the last layer as the target-level detection feature vector group.

[0108] In the joint detection head inside and outside the vehicle, the detection features inside and outside the vehicle are extracted based on the target-level detection feature vector group. The detection features inside and outside the vehicle include vehicle category features, body structure features, window perspective features and occupant distribution features.

[0109] Based on the detection features inside and outside the vehicle, a spatial distribution map of the occupants inside the vehicle is generated through location regression and region mapping. The number of occupants is generated through target counting. The occupant congestion level inside the vehicle is generated by merging the spatial distribution density of the occupants with the occupant area occupancy rate. These results form a joint representation of the inside and outside of the vehicle.

[0110] In this embodiment, the high-efficiency hybrid encoding module guides interaction based on the window area using shallow, mid-level, and deep vehicle visual feature maps to obtain a cross-scale fused vehicle feature map, specifically including:

[0111] The high-efficiency hybrid coding module includes a window area guidance interaction unit, an intra-scale interaction unit, and a cross-scale fusion output unit;

[0112] In the window area, the interactive unit performs edge response convolution and strip region convolution on the scale vehicle visual feature map to obtain the window edge response feature map and the window strip region response feature map.

[0113] The response feature map of the window edge and the response feature map of the window strip area are concatenated by channels, and the window area guiding weight matrix is ​​generated by convolution of 1×1 convolution extraction unit and mapping by Sigmoid function.

[0114] Among them, the scale vehicle visual feature map includes shallow vehicle visual feature map, medium vehicle visual feature map and deep vehicle visual feature map.

[0115] Among them, the edge response convolution consists of a 3×3 convolutional layer and a SiLU activation function layer; the strip region convolution consists of a 1×5 horizontal strip convolutional layer and a 5×1 vertical strip convolutional layer arranged in parallel, and the convolution results are added and fused before being input into the SiLU activation function layer.

[0116] In the scale-interactive unit, the scale vehicle visual feature map is obtained by three sets of linear mappings to obtain the scale vehicle query matrix, scale vehicle key matrix and scale vehicle value matrix.

[0117] The window area guidance weight matrix is ​​extended to the spatial dimension of the scale vehicle query matrix, and modulated by the area guidance modulation coefficient to obtain the area guidance matrix;

[0118] Based on the region-guided matrix, region-guided attention operations are performed on the scale vehicle query matrix, scale vehicle key matrix, and scale vehicle value matrix to obtain a scale-enhanced feature map;

[0119] Specifically, the region-guided attention operation is as follows: multiply the scaled vehicle query matrix by the transpose of the scaled vehicle key matrix and divide by the square root of the dimension of the scaled vehicle key matrix to obtain the scaled vehicle score matrix; add the scaled vehicle score matrix to the region-guided matrix and normalize it using the Softmax function to obtain the scaled vehicle weight matrix; and perform matrix multiplication on the scaled vehicle value matrix based on the scaled vehicle weight matrix to obtain the scale-enhanced feature map.

[0120] In the cross-scale fusion output unit, scale alignment, cross-scale fusion, channel mapping, and feature compression are performed on the enhanced feature maps of the three scales to obtain cross-scale fused vehicle feature maps.

[0121] In this embodiment, the neighboring target difference constraint is specifically as follows:

[0122] The cosine similarity between the intermediate query vector of the i-th occupant and the intermediate query vector of the j-th occupant is calculated, and the neighboring target suppression coefficient is obtained based on the difference between 1 and the cosine similarity.

[0123] Based on the neighbor target suppression coefficient, the vector difference between the intermediate query vector of the i-th occupant and the intermediate query vector of the j-th occupant is scaled to obtain the neighbor difference enhancement vector;

[0124] The nearest difference enhancement vector is added to the i-th passenger intermediate query vector to obtain the i-th decoupled passenger query vector; where i and j are the indices of the passenger intermediate query vector group.

[0125] In this invention, the AI ​​vision joint perception network is a network structure improved from the RT-DETR network. The RT-DETR network includes a backbone feature extraction part, a hybrid encoding part, a target query filtering part, a Transformer decoding part, and a detection head part, which are used to extract multi-scale features from the input image, generate target queries, and complete target detection output. The AI ​​vision joint perception network retains the overall framework of the RT-DETR network and makes targeted improvements to its intermediate structure for the scenario of illegal passenger carrying in vehicles.

[0126] The improvements to the AI ​​vision joint perception network are mainly reflected in two areas. In the efficient hybrid encoding module, a guided interaction mechanism based on the vehicle window region is introduced, addressing the original multi-scale feature interaction structure of RT-DETR. Through a window region guided interaction unit, an intra-scale interaction unit, and a cross-scale fusion output unit, regional guidance enhancement is applied to shallow, mid-level, and deep vehicle visual feature maps, enabling the network to pay more attention to window edges, window strips, and the visible area inside the vehicle during the feature fusion stage. In the Transformer decoding module, a decoding mechanism based on dense occupant perception is introduced, addressing the original target query decoding structure of RT-DETR. Through a query grouping reconstruction unit, a dense occupant perception decoding unit, and a self-attention refinement unit, dense occupant perception, neighboring target difference constraints, and gating fusion processing are performed on the features of the visible area inside the vehicle.

[0127] Through the above improvements, the focus on the window area and the visible area inside the vehicle can be enhanced in the encoding stage, which can improve the ability to express the perspective features of the window and the features of the area inside the vehicle. In the decoding stage, the ability to distinguish between dense occupant targets, occupant targets and adjacent occupant targets can be enhanced, which can improve the accuracy of extracting the spatial distribution map of occupants inside the vehicle, the number of occupants and the degree of crowding of occupants inside the vehicle, thereby improving the overall accuracy of vehicle body detection and occupant perception.

[0128] In this embodiment, step four specifically includes:

[0129] In the embedded mapping module, the spatial location distribution map of the occupants in the vehicle is divided into several spatial regions, and the position response values ​​of all occupants in each spatial region are aggregated and feature-encoded to obtain regional feature vectors; numerical normalization and feature concatenation are performed on the number of occupants and the degree of crowding of occupants in the vehicle to obtain statistical state feature vectors.

[0130] The statistical state feature vector is concatenated with the feature vectors of each region to obtain the passenger-carrying state feature vector. The passenger-carrying state feature vectors of several spatial regions are then combined to form a passenger-carrying state feature vector sequence. The passenger-carrying state feature vector sequence is then converted into a passenger-carrying semantic embedding vector sequence through linear mapping.

[0131] In the location encoding module, a learnable location encoding method is used to position-encode the passenger semantic embedding vector sequence to obtain a location-enhanced passenger semantic vector sequence. Specifically, the learnable location encoding method involves: first, obtaining each passenger semantic embedding vector in the passenger semantic embedding vector sequence, and assigning a location encoding vector to each passenger semantic embedding vector according to the spatial region arrangement order. The location encoding vector is a learnable parameter vector, generated through initialization, and different location encoding vectors correspond to different spatial regions. Each passenger semantic embedding vector is element-wise added to its corresponding location encoding vector to obtain a location-enhanced passenger semantic vector. Finally, the location-enhanced passenger semantic vectors are rearranged according to the original spatial region order to obtain the location-enhanced passenger semantic vector sequence.

[0132] In the bidirectional state space modeling module, the location-enhanced passenger semantic vector sequence is divided into in-vehicle region features and out-of-vehicle region features according to the spatial category to which the spatial region belongs, and forward state scanning and backward state scanning are performed respectively.

[0133] In the forward state scan, the vector difference between the region feature vector of the current spatial region and the region feature vector of the previous spatial region is used to generate a forward region boundary modulation vector through linear mapping and the Sigmoid function. The forward state transfer term from the previous spatial region to the current spatial region is modulated using the forward region boundary modulation vector to generate a forward state feature vector sequence. The forward state transfer term is the product of the forward state transition matrix and the forward state feature vector of the previous spatial region. The forward state transition matrix is ​​generated by linear mapping from the position-enhanced passenger semantic vector of the current spatial region. The forward state feature vector is generated recursively through the forward state scan, and the initial forward state feature vector is a zero vector. By modulating the forward state transfer term with the forward region boundary modulation vector, the state information transmission intensity of the previous spatial region is weakened when the boundary difference between adjacent spatial regions is large, and the state information transmission intensity of the previous spatial region is enhanced when the boundary difference between adjacent spatial regions is small, thereby improving the boundary sensitivity of forward state propagation and the accuracy of passenger semantic state modeling.

[0134] In the backward state scan, the vector difference between the region feature vector of the current spatial region and the region feature vector of the next spatial region is used to generate a backward region boundary modulation vector through linear mapping and the Sigmoid function. The backward state transfer term from the next spatial region to the current spatial region is modulated using the backward region boundary modulation vector to generate a backward state feature vector sequence. The backward state transfer term is the product of the backward state transition matrix and the backward state feature vector of the next spatial region. The backward state transition matrix is ​​generated by linear mapping from the position-enhanced passenger semantic vector of the current spatial region. The backward state feature vector is generated recursively through the backward state scan. The initial backward state feature vector is the zero vector.

[0135] The in-vehicle region features are fused with the forward and backward state feature vector sequences of the in-vehicle region features to obtain the in-vehicle bidirectional state feature vector sequence and the out-of-vehicle bidirectional state feature vector sequence.

[0136] The in-vehicle bidirectional state feature vector sequence and the out-of-vehicle bidirectional state feature vector sequence are fused across sequences to obtain a joint state feature vector sequence.

[0137] In the semantic compression and aggregation module, the joint state feature vector sequence is compressed by one-dimensional convolution, and semantic aggregation is performed by global average pooling to obtain the passenger-carrying semantic state feature vector.

[0138] In the feature generation module, the passenger-carrying semantic state feature vector is linearly mapped to generate the target vehicle passenger-carrying determination vector.

[0139] In this invention, the improved Vision Mamba model primarily improves the intermediate state modeling structure of the Vision Mamba model. It introduces a semantic segmentation mechanism for both inside and outside the vehicle into the bidirectional state space modeling module, and adds a region boundary modulation mechanism based on the feature differences between adjacent spatial regions. This allows the features inside and outside the vehicle to undergo forward and backward state scanning, respectively. Through these improvements, the model acquires boundary sensitivity and semantic partitioning modeling capabilities during state propagation. This reduces invalid information transmission between regions with large boundary differences and enhances effective state propagation between regions with small boundary differences, thereby improving the modeling accuracy of vehicle passenger status, inside-outside spatial relationships, and abnormal passenger patterns.

[0140] In this embodiment, the fusion of illegal rule constraints and visual reasoning specifically includes:

[0141] Based on the target vehicle passenger determination vector, the vehicle category determination vector, passenger number determination vector, passenger space distribution determination vector and passenger crowding determination vector are generated through four parallel linear mapping branches.

[0142] Construct rules to constrain violations, including vehicle category matching rules, passenger capacity constraint rules, in-vehicle space distribution constraint rules, and scenario legality constraint rules;

[0143] The vehicle category determination vector is matched with the vehicle category adaptation rule to obtain the vehicle category constraint vector. For example, the vehicle category determination vector is [0.82, 0.11, 0.04, 0.03], and the four components correspond to small passenger cars, operating passenger cars, freight vehicles, and special-purpose vehicles, respectively. The maximum response value of 0.82 corresponds to small passenger cars. If the vehicle category adaptation rule is "small passenger cars are allowed to be ridden normally, but the number of passengers is not allowed to exceed the capacity, and the cargo compartment is not allowed to carry passengers", since the vehicle category recognition is valid, the code is 1. There is no category conflict, so the code is 0. It needs to enter the capacity constraint verification, so the code is 1. The final generated vehicle category constraint vector is [1, 0, 1].

[0144] The passenger number determination vector is matched with the passenger capacity constraint rule by threshold comparison to obtain the passenger capacity constraint vector. For example, the passenger number determination vector [0.06, 0.12, 0.18, 0.64] corresponds to "2 people, 3 people, 4 people, 5 people" respectively. The maximum response value of 0.64 corresponds to "5 people". If the passenger capacity constraint rule of the small passenger vehicle is "passenger capacity of 4 people", the passenger number prediction value of 5 exceeds the passenger capacity threshold of 4, the passenger number determination is valid and is coded as 1. The passenger number exceeds the passenger capacity and is coded as 1. The over-limit range is 1 person and is coded as 1. Then the generated passenger capacity constraint vector is [1, 1, 1].

[0145] The occupant space distribution determination vector is matched with the in-vehicle space distribution constraint rules to obtain the space distribution constraint vector. For example, the occupant space distribution determination vector [0.15, 0.71, 0.09, 0.05] has four components corresponding to "uniform distribution, dense distribution in the rear row, abnormal clustering in the front row, and abnormal distribution at the edge", where the maximum response value of 0.71 corresponds to "dense distribution in the rear row". If the in-vehicle space distribution constraint rule is "when the occupant density in the rear row area exceeds 1.8 times the occupant density in the front row area and the occupancy rate in the rear row area exceeds 0.75, it is determined to be a space abnormal clustering mode", and the actual rear row area density of this vehicle is 0.82, the front row area density is 0.36, and the rear row area occupancy rate is 0.79. Since the space distribution determination is valid, it is encoded as 1. It does not meet the normal uniform distribution, so it is encoded as 0. It meets the abnormal clustering mode, so it is encoded as 1, thus generating the space distribution constraint vector [1, 0, 1].

[0146] The occupant crowding determination vector is matched with the scene legality constraint rules to obtain the scene crowding constraint vector. For example, the occupant crowding determination vector [0.18, 0.27, 0.55] has three components corresponding to "low crowding, medium crowding, and high crowding", with the maximum response value of 0.55 corresponding to "high crowding". If the scene legality constraint rule is "when a vehicle is in a high-crowding state in the road section near the bus stop, the temporary parking area at the school gate, or the no-stopping area for picking up and dropping off passengers, it is considered a high-risk scene anomaly", and the collection location identifier of the target vehicle corresponds to the road section near the bus stop, then the "high crowding" is matched with the "road section near the bus stop" to generate the scene crowding constraint vector [1, 1], with each component representing "scene matching is established" and "crowding anomaly is established".

[0147] The vehicle category constraint vector, the number of people constraint vector, the spatial distribution constraint vector, and the scene congestion constraint vector are concatenated and fused to obtain the rule constraint vector.

[0148] The target vehicle passenger-carrying determination vector is spliced ​​and fused with the rule constraint vector, and then visual reasoning is performed through a multilayer perceptron to obtain the violation determination response vector.

[0149] The probability of each illegal passenger transport category is obtained by normalizing the response vector of the violation judgment through the Softmax function.

[0150] The maximum probability value among the category probability values ​​of each illegal passenger transport category is taken as the confidence level of illegal passenger transport, and the illegal passenger transport category corresponding to the maximum probability value is taken as the illegal passenger transport category label; among them, the illegal passenger transport categories include normal passenger transport category, overloaded passenger transport category, spatially abnormal passenger transport category, scene abnormal passenger transport category, and composite abnormal passenger transport category;

[0151] Based on the rule constraint vector, a set of illegal triggering criteria is constructed by mapping the constraint response value threshold to the triggering rule item. The set of illegal triggering criteria includes vehicle category triggering criteria, number of people constraint triggering criteria, spatial distribution triggering criteria, and scenario legality triggering criteria.

[0152] The confidence level of illegal passenger transport, the category label of illegal passenger transport, and the set of illegal triggering evidence are used as the results of illegal passenger transport identification.

[0153] In this embodiment, the multi-evidence consistency verification specifically includes:

[0154] Based on the associated information, the identification results of multiple illegal passenger carrying by the target vehicle within a set time window are associated and merged to obtain a group of illegal passenger carrying results to be verified.

[0155] Based on the group of illegal passenger transport results to be verified, the consistency of each illegal passenger transport category label is compared to obtain the category consistency verification result; the confidence stability of each illegal passenger transport confidence is compared to obtain the confidence consistency verification result; and the consistency of each illegal triggering basis set is compared to obtain the triggering basis consistency verification result.

[0156] Based on the association information of the illegal passenger transport result group to be verified, the collection time is compared for temporal continuity, the collection location identifier is compared for spatial proximity, and the device identifier is compared for device association to obtain the consistency verification result of the association information.

[0157] Among them, the category consistency verification result, confidence consistency verification result, trigger basis consistency verification result, and associated information consistency verification result all include pass and fail status;

[0158] When the consistency verification results of category, confidence level, trigger basis, and related information are all passed, the evidence images, related information, illegal passenger carrying category labels, illegal passenger carrying confidence level, and illegal trigger basis set of the target passing vehicle are extracted to generate a standard illegal evidence collection result set.

[0159] For example, within the same target road area, a small passenger vehicle with a license plate characteristic consistent with the vehicle's information is identified as having violated passenger transport regulations three times within 10 seconds by roadside monitoring device A, checkpoint camera device B, and mobile enforcement terminal C. These three violations are then grouped together into a single group for verification. If the violation category label for all three violations is "overloading," the category verification result is "passed." The corresponding confidence levels for these violations are 0.91, 0.89, and 0.93, respectively. The difference between the maximum and minimum values ​​is 0.04, which is less than the set confidence level fluctuation threshold of 0.05. Therefore, the confidence level verification result is "passed." If the set of violation triggering criteria includes both "personnel constraint triggering criteria" and "spatial distribution triggering criteria," the triggering criteria verification result is "passed." Meanwhile, the three data collection times were 08:15:21, 08:15:24, and 08:15:28, satisfying temporal continuity. The data collection location markers were all located near the same bus stop, satisfying spatial proximity. Devices A, B, and C are all associated data collection devices for the target road area, satisfying device association. Therefore, the association information verification result is "passed." Since the category verification result, confidence level verification result, trigger basis verification result, and association information verification result are all "passed," the evidence image, association information, illegal passenger carrying category label "overloaded passenger category," illegal passenger carrying confidence level of 0.93, and illegal trigger basis set of the target passing vehicle are extracted to generate a standard illegal evidence collection result set.

[0160] Example 1: To verify the feasibility of this invention in practice, the method of this invention was applied to an intelligent monitoring project for illegal passenger transport on key roads in the main urban area of ​​a city. The deployment area of ​​this project includes distribution roads around the railway station, auxiliary roads in front of hospitals, temporary parking sections in commercial streets, and high-frequency passenger transport sections in urban-rural fringe areas. These areas are characterized by high traffic volume, complex vehicle types, frequent temporary passenger pick-up and drop-off, significant window reflections, drastic changes in lighting during morning and evening rush hours, and cross-coverage by multiple devices. Traditional methods relying on manual patrols or ordinary video recognition are prone to problems such as incomplete vehicle identification, large discrepancies in the number of passengers in vehicles, insufficient evidence for triggering violations, and inconsistent evidence results across devices, making it difficult to meet the needs of continuous, automated, and highly reliable traffic enforcement.

[0161] In the implementation of this invention, raw visual data of vehicles passing through the target road area is collected jointly by road monitoring equipment, checkpoint cameras, and mobile law enforcement terminals, and the collection time, collection location identifier, and equipment identifier are recorded simultaneously. First, vehicle exterior images, vehicle side images, vehicle front and rear images, and vehicle local window area images from different perspectives are correlated and matched to form a raw vehicle visual dataset. Then, image size normalization, brightness compensation, occlusion suppression, window area enhancement, viewpoint distortion correction, vehicle outline position alignment, window boundary position alignment, and vehicle body structure key point mapping are performed on the raw vehicle visual dataset to obtain a standard vehicle visual feature set.

[0162] In actual operation, a standard vehicle visual feature set is input into the AI ​​vision joint perception network. The backbone feature extraction module is responsible for extracting shallow, medium, and deep vehicle visual features. The efficient hybrid encoding module enhances the response of the window and interior visible areas by guiding interaction through the window area. The target query filtering module filters high-response query vectors from the cross-scale fused vehicle feature map. The Transformer decoding module performs group decoding on the vehicle body, window area, and occupant area, and improves the recognition ability of occupants, adjacent occupants, and low-visibility occupants by leveraging the dense occupant perception mechanism inside the vehicle. Finally, the joint detection head inside and outside the vehicle outputs the spatial distribution map of occupants, the number of occupants, and the occupant crowding level inside the vehicle, and inputs it into the improved Vision Mamba model. Through embedding mapping, position encoding, bidirectional state space modeling, semantic compression aggregation, and judgment feature generation processes, a target vehicle passenger determination vector is formed. Furthermore, the judgment vector is fused with vehicle category adaptation rules, passenger capacity constraint rules, in-vehicle space distribution constraint rules, and scenario legality constraint rules for reasoning, outputting illegal passenger carrying category labels, illegal passenger carrying confidence scores, and a set of illegal triggering evidence. For multiple identification results of the same vehicle within a set time window, category consistency, confidence score consistency, triggering evidence consistency, and associated information consistency checks are performed again, and only results that pass all checks are used to generate a standard illegal evidence collection result set.

[0163] Actual road video data from 30 consecutive days was selected as the test sample, encompassing a total of 12,680 vehicles. Among these, 1,386 vehicles were manually verified for illegal passenger transport, including cases of abnormal operation of ordinary passenger cars, exceeding passenger capacity limits, and localized congestion-related illegal passenger transport. The method of this invention was compared and analyzed with three sets of comparative methods: Method 1 was a traditional manual patrol-assisted judgment scheme; Method 2 was a scheme combining ordinary vehicle detection and passenger counting; and Method 3 was a visual recognition scheme consisting of a standard RT-DETR network without rule constraints and consistency checks, and a Vision Mamba model. The comparison results are shown in Table 1.

[0164] Table 1. Comparison of the effectiveness of different methods in the fault assessment scenario of electricity metering boxes.

[0165] Comparison indicators Method of the present invention Comparison Method 1 Comparison Method 2 Comparison Method 3 Accuracy rate of illegal passenger transport identification / % 96.8 84.7 88.9 91.3 Illegal passenger transport recall rate / % 95.4 76.2 82.8 87.1 False positive rate / % 2.6 9.8 7.4 5.9 False negative rate / % 4.6 23.8 17.2 12.9 Passenger count accuracy rate / % 97.2 85.1 89.6 92.4 Accuracy rate of passenger crowding assessment / % 95.9 80.4 84.8 90.1 Standard certification pass rate / % 94.7 78.6 83.2 88.5 Average processing time per vehicle / s 0.41 5.80 0.36 0.39

[0166] As shown in Table 1, the method of this invention is significantly superior to the comparative methods. Specifically, the accuracy rate of illegal passenger carrying identification by the method of this invention reaches 96.8%, which is 12.1 percentage points higher than comparative method 1, 7.9 percentage points higher than comparative method 2, and 5.5 percentage points higher than comparative method 3. The illegal passenger carrying recall rate reaches 95.4%, significantly higher than the three comparative methods, indicating that the present invention has a stronger ability to detect vehicles illegally carrying passengers in complex traffic scenarios. Meanwhile, the false positive rate and false negative rate of the method of this invention are the lowest among all methods, at 2.6%, indicating that the present invention can effectively reduce false and false identification phenomena. The accuracy rate of occupant number statistics reaches 97.2%, and the accuracy rate of occupant crowding determination reaches 95.9%, indicating that the present invention, through the window area guided interaction mechanism and the occupant density perception decoding mechanism in the AI ​​vision joint perception network, can more accurately extract occupant distribution information and crowding status information.

[0167] Furthermore, the standard evidence collection pass rate of the method of this invention reaches 94.7%, significantly higher than that of the comparative methods, indicating that the present invention can effectively improve the stability and reliability of illegal evidence collection results through the fusion of illegal rule constraints and multi-evidence consistency verification. Although the average processing time per vehicle of the method of this invention is 0.41 seconds, slightly higher than comparative methods 2 and 3, it is far lower than the traditional manual patrol-assisted judgment scheme, and still meets the real-time identification requirements in actual road traffic law enforcement scenarios. Therefore, the present invention significantly improves the accuracy, stability, and reliability of vehicle illegal passenger carrying identification while ensuring processing efficiency, and has good practical application value.

[0168] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for detecting and identifying illegal passenger transport in vehicles based on AI vision, characterized in that, Includes the following steps: Step 1: In a traffic scenario, acquire the original visual data and related information of vehicles traveling within the target road area to generate the original vehicle visual dataset; Step 2: Perform visual preprocessing and cross-view alignment on the original vehicle visual dataset to obtain a standard vehicle visual feature set; Step 3: The standard vehicle visual feature set is used to perform vehicle subject detection and occupant perception through an AI vision joint perception network, generating a joint representation result of the vehicle interior and exterior. The AI ​​vision joint perception network includes a backbone feature extraction module, an efficient hybrid encoding module, a target query and filtering module, a Transformer decoding module, and a joint detection head for the vehicle interior and exterior. Among them, the efficient hybrid encoding module guides interaction based on the window area; the Transformer decoding module decodes occupants based on dense perception of occupants inside the vehicle. Step 4: Based on the joint representation results inside and outside the vehicle, perform passenger-carrying semantic state modeling through the improved Vision Mamba model to obtain the passenger-carrying determination vector of the target vehicle; the improved Vision Mamba model includes an embedding mapping module, a position encoding module, a bidirectional state space modeling module, a semantic compression and aggregation module, and a determination feature generation module; among which, the bidirectional state space modeling module performs bidirectional state transfer based on the semantic segmentation inside and outside the vehicle and the modulation of the region boundary; Step 5: Apply illegal rule constraints and visual reasoning to the passenger-carrying determination vector of the target vehicle to obtain the illegal passenger-carrying identification result; Step Six: Based on the results of illegal passenger transport identification, perform multi-evidence consistency verification to generate a standard illegal evidence collection result set.

2. The method for detecting and identifying illegal passenger transport in vehicles based on AI vision according to claim 1, characterized in that, Step one specifically includes: In traffic scenarios covered by road monitoring equipment, checkpoint cameras, or mobile law enforcement terminals, collect raw visual data of vehicles passing through the target road area; The raw visual data includes external images of the target vehicle, side images of the target vehicle, front and rear images of the target vehicle, and images of local window areas of the target vehicle; Obtain relevant information, including collection time, collection location identifier, and device identifier, and perform association matching on the original visual data based on the relevant information to generate the original vehicle visual dataset.

3. The method for detecting and identifying illegal passenger transport in vehicles based on AI vision according to claim 1, characterized in that, The visual preprocessing and cross-view alignment include image size normalization, brightness compensation, occlusion suppression, window region enhancement, view distortion correction, vehicle outline position alignment, window boundary position alignment, and vehicle body structure key point mapping.

4. The method for detecting and identifying illegal passenger transport in vehicles based on AI vision according to claim 1, characterized in that, Step three specifically includes: In the backbone feature extraction module, the standard vehicle visual feature set is processed by multi-layer convolutional extraction units and staged downsampling to generate shallow vehicle visual feature maps, mid-layer vehicle visual feature maps, and deep vehicle visual feature maps; among them, the convolutional extraction unit consists of a 3×3 convolutional layer, a batch normalization layer, and a SiLU activation function layer. In the efficient hybrid coding module, shallow vehicle visual feature maps, mid-level vehicle visual feature maps, and deep vehicle visual feature maps are used to guide interaction based on the window area to obtain cross-scale fused vehicle feature maps. In the target query filtering module, target response sorting and target query filtering are performed based on the cross-scale fused vehicle feature map to obtain an initial target query vector group; The Transformer decoding module includes several layers of Transformer decoding, wherein each layer of Transformer decoding includes a query grouping reconstruction unit, an in-vehicle occupant dense perception decoding unit, and a self-attention refinement unit; In the query grouping reconstruction unit, the initial target query vector group is divided into query vector groups to obtain the vehicle body query vector group, the window area query vector group, and the occupant area query vector group. In the dense occupant perception decoding unit, based on the cross-scale fusion of vehicle feature maps, the visible area feature map inside the vehicle is extracted, and the visible area feature map inside the vehicle is generated into an occupant dense perception matrix through linear mapping. Based on the occupant dense perception matrix, cross-attention operation is performed between the in-vehicle visible area feature map and the occupant dense perception matrix to obtain the occupant intermediate query vector group. Based on the passenger intermediate query vector group, the neighboring target difference constraint is executed to obtain the decoupled passenger query vector group; The decoupled occupant query vector group and the window area query vector group are subjected to gating fusion to obtain the occupant enhanced query vector group. In the self-attention refinement unit, the vehicle main query vector group, the window area query vector group, and the occupant enhancement query vector group are concatenated to obtain a joint query vector group, and self-attention operation is performed on the joint query vector group to obtain a refined query vector group. Perform feedforward update on the refined query vector group to obtain the decoded output query vector group, and use the decoded output query vector group generated in the last layer as the target-level detection feature vector group. In the combined in-vehicle and out-of-vehicle detection head, in-vehicle and out-of-vehicle detection features are extracted based on the target-level detection feature vector group; Based on the detection features inside and outside the vehicle, a spatial distribution map of the occupants inside the vehicle is generated through location regression and region mapping. The number of occupants is generated through target counting. The occupant congestion level inside the vehicle is generated by merging the spatial distribution density of the occupants with the occupant area occupancy rate. These results form a joint representation of the inside and outside of the vehicle.

5. The method for detecting and identifying illegal passenger transport in vehicles based on AI vision according to claim 4, characterized in that, In the high-efficiency hybrid encoding module, shallow, mid-level, and deep vehicle visual feature maps are used to guide interaction based on the window area to obtain a cross-scale fused vehicle feature map, specifically including: The high-efficiency hybrid coding module includes a window area guidance interaction unit, an intra-scale interaction unit, and a cross-scale fusion output unit; In the window area, the interactive unit performs edge response convolution and strip region convolution on the scale vehicle visual feature map to obtain the window edge response feature map and the window strip region response feature map. The response feature map of the window edge and the response feature map of the window strip area are concatenated by channels, and the window area guiding weight matrix is ​​generated by convolution of 1×1 convolution extraction unit and mapping by Sigmoid function. Among them, the scale vehicle visual feature map includes shallow vehicle visual feature map, medium vehicle visual feature map and deep vehicle visual feature map. Among them, the edge response convolution consists of a 3×3 convolutional layer and a SiLU activation function layer; the strip region convolution consists of a 1×5 horizontal strip convolutional layer and a 5×1 vertical strip convolutional layer arranged in parallel, and the convolution results are added and fused before being input into the SiLU activation function layer. In the scale-interactive unit, the scale vehicle visual feature map is obtained by three sets of linear mappings to obtain the scale vehicle query matrix, scale vehicle key matrix and scale vehicle value matrix. The window area guidance weight matrix is ​​extended to the spatial dimension of the scale vehicle query matrix, and modulated by the area guidance modulation coefficient to obtain the area guidance matrix; Based on the region-guided matrix, region-guided attention operations are performed on the scale vehicle query matrix, scale vehicle key matrix, and scale vehicle value matrix to obtain a scale-enhanced feature map; In the cross-scale fusion output unit, scale alignment, cross-scale fusion, channel mapping, and feature compression are performed on the enhanced feature maps of the three scales to obtain cross-scale fused vehicle feature maps.

6. The method for detecting and identifying illegal passenger transport in vehicles based on AI vision according to claim 4, characterized in that, The neighboring target difference constraint is specifically as follows: The cosine similarity between the intermediate query vector of the i-th occupant and the intermediate query vector of the j-th occupant is calculated, and the neighboring target suppression coefficient is obtained based on the difference between 1 and the cosine similarity. Based on the neighbor target suppression coefficient, the vector difference between the intermediate query vector of the i-th occupant and the intermediate query vector of the j-th occupant is scaled to obtain the neighbor difference enhancement vector; The nearest difference enhancement vector is added to the i-th passenger intermediate query vector to obtain the i-th decoupled passenger query vector; where i and j are the indices of the passenger intermediate query vector group.

7. The method for detecting and identifying illegal passenger transport in vehicles based on AI vision according to claim 1, characterized in that, Step four specifically includes: In the embedded mapping module, the spatial location distribution map of the occupants in the vehicle is divided into several spatial regions, and the position response values ​​of all occupants in each spatial region are aggregated and feature-encoded to obtain regional feature vectors; numerical normalization and feature concatenation are performed on the number of occupants and the degree of crowding of occupants in the vehicle to obtain statistical state feature vectors. The statistical state feature vector is concatenated with the feature vectors of each region to obtain the passenger-carrying state feature vector. The passenger-carrying state feature vectors of several spatial regions are then combined to form a passenger-carrying state feature vector sequence. The passenger-carrying state feature vector sequence is then converted into a passenger-carrying semantic embedding vector sequence through linear mapping. In the location encoding module, the passenger semantic embedding vector sequence is position-encoded using a learnable location encoding method to obtain a location-enhanced passenger semantic vector sequence. In the bidirectional state space modeling module, the location-enhanced passenger semantic vector sequence is divided into in-vehicle region features and out-of-vehicle region features according to the spatial category to which the spatial region belongs, and forward state scanning and backward state scanning are performed respectively. In the forward state scan, the vector difference between the region feature vector of the current spatial region and the region feature vector of the previous spatial region is used to generate a forward region boundary modulation vector through linear mapping and the Sigmoid function; the forward state transfer term from the previous spatial region to the current spatial region is modulated using the forward region boundary modulation vector to generate a forward state feature vector sequence. In the backward state scan, the vector difference between the regional feature vector of the current spatial region and the regional feature vector of the next spatial region is used to generate the backward region boundary modulation vector through linear mapping and the Sigmoid function; the backward region boundary modulation vector is used to modulate the backward state transfer term from the next spatial region to the current spatial region to generate the backward state feature vector sequence. The in-vehicle region features are fused with the forward and backward state feature vector sequences of the in-vehicle region features to obtain the in-vehicle bidirectional state feature vector sequence and the out-of-vehicle bidirectional state feature vector sequence. The in-vehicle bidirectional state feature vector sequence and the out-of-vehicle bidirectional state feature vector sequence are fused across sequences to obtain a joint state feature vector sequence. In the semantic compression and aggregation module, the joint state feature vector sequence is compressed by one-dimensional convolution, and semantic aggregation is performed by global average pooling to obtain the passenger-carrying semantic state feature vector. In the feature generation module, the passenger-carrying semantic state feature vector is linearly mapped to generate the target vehicle passenger-carrying determination vector.

8. The method for detecting and identifying illegal passenger transport in vehicles based on AI vision according to claim 1, characterized in that, The fusion of illegal rule constraints and visual reasoning specifically includes: Based on the target vehicle passenger determination vector, the vehicle category determination vector, passenger number determination vector, passenger space distribution determination vector and passenger crowding determination vector are generated through four parallel linear mapping branches. Construct rules to constrain violations, including vehicle category matching rules, passenger capacity constraint rules, in-vehicle space distribution constraint rules, and scenario legality constraint rules; The vehicle category determination vector is matched with the vehicle category adaptation rule to obtain the vehicle category constraint vector; The passenger number determination vector is matched with the maximum passenger capacity constraint rule by threshold comparison to obtain the passenger capacity constraint vector; The spatial pattern matching between the occupant spatial distribution determination vector and the in-vehicle spatial distribution constraint rules is performed to obtain the spatial distribution constraint vector. The occupant crowding determination vector in the vehicle is matched with the scene legality constraint rules to obtain the scene crowding constraint vector. The vehicle category constraint vector, the number of people constraint vector, the spatial distribution constraint vector, and the scene congestion constraint vector are concatenated and fused to obtain the rule constraint vector. The target vehicle passenger-carrying determination vector is spliced ​​and fused with the rule constraint vector, and then visual reasoning is performed through a multilayer perceptron to obtain the violation determination response vector. The probability of each illegal passenger transport category is obtained by normalizing the response vector of the violation judgment through the Softmax function. The maximum probability value among the category probability values ​​of each illegal passenger transport category is taken as the confidence level of illegal passenger transport, and the illegal passenger transport category corresponding to the maximum probability value is taken as the illegal passenger transport category label; Based on the rule constraint vector, a set of illegal triggering criteria is constructed by mapping the constraint response value threshold to the triggering rule item. The set of illegal triggering criteria includes vehicle category triggering criteria, number of people constraint triggering criteria, spatial distribution triggering criteria, and scene legality triggering criteria. The confidence level of illegal passenger transport, the category label of illegal passenger transport, and the set of illegal triggering evidence are used as the results of illegal passenger transport identification.

9. The method for detecting and identifying illegal passenger transport in vehicles based on AI vision according to claim 1, characterized in that, The multi-evidence consistency verification specifically includes: Based on the associated information, the identification results of multiple illegal passenger carrying by the target vehicle within a set time window are associated and merged to obtain a group of illegal passenger carrying results to be verified. Based on the group of illegal passenger transport results to be verified, the consistency of each illegal passenger transport category label is compared to obtain the category consistency verification result; the confidence stability of each illegal passenger transport confidence is compared to obtain the confidence consistency verification result; and the consistency of each illegal triggering basis set is compared to obtain the triggering basis consistency verification result. Based on the association information of the illegal passenger transport result group to be verified, the collection time is compared for temporal continuity, the collection location identifier is compared for spatial proximity, and the device identifier is compared for device association to obtain the consistency verification result of the association information. Among them, the category consistency verification result, confidence consistency verification result, trigger basis consistency verification result, and associated information consistency verification result all include pass and fail status; When the consistency verification results of category, confidence level, trigger basis, and related information are all passed, the evidence images, related information, illegal passenger carrying category labels, illegal passenger carrying confidence level, and illegal trigger basis set of the target passing vehicle are extracted to generate a standard illegal evidence collection result set.

Images