Parking scene pedestrian collision risk early warning method, device, equipment and medium

Abstract

The invention provides a parking scene pedestrian collision risk early warning method, device and equipment and a medium, and belongs to the technical field of vehicle intelligent perception, and the method comprises the steps: obtaining image data and ultrasonic data in a parking area in real time, carrying out the time-space alignment, and determining a pedestrian shielding area in the image data; extracting an original image feature of an area without pedestrian occlusion and a complemented image feature of an area with pedestrian occlusion in the image data; performing data fusion on the original image features of the image data, the complemented image features and the corresponding ultrasonic data under space-time alignment to obtain a detection result of the multi-target pedestrian; and based on the detection results of the multi-target pedestrians in the historical frame and the current frame, predicting a prediction result of the multi-target pedestrians in a future frame, and carrying out real-time collision early warning. Through the technical scheme in the embodiment of the invention, the trajectory of each pedestrian can be predicted under the condition that the pedestrian is shielded, the collision risk can be sensed in advance, and early warning can be made in time.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle intelligent perception, in particular to a pedestrian collision risk early warning method and device in a parking scenario, equipment and medium. BACKGROUND

[0002] With the acceleration of urbanization, the intersection of pedestrians and vehicles in areas such as communities and shopping malls is becoming more and more frequent. Parking, as a high-frequency and low-speed scenario, is prone to collision with suddenly intruding pedestrians (especially children and the elderly) due to large blind spots and focus on parking space operation. The traditional solution relies on the passive observation mode of rearview mirror and reversing image, which is difficult to fully cover the visual dead angle and cannot provide real-time early warning of sudden risks. Under the trend of automobile intelligence and networking, active collision warning solutions based on vehicle-mounted sensors have been widely applied.

[0003] The solution of vehicle-mounted sensors for vehicle surrounding environment perception can detect obstacles in the parking scenario, but this detection capability is more limited to static objects or only makes a warning when a collision is about to occur, and cannot predict the potential collision risk of dynamic pedestrians in advance. Especially during parking, due to changes in vehicle perception angle or occlusion of pedestrians during movement, missed detection and false detection often occur. When the vehicle-mounted sensor detects the pedestrian again, the reaction time left for the driver is very short, which increases the risk of vehicle-pedestrian collision.

[0004] Therefore, it is urgent to propose a more safe and effective collision risk warning solution that can perceive the collision risk in advance and provide timely warning. SUMMARY

[0005] The present application aims to solve at least one of the technical problems in the prior art and proposes a pedestrian collision risk early warning method, device, equipment and storage medium in a parking scenario.

[0006] In a first aspect, the present application provides a pedestrian collision risk early warning method in a parking scenario, comprising:

[0007] Real-time acquisition of image data and ultrasonic data in a parking area and spatiotemporal alignment, based on the corresponding ultrasonic data of the image data under spatiotemporal alignment, region information labeling of pedestrian regions in the image data, and confirmation of the image data with pedestrian occlusion regions;

[0008] Based on the results of region information labeling, the original image features of the pedestrian occlusion regions are completed in structure and semantic features to obtain completed image features. The original image features of the image data without pedestrian occlusion regions and the completed image features of the pedestrian occlusion regions are extracted;

[0009] fuse the original image features, the completed image features and the corresponding ultrasonic data under the spatio-temporal alignment of the image data, and dynamically adjust the weights of the data based on the confidence of each data in a specific spatial region to obtain a detection result of the multiple target pedestrians;

[0010] Based on the detection results of the multiple target pedestrians in the historical frames and the current frame, a trajectory of the multiple target pedestrians is constructed, and the prediction results of the multiple target pedestrians in the future frame are predicted in combination with the pedestrian information of each pedestrian and the cross influence between the pedestrians. Real-time collision warning is performed based on the vehicle position and the prediction results of the multiple target pedestrians in the future frame.

[0011] In some embodiments, the process of spatio-temporal alignment of the image data and the ultrasonic data specifically includes:

[0012] Based on a unified time synchronization mechanism, the timestamps of the image data and the ultrasonic data are recorded respectively;

[0013] For each frame of image data, ultrasonic data with the smallest time difference is matched. If the smallest time difference exceeds a preset threshold, the ultrasonic data is interpolated and filled before matching;

[0014] A unified reference coordinate system is constructed, and the image data in the camera coordinate system and the ultrasonic data in the ultrasonic coordinate system are projected into the unified reference coordinate system.

[0015] In some embodiments, the process of region information labeling of the pedestrian regions in the image data specifically includes:

[0016] All pedestrian regions in the image data are boxed by pedestrian boxes, and the coordinates of each pedestrian box and the pedestrian occlusion degree are obtained to preliminarily label the pedestrian regions in the image data;

[0017] Based on the matched ultrasonic data of the image data, the distance data of all pedestrian regions in the ultrasonic data are obtained to secondarily label the pedestrian regions in the image data.

[0018] In some embodiments, the process of structure and semantic feature completion of the original image features specifically includes:

[0019] Local features and global features in the original image features are extracted in the encoding stage;

[0020] High-level semantic information of the original image features is generated by implicit context reasoning;

[0021] Low-level structure information of the original image features is generated by explicit structure guidance;

[0022] Fusing the low-level detail information and the high-level semantic information in the decoding stage, feature completion is performed on the local features and the global features in the original image features.

[0023] In some embodiments, the process further comprises training the structure and semantic feature completion process using a weakly supervised and occlusion simulation mechanism.

[0024] The process of training the structure and semantic feature completion process specifically comprises:

[0025] In the training process, partial occlusion images are used as input, and complete images are used as supervision signals to construct paired samples.

[0026] The paired samples are expanded using random occlusion and copy-paste occlusion data augmentation methods to generate diverse occlusion samples.

[0027] Based on the diverse occlusion samples, the structure and semantic feature completion process is trained.

[0028] In some embodiments, the process of obtaining the detection result of the multiple target pedestrians specifically comprises:

[0029] Based on the differences in perception ability of the original image features, the completed image features, and the ultrasonic data under different conditions, a corresponding confidence is set for each data under a specific condition.

[0030] Based on the corresponding confidence set for each data under a specific condition, the weight of each data in a specific spatial region is dynamically adjusted.

[0031] Based on the weight of each data in a specific spatial region, a fusion feature map is generated.

[0032] Based on the spatial structure of the fusion feature map, the detection result of the multiple target pedestrians is predicted.

[0033] In some embodiments, the process of predicting the prediction result of the multiple target pedestrians in the future frame specifically comprises:

[0034] Based on the pedestrian trajectories of each target pedestrian, the appearance features and spatial continuity features of each target pedestrian are combined to perform cross-frame target association. For newly appearing or temporarily disappearing target pedestrians, a trajectory ID is assigned and a timestamp is maintained.

[0035] The pedestrian trajectories of a single target pedestrian are modeled in time sequence, the pedestrian trajectories of multiple target pedestrians are modeled in target relationship graph, and the potential interaction relationship between multiple target pedestrians is focused on.

[0036] Based on temporal modeling of a single target pedestrian and target relationship graph modeling of multiple target pedestrians, we can explore the interaction relationship and temporal evolution pattern between pedestrian trajectories and predict the prediction results of multiple target pedestrians in future frames.

[0037] Secondly, embodiments of the present invention provide a pedestrian collision risk warning device for parking scenarios, comprising:

[0038] The perception data acquisition module is used to acquire image data and ultrasonic data in the parking area in real time and perform spatiotemporal alignment. Based on the ultrasonic data corresponding to the image data under spatiotemporal alignment, the module marks the pedestrian area in the image data with regional information and confirms the pedestrian-occluded area in the image data.

[0039] The region feature completion module is used to complete the original image features of the pedestrian-occluded region with structural and semantic features based on the results of region information annotation, so as to obtain the completed image features; and to extract the original image features of the non-pedestrian-occluded region and the completed image features of the pedestrian-occluded region from the image data.

[0040] The multi-target pedestrian detection module is used to fuse the original image features, the completed image features, and the corresponding ultrasonic data under spatiotemporal alignment of the image data, and dynamically adjust the weight of each data based on the confidence of each data in a specific spatial region to obtain the detection result of multi-target pedestrians.

[0041] The pedestrian trajectory prediction module is used to construct multi-target pedestrian trajectories based on the detection results of multi-target pedestrians in historical frames and the current frame, and to predict the prediction results of multi-target pedestrians in future frames by combining the pedestrian information of each pedestrian and the cross-influence between pedestrians; and to perform real-time collision warning based on the vehicle position and the prediction results of multi-target pedestrians in the future frames.

[0042] Thirdly, embodiments of the present invention provide an electronic device, the electronic device comprising:

[0043] At least one processor; and a memory communicatively connected to the at least one processor;

[0044] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the steps of the method according to any embodiment of the present invention.

[0045] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer instructions that are used to cause a processor to execute the steps of any embodiment of the method of the present invention.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] The pedestrian collision risk warning method for parking scenarios provided by this invention first acquires image data and ultrasonic data within the parking area in real time and performs spatiotemporal alignment, identifying areas in the image data that are obscured by pedestrians. Then, it extracts the original image features of areas without pedestrian obscuration and the completed image features of areas with pedestrian obscuration from the image data. Next, it fuses the original image features, completed image features, and the corresponding ultrasonic data under spatiotemporal alignment to obtain the detection results of multiple pedestrian targets. Finally, based on the detection results of multiple pedestrian targets in historical frames and the current frame, it predicts the prediction results of multiple pedestrian targets in future frames, providing real-time collision warnings. Through the technical solution of this invention, the trajectories of each pedestrian can be predicted even when pedestrians are obscured, allowing for earlier detection of collision risks and timely warnings. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only preferred embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart illustrating a pedestrian collision risk warning method for a parking scenario provided by an embodiment of the present invention;

[0050] Figure 2 A flowchart illustrating another pedestrian collision risk warning method for parking scenarios provided by an embodiment of the present invention;

[0051] Figure 3 This is a flowchart illustrating a method for feature completion in areas obscured by pedestrians, provided by an embodiment of the present invention.

[0052] Figure 4 This is a flowchart illustrating a method for multi-target pedestrian detection by fusing multimodal feature data, provided in an embodiment of the present invention.

[0053] Figure 5 This is a flowchart illustrating a method for multi-target pedestrian trajectory prediction based on pedestrian spatial distribution, provided by an embodiment of the present invention.

[0054] Figure 6 This is a structural block diagram of a pedestrian collision risk warning device for parking scenarios provided in an embodiment of the present invention;

[0055] Figure 7 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0058] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0059] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0060] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0061] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0062] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0063] Among related technologies, there are several solutions for pedestrian collision risk warning in parking scenarios:

[0064] Option 1 involves using an in-vehicle surround-view camera to capture video images of the vehicle's surroundings and identify obstacles within those images. Warnings are then issued based on the relative positions of the obstacles and the vehicle. However, Option 1 is more suitable for detecting static obstacles and lacks the ability to anticipate potential risks from moving pedestrians. This can easily lead to missed or false detections in parking scenarios due to pedestrian obstruction.

[0065] Option 2 involves visually tracking multiple dynamic obstacles around the vehicle using both visible light and infrared image features. However, Option 2 struggles to adapt to the changing camera perspective caused by vehicle movement and target loss due to temporary occlusion in parking scenarios.

[0066] In summary, in parking scenarios, changes in the vehicle's perspective or obstruction of pedestrians result in poor image quality and a lack of complete feature information captured by onboard sensors, creating a potential collision risk during parking.

[0067] One of the core aspects of the technical solution in this application is that, considering pedestrian occlusion and image degradation, the future location prediction and collision assessment are combined with the trajectories of pedestrians and vehicles to achieve early warning and proactive prevention of potential pedestrian collisions during parking.

[0068] To address at least one of the technical problems existing in the aforementioned related technologies, the present invention provides a method for warning of pedestrian collision risk in parking scenarios. Figure 1 This is a flowchart illustrating a pedestrian collision risk warning method for parking scenarios provided by an embodiment of the present invention. This method is particularly applicable to situations where there are dense pedestrians and overlapping / obstructed pedestrians in parking scenarios. This method can be executed by a pedestrian collision risk warning device for parking scenarios, which can be implemented in software and / or hardware and can be configured in an electronic device.

[0069] like Figure 1 As shown, the method specifically includes:

[0070] S1: Real-time acquisition of image data and ultrasonic data within the parking area and spatiotemporal alignment; based on the ultrasonic data corresponding to the image data under spatiotemporal alignment, marking of pedestrian areas in the image data and confirming areas in the image data that are obscured by pedestrians.

[0071] In one specific implementation, one or more vehicle-mounted cameras are positioned at the front, front left, front right, rear, rear left, and rear right of the vehicle to continuously acquire image data within the parking area. For example, the resolution of these cameras is set to 1600×900, and the frame rate is set to 15Hz to ensure image detail and temporal continuity. Meanwhile, one or more ultrasonic sensors are distributed at the front left, front right, rear left, and rear right corners of the vehicle to synchronously acquire ultrasonic data within the parking area. Furthermore, the ranging value (in meters) of each ultrasonic sensor in each frame of image data, along with its installation position, orientation, and other metadata, are acquired.

[0072] In some embodiments, the process of spatiotemporal alignment of image data and ultrasound data specifically includes:

[0073] S1101 records the timestamps of image data and ultrasound data respectively, based on a unified time synchronization mechanism.

[0074] Specifically, in the process of spatiotemporal alignment, it is necessary to ensure that the image frames and ultrasonic data correspond at the same moment to achieve time alignment. On the one hand, a unified time synchronization mechanism is used, using the host's internal clock to enable the image acquisition system and the ultrasonic acquisition system to share a unified time source; on the other hand, a timestamp is recorded when each image frame is acquired, and a timestamp is also recorded when the ultrasonic sensor reads it. The ultrasonic sampling frequency is slightly higher than the camera frequency, for example, the ultrasonic sampling frequency is set to 20Hz.

[0075] S1102, matches the ultrasonic data with the smallest time difference for each frame of image data. If the minimum time difference exceeds a preset threshold, the ultrasonic data is interpolated and then matched.

[0076] Specifically, for each frame of image data, the ultrasonic data that is closest in time is found; if the time difference is less than a preset threshold (±50ms), the synchronization is considered successful; if the time difference is large, interpolation can be used to estimate and fill the ultrasonic data.

[0077] S1103, construct a unified reference coordinate system, and project the image data in the camera coordinate system and the ultrasonic data in the ultrasonic coordinate system onto the unified reference coordinate system.

[0078] Specifically, the spatiotemporal alignment process also requires obtaining the spatial geometric relationship (extrinsic parameters) between the ultrasonic sensors and the camera to project their data onto a unified reference coordinate system. First, the vehicle coordinate system is selected as the unified reference coordinate system, and the positions (translation vectors) and orientations (rotation matrices) of the camera and each ultrasonic sensor relative to the vehicle coordinate system are obtained. For the camera, intrinsic and extrinsic parameters can be obtained through traditional methods such as checkerboard calibration. For the ultrasonic sensors, their physical installation positions and angles are recorded. Then, transformation matrices from the ultrasonic coordinate system to the vehicle coordinate system and from the camera coordinate system to the vehicle coordinate system are constructed. Finally, the transformation relationship between the ultrasonic coordinate system and the camera coordinate system is calculated, allowing the ultrasonic data to be projected onto the image to further verify the accuracy of the data.

[0079] In this embodiment, by performing time synchronization and spatial calibration on the image data acquired by the camera and the ultrasonic data acquired by the ultrasonic sensor, the extrinsic parameter transformation matrix between the two is obtained, thereby achieving spatiotemporal alignment of multi-source data.

[0080] In some embodiments, the process of annotating pedestrian regions in image data specifically includes:

[0081] S1201: Select all pedestrian regions in the image data by using pedestrian bounding boxes, obtain the coordinates of each pedestrian bounding box and the degree of pedestrian occlusion, and perform initial annotation of the pedestrian regions in the image data.

[0082] Specifically, after spatiotemporal alignment of image and ultrasonic data, all pedestrian regions in the image data can be identified. These pedestrian regions include: regions with pedestrian occlusion and regions without pedestrian occlusion. Regions with pedestrian occlusion refer to areas where pedestrians are obscured by obstacles or other pedestrians, while regions without pedestrian occlusion refer to areas where pedestrians are present and not obscured by obstacles or other pedestrians. Preliminary annotation of all pedestrian regions in the image data is performed using manual or semi-automatic annotation methods. The preliminary annotation information includes: pedestrian bounding box coordinates, occlusion degree (full occlusion / partial occlusion / no occlusion), etc.

[0083] S1202: Based on the ultrasonic data matched with the image data, obtain the distance data of all pedestrian areas in the ultrasonic data, and perform secondary annotation on the pedestrian areas in the image data.

[0084] Specifically, after the initial labeling of the pedestrian area, a secondary labeling is performed using ultrasonic data matched with the image data. The secondary labeling information includes: the RGB image from the image data, the distance data of the pedestrian area from the ultrasonic data, the sensor ID and spatial location, timestamp information, and pedestrian bounding box information (including bounding boxes, segmentation masks, occlusion markers, etc.). The bounding box (BBox) can select pedestrians within the sensor-captured image using a rectangular bounding box, clearly defining the pedestrian's specific location and range within the image, providing foundational data for subsequent target tracking and risk assessment. The advantages of the bounding box are its fast computation speed and low resource consumption, making it suitable for rapid initial target localization. The segmentation mask can label the pedestrian area pixel by pixel, clearly separating pedestrians from surrounding environmental elements such as trees, railings, and the ground, more accurately capturing pedestrian movements and trajectories. The advantages of the segmentation mask are its high recognition accuracy and strong anti-interference capability, making it suitable for precise perception in complex environments.

[0085] Figure 2 This is a flowchart illustrating another pedestrian collision risk warning method for parking scenarios provided by an embodiment of the present invention, as shown below. Figure 2 As shown, the entire implementation process includes: First, acquiring image data and ultrasonic data through an onboard camera and ultrasonic sensor respectively, establishing temporal synchronization and spatial calibration relationships between the image and ultrasonic data, and projecting the image and ultrasonic data onto a unified reference coordinate system using an extrinsic transformation matrix. Second, filtering the ultrasonic data and extracting image features from the image data. For original image features in areas obscured by pedestrians, feature completion is performed using a Structured Perception Completion Network (STC-Net) to obtain completed image features. The original image features in areas without pedestrian obstruction and the completed image features in areas with pedestrian obstruction are then stitched together. Third, pedestrian detection is performed using a Multimodal Perception Network (MP-Transformer) in a unified reference coordinate system, combining the filtered ultrasonic data and the stitched image data. Fourth, the future location of pedestrians is predicted using a Trajectory Interaction Network (TI-Transformer), ultimately leading to risk assessment and parking warning.

[0086] S2, based on the results of region information annotation, the original image features of the pedestrian-occluded region are completed with structural and semantic features to obtain the completed image features; the original image features of the unoccluded region and the completed image features of the pedestrian-occluded region are extracted from the image data.

[0087] Specifically, during the feature extraction process, a structure-aware completion network is designed and trained to complete the structural and semantic features of the original image features in areas without pedestrian occlusion. Finally, the original image features in areas without pedestrian occlusion and the completed image features in areas with pedestrian occlusion are spliced ​​together to generate complete image features.

[0088] Figure 3 This is a flowchart illustrating a method for feature completion in areas obscured by pedestrians, as provided in an embodiment of the present invention. Figure 3 As shown, the entire scheme is implemented using a structure-aware completion network. The implementation process includes: First, obtaining the standard dimensional format of the original RGB color image of the image with pedestrian occlusion, including RGB three channels, image height, and image width. Second, extracting local and global features of the image through an encoder (combining multi-layer convolution and downsampling). Third, completing the semantic features of the pedestrian-occluded region through a semantic feature branch (combining dilated convolution and dilated spatial pyramid pooling) to bridge the semantic differences between the occluded region and its context, thus maintaining the semantic consistency of the occluded target during the completion process. Fourth, completing the structural features of the pedestrian-occluded region through a structural feature branch (combining convolutional neural networks and edge detection) to maintain the structural coherence of the occluded target during the completion process. Fifth, fusing features through a feature fusion module (combining attention mechanisms and feature concatenation). Sixth, fusing low-level detail features with high-level semantic information through a decoder (combining upsampling and skip connections) to enhance reconstruction quality. Seventh, completing the feature map.

[0089] In some embodiments, the process of performing structural and semantic feature completion on the original image features specifically includes:

[0090] S2101, extracts local and global features from the original image features during the encoding stage.

[0091] S2102 generates high-level semantic information of the original image features through implicit contextual reasoning.

[0092] S2103 generates low-level structural information of the original image features through explicit structural guidance.

[0093] S2104 integrates low-level detail information and high-level semantic information during the decoding stage to complete the local and global features in the original image.

[0094] Specifically, for partially occluded pedestrian regions in an image (regions with pedestrian occlusion), a specialized structure-aware completion network is designed to restore their complete structural and semantic information. The overall network architecture of the structure-aware completion network adopts an encoder-decoder structure. In the encoding stage, local and global features of the image are extracted progressively, and in the decoding stage, spatial resolution is restored layer by layer to achieve structural reconstruction of the occluded region. The encoder consists of multiple stacked convolutional blocks, and residual connections are introduced to improve feature extraction capabilities. The decoder uses transposed convolutions and skip connections to fuse low-level structural information with high-level semantic information, thereby enhancing reconstruction quality.

[0095] Furthermore, a structure-aware module is introduced into the structure-aware completion network to enhance the network's understanding and modeling of occlusion structures. A non-local attention mechanism is used to capture long-range dependencies in image data of pedestrian-occluded regions, generating high-level semantic information of the original image features and bridging the semantic differences between the occluded region and its context. Long-range dependencies refer to the associations between distant pixels in the image, such as the positional relationship between distant pedestrians and vehicles, or the scene associations between scattered pedestrians. The structure-aware module uses a combination of explicit structural guidance (such as edges and poses) and implicit contextual reasoning to guide the network to maintain the structural coherence and semantic consistency of the occluded target during the completion process. The structure-aware module is placed in the mid-to-high-level stages of the encoder, enabling the network to fully integrate low-level structural information with high-level semantic information.

[0096] In some embodiments, the method further includes: training the process of structural and semantic feature completion using a weak supervision and occlusion simulation mechanism.

[0097] The training process for structural and semantic feature completion specifically includes:

[0098] S2201 uses partially occluded images as input and complete images as supervision signals during training to construct paired samples.

[0099] S2202 uses random occlusion and copy-paste occlusion data augmentation techniques to expand paired samples and generate diverse occluded samples.

[0100] S2203 is trained on the process of structural and semantic feature completion based on diverse occlusion samples.

[0101] To improve the model's generalization ability in real-world occluded scenarios, a weak supervision mechanism and an occlusion simulation mechanism are introduced during training. Furthermore, edge-preserving loss, perceptual loss, and adversarial loss are incorporated to guide the model in generating realistic and structurally clear completed regions from multiple dimensions. After training, the model receives occluded image input during the inference phase and outputs a high-dimensional feature map of the completed image. These features not only contain semantic completion information of the occluded region but also possess a complete structural representation of the pedestrian, providing semantically enhanced input for subsequent multimodal perception networks and effectively improving the accuracy and robustness of pedestrian detection and tracking.

[0102] The technical solution in this embodiment proposes a structure-aware occlusion completion network, specifically designed for completing the structural and semantic features of occluded pedestrian areas in parking scenarios. By automatically recovering the complete pedestrian features of occluded areas through deep learning, the completeness and accuracy of pedestrian detection are improved, effectively solving the problem of information loss caused by occlusion.

[0103] S3 performs data fusion on the original image features, the completed image features, and the corresponding ultrasonic data under spatiotemporal alignment of the image data, and dynamically adjusts the weight of each data based on the confidence of each data in a specific spatial region to obtain the detection results of multi-target pedestrians.

[0104] Specifically, during the data fusion process, the original image features, the completed image features, and the ultrasonic data are input into a multimodal perception network for dynamic weighted fusion. Simultaneously, an attention mechanism is introduced into the multimodal perception network to achieve dynamic weighting of the multimodal feature data, thereby completing pedestrian detection in occluded environments.

[0105] Figure 4 This is a flowchart illustrating a method for multi-target pedestrian detection by fusing multimodal feature data, as provided in an embodiment of the present invention. Figure 4 As shown, the entire scheme is implemented using a multimodal perception network. The implementation process includes: First, inputting multimodal feature data into a specific encoder to extract embedded representations. Embedded representations are the core bridge connecting explicit structural features, implicit contextual information, and the network model, transforming different types of features into a vector form that can be efficiently processed by a computer while preserving key semantic and structural information. Second, aligning image features and completed features spatially using a feature alignment module (fully connected layer) and concatenating them, while introducing encoded ultrasonic data as a third modality input to form a fused feature sequence. Third, assigning weights to the inputs of the three modalities by introducing a multi-head self-attention mechanism in the cross-modal attention module. Fourth, performing data fusion based on the weights of each modality input. Fifth, detecting multiple pedestrian targets using a detection head and outputting the detection results.

[0106] In some embodiments, the process of obtaining the detection results of multiple pedestrian targets specifically includes:

[0107] S3101 sets a corresponding confidence level for each data point under specific conditions based on the differences in perception capabilities of the original image features, the completed image features, and the ultrasonic data under different conditions.

[0108] S3102, dynamically adjusts the weight of each data point in a specific spatial region based on the corresponding confidence level set under specific conditions.

[0109] S3103 generates a fused feature map based on the weights of each data point in a specific spatial region.

[0110] S3104 predicts the detection results of multiple pedestrians based on the spatial structure of fused feature maps.

[0111] The goal of this embodiment is to build a unified multimodal perception network framework that deeply fuses original image features, completed image features, and ultrasonic data to improve pedestrian detection accuracy in occluded environments. The multimodal perception network takes the image domain as its primary perspective. First, it extracts image features (original image features) from the image data through a backbone convolutional network. These features are then aligned spatially with the completed features (complete image features) generated by a structure-aware completion network and concatenated. Simultaneously, the ultrasonic data is mesh-encoded and converted into a tensor of the same dimension as the image features, serving as a third modality input to form a fused feature sequence.

[0112] A fusion module is introduced into the multimodal perception network. To effectively mine the complementary information relationships between feature data from different modalities, a multi-head self-attention mechanism is incorporated into the fusion module. This mechanism takes the fused feature sequence as input and dynamically adjusts the weights of different modal features in specific spatial regions by calculating the correlation between Query, Key, and Value. This suppresses redundant information and strengthens key clues. The attention mechanism possesses position independence and global modeling capabilities, enabling it to capture the interaction features between image context, supplementary information, and near-field detection (ultrasound) in complex occluded environments, effectively improving detection accuracy.

[0113] A modality adaptation and selection module is also introduced into the multimodal perception network. Considering the differences in perception capabilities of different sensors under different conditions (for example, ultrasound has a strong ability to perceive the distance of occluded targets, while image features are better for semantic understanding), the multimodal perception network specifically designs a modality adaptation module. This module learns the confidence levels of different modal feature data through a guided attention mechanism and dynamically adjusts their fusion weights to avoid weak modalities interfering with the overall recognition results. Simultaneously, a DropMod strategy is introduced during the training phase, which randomly occludes a certain modality in some iterations, forcing the model to learn redundant and collaborative representations of each modality, thereby enhancing the network's robustness and anti-interference ability.

[0114] A lightweight detection head is introduced into the multimodal perception network. The final pedestrian detection result is output through a unified detection head, while the fused feature map output by the fusion module is fed into the lightweight detection head to complete multi-target pedestrian detection. The detection head predicts the target category and bounding box information based on the spatial structure of the fused feature map. During training, multi-task loss functions, including FocalLoss and GIoU Loss, are used to guide the network in optimizing classification and localization performance.

[0115] The technical solution in this embodiment proposes a multimodal perception network that can fuse original image features, completed image features, and ultrasonic data. By dynamically weighting and fusing multi-source information through an adaptive attention mechanism, accurate pedestrian detection is achieved in complex occlusion environments, improving the robustness and reliability of the perception system.

[0116] S4. Based on the detection results of multiple pedestrians in historical frames and the current frame, construct the pedestrian trajectory of multiple pedestrians; based on the pedestrian trajectory of multiple pedestrians, combine the pedestrian information of multiple pedestrians and the cross-influence between multiple pedestrians, predict the prediction result of multiple pedestrians in future frames; based on the current position of the vehicle and the prediction result of multiple pedestrians in future frames, provide real-time warning.

[0117] Specifically, pedestrian trajectories are constructed based on the detection results of the current frame and historical frames, and the future positions of pedestrians are predicted using a trajectory interaction network. Pedestrian information refers to the explicit and implicit information of all pedestrians in the pedestrian region that can be directly obtained or indirectly inferred from the fused feature map, including at least the historical position sequence, velocity vector, and timestamp of each pedestrian.

[0118] Figure 5 This is a flowchart illustrating a method for multi-target pedestrian trajectory prediction based on pedestrian spatial distribution, as provided in an embodiment of the present invention. Figure 5As shown, the entire scheme is implemented using a trajectory interaction network. The implementation process includes: First, inputting the position information of the target pedestrian in multiple past frames, and generating a trajectory feature sequence through a trajectory embedding layer. Second, constructing a pedestrian interaction graph through a graph structure construction module. Third, focusing on targets with potentially close interaction relationships through a graph neural network layer. Fourth, introducing a multi-head attention mechanism in the encoding layer to output the pedestrian position in future multiple frames. Fifth, introducing a multilayer perceptron regression through a trajectory prediction head to output the future trajectory coordinate sequence of the pedestrian.

[0119] In some embodiments, the process of predicting the prediction results of multiple pedestrians in a future frame specifically includes:

[0120] S4101: Based on the pedestrian trajectory of each target pedestrian, and combined with the appearance features and spatial continuity features of each target pedestrian, cross-frame target association is performed; for newly appearing or temporarily disappearing target pedestrians, trajectory IDs are assigned and timestamps are maintained.

[0121] S4102 performs temporal modeling of the pedestrian trajectory of a single target pedestrian, performs target relationship graph modeling of the pedestrian trajectories of multiple target pedestrians, and focuses on the potential interaction relationships between multiple target pedestrians.

[0122] S4103, based on the temporal modeling of a single target pedestrian and the target relationship graph modeling of multiple target pedestrians, mines the interaction relationship and temporal evolution law between pedestrian trajectories and predicts the prediction results of multiple target pedestrians in future frames.

[0123] In the trajectory interaction network, multi-target trajectories are constructed based on temporal detection results to predict future pedestrian behavior in occluded scenes. Stable pedestrian trajectories are built using a data association algorithm based on the detection results of the current frame and cached historical frames. Specifically, for each frame's detection box, information such as its center point coordinates and velocity direction is extracted. Combined with pedestrian appearance features (encoded by the visual network) and spatial continuity features, Hungarian matching is used for cross-frame target association. For newly appearing or temporarily disappearing pedestrians, the system automatically assigns trajectory IDs and maintains timestamps, thereby constructing a cross-temporal, multi-target pedestrian trajectory sequence, providing continuous input for trajectory prediction.

[0124] In the trajectory interaction network, a temporal modeling network is also designed to capture motion trends, encode and model the constructed historical trajectories, and explore the interaction relationships and temporal evolution patterns between trajectories. Pedestrian information, including the historical position sequence, velocity vector, and timestamp for each pedestrian, is input into the network and encoded into a fixed-dimensional trajectory feature sequence through a trajectory embedding layer before being fed into the main network. A multi-head temporal self-attention mechanism is employed to model the temporal dependence of single targets, while an inter-trajectory interaction attention module is used to explicitly model the cross-influence between multiple targets, thereby improving the accuracy and robustness of predictions.

[0125] In the trajectory interaction network, target relationship modeling is introduced to improve prediction accuracy. This involves not only temporal modeling of individual trajectories but also a dedicated target relationship graph modeling module. Based on the pedestrian spatial distribution of the current frame, this module constructs a graph structure representation of targets, with edge weights defined by distance, relative velocity, or occlusion relationships. By embedding adjacency information from this graph structure, a graph attention mechanism guides the trajectory prediction module to focus on targets with potentially close interactions, thus avoiding the problem of slow response to changes in the surrounding environment. This effectively handles common problems in complex parking lots, such as dense pedestrian traffic and occlusion / intersection, improving prediction accuracy.

[0126] In the trajectory interaction network, multi-step future position prediction and model optimization are also introduced to output the pedestrian position in the next few frames. The mean squared error loss function is used during training, and data augmentation strategy is combined to improve the model's adaptability to occlusion and sudden behavior. The prediction results are used for real-time collision risk assessment.

[0127] Furthermore, by combining the vehicle's current location with the predicted trajectory, a collision risk assessment is conducted, and a parking warning mechanism is triggered when the risk value exceeds a set threshold.

[0128] In some embodiments, the real-time warning process includes: mapping the positional relationship between the pedestrian and the vehicle to a reference coordinate system based on the vehicle's current positioning information and the predicted future trajectory of the pedestrian; calculating the spatial distance and movement trend between the pedestrian and the vehicle in real time; and dynamically assessing potential collision risks.

[0129] In some embodiments, the real-time warning process further includes: defining risk assessment indicators and risk thresholds, designing a multi-dimensional risk indicator system, and calculating a risk score based on comprehensive weights. The multi-dimensional risk indicator system includes minimum distance, relative speed, and differences in movement direction. Risk thresholds are reasonably set according to the characteristics of parking scenarios to ensure the timeliness and accuracy of warnings and reduce false alarms and missed alarms.

[0130] In some embodiments, the real-time warning process further includes activating a parking collision warning when the risk score exceeds a risk threshold. The parking collision warning includes audible and visual alarms and in-vehicle display prompts. This assists the driver in taking timely evasive action upon receiving the warning. Simultaneously, it supports tiered processing of warning levels to achieve multi-layered safety protection.

[0131] In some embodiments, the real-time warning process further includes: receiving the latest vehicle status and pedestrian information in real time, and dynamically adjusting the risk threshold. Historical data is used for adaptive model optimization to improve the system's ability to identify collision risks and its response speed in complex dynamic environments.

[0132] The technical solution in this embodiment proposes a trajectory interaction network that constructs temporal trajectories based on pedestrian detection results from current and historical frames, accurately predicting the future positions of multiple pedestrians. Addressing the complexity of pedestrian movement in parking scenarios, it effectively captures trajectory interaction information, improving the timeliness and accuracy of warnings. Simultaneously, a collision risk assessment and warning mechanism integrating vehicle positioning is proposed. Combining the vehicle's current position and predicted trajectory, a collision risk assessment model is established to achieve real-time dynamic monitoring of potential collision risks. When the risk value exceeds a preset threshold, the system automatically triggers a parking collision warning, effectively ensuring the safety of pedestrians and vehicles and improving the safety management level of the parking environment.

[0133] The technical solution in this embodiment of the invention firstly achieves effective completion of pedestrian structure and semantic features in occluded environments through multimodal information fusion, improving the accuracy and completeness of pedestrian detection. Secondly, by combining original image features, completed image features, and ultrasonic data, and using an attention mechanism for dynamic weighted fusion, the system's robust perception capability for occluded pedestrians in complex parking scenarios is enhanced, ensuring stable and reliable detection results. Then, based on a trajectory interaction network, the future position of pedestrians is accurately predicted, potential collision risks are identified in advance, and the level of safety warning during parking is significantly improved. Finally, through a risk assessment mechanism, vehicle positioning and pedestrian trajectory information are fused in real time to achieve dynamic monitoring and intelligent warning of collision risks, effectively ensuring the safety of vehicles and pedestrians and supporting batch practical deployment.

[0134] Based on the same inventive concept, embodiments of the present invention also provide a pedestrian collision risk warning device for parking scenarios. Figure 6 This is a structural block diagram of a pedestrian collision risk warning device for parking scenarios provided in an embodiment of the present invention, as shown below. Figure 6 As shown, the device specifically includes:

[0135] The perception data acquisition module 100 is used to acquire image data and ultrasonic data in the parking area in real time and perform spatiotemporal alignment. Based on the ultrasonic data corresponding to the image data under spatiotemporal alignment, the module marks the pedestrian area in the image data and confirms the pedestrian occlusion area in the image data.

[0136] The region feature completion module 200 is used to complete the original image features of the pedestrian-occluded region with structural and semantic features based on the results of region information annotation, so as to obtain the completed image features; and to extract the original image features of the non-pedestrian-occluded region and the completed image features of the pedestrian-occluded region from the image data.

[0137] The multi-target pedestrian detection module 300 is used to fuse the original image features, the completed image features, and the corresponding ultrasonic data under spatiotemporal alignment of the image data, and dynamically adjust the weight of each data based on the confidence of each data in a specific spatial region to obtain the detection results of multi-target pedestrians.

[0138] The pedestrian trajectory prediction module 400 is used to construct multi-target pedestrian trajectories based on the detection results of multi-target pedestrians in historical frames and the current frame, and to predict the prediction results of multi-target pedestrians in future frames by combining the pedestrian information of each pedestrian and the cross-influence between pedestrians; and to perform real-time collision warning based on the vehicle position and the prediction results of multi-target pedestrians in future frames.

[0139] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 7 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 7 As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement a pedestrian collision risk warning method for any parking scenario as described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0140] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (BUS).

[0141] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0142] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0143] This invention also provides a computer-readable medium. The computer-readable medium stores a computer program, which, when executed by a processor, implements the steps in any of the parking scenario pedestrian collision risk warning methods described in the above embodiments. The computer-readable storage medium can be volatile or non-volatile.

[0144] This invention also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described pedestrian collision risk warning method for parking scenarios.

[0145] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0146] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0147] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0148] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0149] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0150] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0151] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0152] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0154] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A method for pedestrian collision risk warning in a parking scenario, characterized in that, The method comprises the following steps: real-time acquisition of image data and ultrasonic data in a parking area and spatio-temporal alignment, region information labeling of pedestrian regions in the image data based on corresponding ultrasonic data of the image data under spatio-temporal alignment, and confirmation of the pedestrian occlusion regions in the image data; based on the results of region information labeling, structural and semantic feature completion of original image features of the pedestrian occlusion regions, extraction of original image features of non-pedestrian occlusion regions in the image data and completed image features of the pedestrian occlusion regions; data fusion of original image features, completed image features of the image data and corresponding ultrasonic data under spatio-temporal alignment, dynamic adjustment of weights of each data based on confidence of each data in a specific spatial region, and detection results of multiple target pedestrians; based on detection results of multiple target pedestrians in historical frames and current frames, construction of multiple target pedestrian trajectories, prediction of prediction results of multiple target pedestrians in future frames in combination with pedestrian information of each pedestrian and cross influence between pedestrians; real-time collision warning based on vehicle position and prediction results of multiple target pedestrians in the future frames.

2. The method of claim 1, wherein, The process of spatio-temporal alignment of the image data and the ultrasonic data comprises the following steps: recording of time stamps of the image data and the ultrasonic data based on a unified time synchronization mechanism; matching of ultrasonic data with the smallest time difference for each frame of image data, interpolation filling of the ultrasonic data and then matching if the smallest time difference exceeds a preset threshold; projection of image data in a camera coordinate system and ultrasonic data in an ultrasonic coordinate system into a unified reference coordinate system.

3. The method of claim 1, wherein, The process of region information labeling of pedestrian regions in the image data comprises the following steps: all pedestrian regions in the image data are selected by pedestrian boxes, coordinates of each pedestrian box and pedestrian occlusion degree are obtained, and the pedestrian regions in the image data are labeled for the first time; distance data of all pedestrian regions in the ultrasonic data are obtained based on the matched ultrasonic data of the image data, and the pedestrian regions in the image data are labeled for the second time.

4. The method of claim 1, wherein, The process of structural and semantic feature completion of the original image features comprises the following steps: local features and global features in the original image features are extracted in an encoding stage; high-level semantic information of the original image features is generated by implicit context reasoning; low-level structural information of the original image features is generated by explicit structure guidance; local features and global features in the original image features are completed by fusion of the low-level detail information and the high-level semantic information in a decoding stage.

5. The method of claim 1, wherein, The method further comprises the following steps: training of the process of structural and semantic feature completion by using a weak supervision and occlusion simulation mechanism; The process of training of the process of structural and semantic feature completion comprises the following steps: in the training process, part of the occlusion image is taken as input, and the complete image is taken as a supervision signal to construct a paired sample. The paired samples are expanded by using a data enhancement method of random occlusion and copy-paste occlusion to generate diversified occlusion samples; Based on the diversified occlusion samples, a process of structure and semantic feature completion is trained.

6. The method of claim 1, wherein, The process of obtaining the detection result of the multiple target pedestrians specifically includes: Based on the differences in the perception abilities of the original image features, the completed image features and the ultrasonic data under different conditions, a corresponding confidence is set for each data under a specific condition; Based on the corresponding confidence set for each data under a specific condition, the weight of each data in a specific spatial region is dynamically adjusted; Based on the weight of each data in a specific spatial region, a fused feature map is generated; Based on the spatial structure of the fused feature map, the detection result of the multiple target pedestrians is predicted.

7. The method of claim 1, wherein, The process of predicting the prediction result of the multiple target pedestrians in the future frame specifically includes: Based on the pedestrian trajectories of each target pedestrian, cross-frame target association is performed in combination with the appearance features and spatial continuity features of each target pedestrian; for a newly appearing or temporarily disappearing target pedestrian, a trajectory ID is assigned and a timestamp is maintained; The pedestrian trajectories of a single target pedestrian are time-series modeled, the pedestrian trajectories of multiple target pedestrians are target relationship graph modeled, and the potential interaction relationship among the multiple target pedestrians is focused on; Based on the time-series modeling of a single target pedestrian and the target relationship graph modeling of multiple target pedestrians, the interaction relationship and time-series evolution law among the pedestrian trajectories are mined to predict the prediction result of the multiple target pedestrians in the future frame.

8. A pedestrian collision risk warning device for a parking scenario, characterized in that The device is configured to implement the method of any one of claims 1-7, and the device includes: An awareness data acquisition module, configured to acquire image data and ultrasonic data in a parking area in real time and perform spatio-temporal alignment, perform region information labeling on pedestrian regions in the image data based on the ultrasonic data corresponding to the image data under spatio-temporal alignment, and confirm the pedestrian occlusion regions in the image data; A region feature completion module, configured to perform structure and semantic feature completion on original image features of the pedestrian occlusion regions based on the results of the region information labeling to obtain completed image features, and extract original image features of non-pedestrian occlusion regions in the image data and the completed image features of the pedestrian occlusion regions; A multiple target pedestrian detection module, configured to perform data fusion on the original image features, the completed image features of the image data and the ultrasonic data corresponding thereto under spatio-temporal alignment, dynamically adjust the weights of the data based on the confidence of each data in a specific spatial region, and obtain the detection result of the multiple target pedestrians; A pedestrian trajectory prediction module, configured to construct multiple target pedestrian trajectories based on the detection results of the multiple target pedestrians in historical frames and a current frame, predict the prediction result of the multiple target pedestrians in a future frame in combination with the pedestrian information of each pedestrian and the cross-influence among the pedestrians, and perform real-time collision warning based on the vehicle position and the prediction result of the multiple target pedestrians in the future frame.

9. An electronic device, comprising: The electronic device includes: at least one processor, and a memory connected in communication with the at least one processor; The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the steps of the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing a processor to implement the steps of the method of any one of claims 1-7 when executed.

Images