Vehicle trunk automatic opening method and device and computer program product

Abstract

The invention discloses a vehicle trunk automatic opening method and device and a computer program product, and the method comprises the steps: firstly, after the device fingerprint verification of a target vehicle is passed, obtaining N paths of surrounding video streams through a whole vehicle panoramic camera composed of N vehicle-mounted cameras; and then pedestrian detection and cross-shot pedestrian tracking processing are carried out on each frame of image in the N paths of video streams to obtain a pedestrian detection result and a pedestrian tracking result. Generating a BEV pedestrian trajectory according to a pedestrian tracking result; a target pedestrian BEV track is screened out from the target pedestrian BEV track; and after the identity verification of the target pedestrian is passed, obtaining a state diagram of the target pedestrian according to a pedestrian detection result, inputting the BEV track of the target pedestrian, the state diagram of the target pedestrian and the scene text information of the landmark building within the preset range around the target vehicle into a VLM in combination with prompt, and when the VLM judges that the target pedestrian has the intention of opening the trunk, opening the trunk. And the trunk is automatically opened, so that the safety of the target vehicle and the vehicle use experience of the target pedestrian are improved.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and in particular to a method, device and computer program product for automatically opening a vehicle trunk. Background Technology

[0002] With the improvement of people's living standards and the rapid development of the social economy, more and more cars have entered people's lives, and people's requirements for the convenience and comfort of vehicles are also getting higher and higher. As an important part of car use, the car trunk has a significant impact on the user's convenience, and its performance and ease of use largely determine the user experience.

[0003] Currently, existing methods for assisting with opening vehicle trunks typically fall into two categories: one is sensor-based, which relies on a physical contact trigger mechanism and is susceptible to environmental interference in practical use. For example, mud or dirt on the vehicle surface may cause false triggering of the sensors. Furthermore, this method requires high-precision sensor hardware, resulting in higher costs, and it cannot effectively differentiate between different user access permissions, posing certain security risks. The second method is vision-based, which requires the user to actively perform a preset action (such as a leg sweep) as a trigger condition. This artificially set interaction method violates the principles of natural interaction, especially when the user's hands are occupied, leading to a poor user experience and potential security risks. Additionally, this method is limited by the camera's field of view and lighting conditions, creating blind spots that may result in missed or false detections. Clearly, both existing methods for assisting with opening vehicle trunks have significant problems and negatively impact the user experience. Summary of the Invention

[0004] The main objective of this application is to provide a method, device, and computer program product for automatically opening a vehicle trunk. This method utilizes an external camera array to coordinate continuous user identity verification, dynamic tracking of movement trajectory, and intelligent analysis of behavioral intent, thereby achieving seamless automatic opening of the vehicle trunk. This not only significantly reduces the probability of false triggering and hardware costs but also improves vehicle safety and the user's driving experience.

[0005] This application provides a method for automatically opening a vehicle trunk, including:

[0006] When the target vehicle is in a parked and locked state, if the device fingerprint verification of the target vehicle is successful, the target vehicle is unlocked, and the N video streams around the target vehicle are obtained by using the N on-board cameras that make up the whole vehicle panoramic camera; where N is a positive integer greater than 0.

[0007] Pedestrian detection and cross-camera pedestrian tracking are performed on each frame of the N video streams to obtain pedestrian detection results and pedestrian tracking results;

[0008] Based on the pedestrian tracking results, generate BEV pedestrian trajectories from a bird's-eye view; and filter out the target pedestrian BEV trajectories from them.

[0009] The identity of the target pedestrian is verified using images of the target pedestrian contained in the N video streams; and after the verification is successful, the state map of the target pedestrian is obtained based on the pedestrian detection results.

[0010] The BEV trajectory of the target pedestrian, the state diagram of the target pedestrian, and the scene text information corresponding to landmark buildings within a preset range around the target vehicle, combined with the prompt instruction, are input into a preset visual-language model (VLM) to obtain the model output a determination result on whether the target pedestrian intends to open the trunk of the target vehicle; and when it is determined that the target pedestrian intends to open the trunk of the target vehicle, the trunk of the target vehicle is automatically opened.

[0011] In one possible implementation, the pedestrian detection result is determined as follows:

[0012] The pedestrian detection model is used to predict the first frame images of the N video streams to obtain the prediction results of the detection boxes for the pedestrian's type in the N first frame images.

[0013] Using the predicted detection bounding boxes for pedestrian types, the corresponding N first-frame images are matted to obtain the matted images of the N first-frame images, which are used as pedestrian detection results.

[0014] In one possible implementation, the pedestrian tracking result is determined as follows:

[0015] The matted images of the N first-frame images are input into the ReID feature extractor for feature extraction to obtain the matted features of the N first-frame images;

[0016] Calculate the cosine similarity between the matting features of different first-frame images in the N streams, and select the highest cosine similarity. Based on the unique identifier of the pedestrian, construct the feature dictionary corresponding to the first-frame image in the N streams.

[0017] Repeat the aforementioned steps of generating feature dictionaries, replacing the first frame image with the second frame image to form the feature dictionary corresponding to the second frame image in the N video streams. Based on the feature dictionary corresponding to the first frame image and the feature dictionary corresponding to the second frame image, the K-nearest neighbor algorithm is used to determine the association dictionary. Similarly, after calculating the association dictionary for multiple frames in the N video streams, the tracking result for each pedestrian is determined.

[0018] In one possible implementation, generating the bird's-eye view BEV pedestrian trajectory based on the pedestrian tracking results includes:

[0019] Based on the intrinsic parameters of the vehicle-mounted cameras consisting of N onboard cameras on the target vehicle, the pedestrian tracking results are mapped to a trajectory in three-dimensional space.

[0020] By projecting the pedestrian tracking results onto a three-dimensional trajectory in BEV space, and then onto a two-dimensional plane, the BEV pedestrian trajectory is obtained.

[0021] In one possible implementation, the step of verifying the identity of the target pedestrian using images of the target pedestrian contained in the N video streams includes:

[0022] The target pedestrian's face is recognized by using images of the target pedestrian contained in the N video streams, and the face recognition result is used to verify the identity of the target pedestrian.

[0023] In one possible implementation, the preset visual-language model (VLM) includes a visual encoder, a projection layer, and a large language model (LLM). The process involves inputting the target pedestrian's BEV trajectory, the target pedestrian's state map, and scene text information corresponding to landmarks within a preset range around the target vehicle, combined with a prompt instruction, into the preset visual-language model (VLM). The result obtained is the model's output determining whether the target pedestrian intends to open the target vehicle's trunk, including:

[0024] The BEV trajectory of the target pedestrian and the state map of the target pedestrian are stitched together to obtain a stitched hypermap; the hypermap is then input into the visual encoder for feature extraction to obtain image encoding features.

[0025] The image coding features are input into the projection layer for dimensionality reduction to obtain the dimensionality-reduced image coding features.

[0026] The scene text information corresponding to landmark buildings within a preset range around the target vehicle is encoded to obtain text encoding features;

[0027] The reduced image encoding features and the text encoding features are input into the LLM through a combination of system prompt instructions and traction information to obtain the model output a determination result on whether the target pedestrian intends to open the trunk of the target vehicle.

[0028] In one possible implementation, the device fingerprint of the target vehicle includes a unique identifier of at least one of the following hardware: Bluetooth key, Near Field Communication (NFC), and Ultra Wideband (UWB).

[0029] In one possible implementation, the value of N is 8; the vehicle-mounted panoramic camera consisting of N vehicle-mounted cameras on the target vehicle includes two vehicle-mounted cameras located in front of the target vehicle, two vehicle-mounted cameras located in front left and front right of the target vehicle, two vehicle-mounted cameras located in rear left and rear right of the target vehicle, and two vehicle-mounted cameras located in rear view of the target vehicle.

[0030] This application embodiment also provides a vehicle trunk automatic opening device, including: a processor, a memory, and a system bus;

[0031] The processor and the memory are connected via the system bus;

[0032] The memory is used to store one or more programs, the one or more programs including instructions, which, when executed by the processor, cause the processor to perform any of the above-described implementations of the automatic opening method for the vehicle trunk.

[0033] This application also provides a computer program product, which, when run on a terminal device, causes the terminal device to execute any of the above-described methods for automatically opening the vehicle trunk.

[0034] This application provides a method, device, and computer program product for automatically opening a vehicle trunk. First, when the target vehicle is in a parked and locked state, if the fingerprint verification of the target vehicle is successful, the target vehicle is unlocked, and N video streams around the target vehicle are acquired using a panoramic camera consisting of N on-board cameras on the target vehicle, where N is a positive integer greater than 0. Then, pedestrian detection and cross-camera pedestrian tracking are performed on each frame of the N video streams to obtain pedestrian detection results and pedestrian tracking results. Next, based on the pedestrian tracking results, a bird's-eye view (BEV) pedestrian trajectory is generated; the target pedestrian's BEV trajectory is then selected from this trajectory; the target pedestrian's identity is verified using images of the target pedestrian from N video streams; after successful verification, the target pedestrian's state map is obtained based on the pedestrian detection results; the target pedestrian's BEV trajectory, the target pedestrian's state map, and the scene text information corresponding to landmarks within a preset range around the target vehicle, combined with a prompt instruction, are input into a preset Vision-Language Model (VLM) to obtain the model's output determining whether the target pedestrian intends to open the target vehicle's trunk; and when it is determined that the target pedestrian intends to open the target vehicle's trunk, the target vehicle's trunk is automatically opened.

[0035] As can be seen, this application utilizes a full-vehicle panoramic camera system composed of N onboard cameras on the target vehicle to acquire N video streams around the target vehicle as a basis for judgment. Combined with VLM (Virtual Machine Model), it achieves coordinated processing of continuous user identity verification, dynamic tracking of movement trajectory, and intelligent analysis of behavioral intent. When it is determined that the target pedestrian has the intent to open the target vehicle's trunk, the trunk of the target vehicle is automatically opened. Compared with traditional sensor-based assisted trunk opening solutions, this not only reduces the dependence on high-precision sensor hardware and significantly reduces costs, but also, after the target vehicle's device fingerprint verification is passed, this application superimposes continuous verification of the pedestrian's identity, thereby effectively distinguishing the operating permissions of different pedestrians and eliminating security risks.

[0036] Compared to traditional vision-based trunk opening assistance solutions, this application not only eliminates the need for pedestrians to actively perform preset actions (such as leg sweeping), but also accurately determines the pedestrian's intention to open the vehicle's trunk through VLM, improving convenience and user experience. Furthermore, this application constructs a comprehensive monitoring network based on a combination of full-vehicle panoramic camera shooting, pedestrian detection, and cross-camera pedestrian tracking, ensuring that the needs of pedestrians can be captured in real time under any conditions. This eliminates monitoring blind spots (such as those limited by camera field of view and lighting conditions in vision-based trunk opening assistance solutions), significantly reducing the probability of false triggers and improving vehicle safety and pedestrian user experience. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A flowchart illustrating a method for automatically opening a vehicle trunk, provided as an embodiment of this application;

[0039] Figure 2 A diagram illustrating the device fingerprint verification process for the target vehicle provided in this application embodiment;

[0040] Figure 3 An example diagram showing the field of view of a full-vehicle panoramic camera consisting of eight onboard cameras provided in an embodiment of this application;

[0041] Figure 4 An example diagram illustrating the process of pedestrian detection and cross-camera pedestrian trajectory tracking provided in the embodiments of this application;

[0042] Figure 5 An example diagram illustrating the process of drawing pedestrian trajectories for BEVs provided in this application embodiment;

[0043] Figure 6 An example diagram illustrating the process of determining whether a target pedestrian intends to open the trunk, as provided in this application embodiment;

[0044] Figure 7 This is a schematic diagram of the structure of an automatic trunk opening device for a vehicle, provided in an embodiment of this application. Detailed Implementation

[0045] In today's era of booming vehicle intelligence, the trunk opening assist function has become an important part of improving vehicle usability and user experience. Currently, the industry mainly adopts the following two technical solutions to implement this function:

[0046] (1) Sensor-based assisted trunk opening solution.

[0047] Its core principle lies in non-contact human motion sensing. Specifically, this solution uses a three-dimensional sensing array composed of capacitive, millimeter-wave radar, and infrared sensors to monitor the user's lower limb movement trajectory in real time. The signal is analyzed in the time and frequency domains by the ECU to identify features such as leg lift height and swing frequency, triggering the opening and closing of the electric tailgate. The hardware used includes a main control sensor module, a central control unit, a drive system, and safety protection components, forming a complete event-driven interactive system. After unlocking the vehicle, the user must stand in the designated area and perform a specified action (such as lifting their leg 15-20cm off the ground and holding it for 0.5 seconds). The sensors collect parameters such as displacement and speed, which are then filtered by Kalman filtering and compared with a pre-stored template. If the confidence level exceeds 85%, the tailgate is triggered to open slowly.

[0048] (2) Vision-based assisted trunk opening solution.

[0049] This solution uses high-precision visual capture equipment to acquire the user's image information and employs advanced image processing algorithms to analyze the information and determine whether the user has performed a specific action. Only after confirming the user's identity and that the action meets preset conditions will the trunk opening operation be executed. This vision-based identity verification mechanism improves the security and reliability of assisted trunk opening.

[0050] However, both of the above-mentioned existing methods for assisting in opening the vehicle's trunk have significant problems, affecting the user's driving experience. Specifically, these problems manifest in the following two ways:

[0051] (1) While sensor-based (e.g., capacitive field sensors, millimeter-wave radar) assisted trunk opening solutions offer high detection accuracy, their technical architecture has significant limitations. Firstly, this solution relies on a physical contact triggering mechanism, which is susceptible to environmental interference in actual use. For example, mud or dirt on the vehicle surface may cause false triggering of the sensor. Secondly, such solutions require high-precision sensor hardware, which has high manufacturing costs, significantly increasing hardware costs during mass production. More importantly, this solution lacks a biometric verification process for users, failing to effectively differentiate user access permissions, i.e., it cannot distinguish between authorized and unauthorized users, posing certain security risks. Furthermore, when users are carrying heavy loads or holding infants, performing specific limb movements is both inconvenient and poses security risks.

[0052] (2) The traditional vision-based trunk opening solution, compared to the sensor-based solution, can verify the user's biometric identity through image analysis, but its application is still subject to several limitations. First, this solution requires the user to actively complete a preset action (such as a leg sweep) as a trigger condition. This artificially set interaction method violates the principle of natural interaction, especially when the user's hands are occupied, such as when the user is holding an object or a child. The operation experience is poor and there are safety hazards, which may cause items to fall or people to be injured due to excessive movement or improper posture. Second, this solution is limited by the camera's field of view and lighting conditions, making it difficult to achieve full coverage of the cabin space. There are blind spots in the monitoring, which may lead to missed detections or false detections. Third, traditional vision algorithms are not robust enough to complex scenes (such as clothing obstruction and changes in light and shadow), which limits the accuracy of user identification and the reliability of behavior judgment.

[0053] To address the aforementioned shortcomings, this application provides a method for automatically opening a vehicle trunk. First, when the target vehicle is in a parked and locked state, if the fingerprint verification of the target vehicle's device is successful, the target vehicle is unlocked, and N video streams around the target vehicle are acquired using a panoramic camera system composed of N onboard cameras, where N is a positive integer greater than 0. Then, pedestrian detection and cross-camera pedestrian tracking are performed on each frame of the N video streams to obtain pedestrian detection results and pedestrian tracking results. Next, based on the pedestrian tracking results, BEV pedestrian trajectories are generated; and the BEV trajectories of target pedestrians (which can be understood as pedestrians whose final location is within a preset range near the trunk) are filtered out; then, the identity of the target pedestrian is verified using images of the target pedestrian contained in N video streams; after the verification is passed, the state map of the target pedestrian is obtained based on the pedestrian detection results, and the BEV trajectory of the target pedestrian, the state map of the target pedestrian, and the scene text information corresponding to the landmark buildings within the preset range around the target vehicle, combined with the prompt instruction, are input into the preset VLM to obtain the model output of the determination result of whether the target pedestrian has the intention to open the trunk of the target vehicle; and when it is determined that the target pedestrian has the intention to open the trunk of the target vehicle, the trunk of the target vehicle is automatically opened.

[0054] As can be seen, this application utilizes a full-vehicle panoramic camera system composed of N onboard cameras on the target vehicle to acquire N video streams around the target vehicle as a basis for judgment. Combined with VLM (Virtual Machine Model), it achieves coordinated processing of continuous user identity verification, dynamic tracking of movement trajectory, and intelligent analysis of behavioral intent. When it is determined that the target pedestrian has the intent to open the target vehicle's trunk, the trunk of the target vehicle is automatically opened. Compared with existing sensor-based assisted trunk opening solutions, this not only reduces the dependence on high-precision sensor hardware and significantly reduces costs, but also, after the target vehicle's device fingerprint verification is passed, this application superimposes continuous verification of the pedestrian's identity, thereby effectively distinguishing the operating permissions of different pedestrians and eliminating security risks.

[0055] Compared to existing vision-based trunk opening assistance solutions, this application not only eliminates the need for pedestrians to actively perform preset actions (such as leg sweeping), but also accurately determines the pedestrian's intention to open the vehicle's trunk through VLM, improving convenience and user experience. Furthermore, this application constructs a comprehensive monitoring network based on a combination of full-vehicle panoramic camera shooting, pedestrian detection, and cross-camera pedestrian tracking, ensuring that the needs of pedestrians can be captured in real time under any conditions. This eliminates monitoring blind spots (such as those limited by camera field of view and lighting conditions in vision-based trunk opening assistance solutions), significantly reducing the probability of false triggers and improving vehicle safety and pedestrian user experience.

[0056] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] See Figure 1 This is a flowchart illustrating a method for automatically opening a vehicle trunk according to this embodiment. The method includes the following steps:

[0058] S101: When the target vehicle is in the parked and locked state, if the device fingerprint verification of the target vehicle is successful, the target vehicle is unlocked, and the N video streams around the target vehicle are obtained by using the N on-board cameras on the target vehicle to form a full-vehicle panoramic camera; where N is a positive integer greater than 0.

[0059] In this embodiment, any vehicle used to determine whether the trunk automatically opens using this embodiment is defined as the target vehicle. Furthermore, to achieve seamless automatic opening of the target vehicle's trunk while ensuring the vehicle's security, this application assumes the target vehicle is in a parked and locked state (e.g., ...). Figure 2 When the target vehicle is engaged in Park and locked, the device fingerprint (the specific content is not limited, but may include, but is not limited to, a unique identifier of at least one of the following hardware: Bluetooth key, Near Field Communication (NFC), and Ultra-Wideband (UWB)) is first verified. The specific verification method is not limited; for example, using a Bluetooth key... Figure 2 As shown, when a Bluetooth key is detected approaching, a feedback signal from the Bluetooth key can be obtained. This feedback signal carries the security information stored in the Bluetooth key. The security information is then matched with the pre-stored security information. If the match is successful, the verification is successful; otherwise, the verification fails, thus ensuring the legitimacy of the terminal.

[0060] Furthermore, when the target vehicle's device fingerprint verification is successful, the target vehicle can be unlocked, and the vehicle's panoramic camera can be activated to obtain video streams around the target vehicle. The panoramic camera consists of N (the specific value is not limited and can be any positive integer greater than 0) on-board cameras on the target vehicle, with no limit on the specific number or installation location. One optional implementation is that N can be 8; and the panoramic camera consisting of N on-board cameras can include, but is not limited to, two on-board cameras located in front of the target vehicle, two on the left front and right front of the target vehicle, two on the left rear and right rear of the target vehicle, and two on-board cameras located in the rear view of the target vehicle, such as... Figure 3 As shown. The maximum field of view of these eight vehicle-mounted cameras is 12m, and they are fully visible within a 2m radius around the vehicle. Specifically, the two front-view cameras have horizontal viewing angles of 30° and 120°, with viewing distances of 12m and 2.5m respectively; the two front left and right cameras have a horizontal viewing angle of 120° and a viewing distance of 2m; the two rear left and right cameras have a horizontal viewing angle of 60° and a viewing distance of 5m; and the two rear-view cameras have horizontal viewing angles of 60° and 120°, with viewing distances of 5m and 2.5m respectively.

[0061] In this way, by using N onboard cameras on the target vehicle to form a full-vehicle panoramic camera system, N video streams around the target vehicle can be acquired. For example, using... Figure 3 After the eight vehicle-mounted cameras have acquired eight video streams, the subsequent step S102 can be executed.

[0062] S102: Perform pedestrian detection and cross-camera pedestrian tracking processing on each frame of the N video streams to obtain pedestrian detection results and pedestrian tracking results.

[0063] In this embodiment, after acquiring N video streams around the target vehicle using a full-vehicle panoramic camera composed of N onboard cameras in step S101, in order to more accurately achieve the seamless automatic opening of the target vehicle's trunk, it is further necessary to perform pedestrian detection and pedestrian tracking processing on each frame of these N video streams to obtain pedestrian detection results and pedestrian tracking results, thereby realizing pedestrian detection and full-domain pedestrian trajectory tracking across (camera) lenses, which is then used to execute the subsequent step S103.

[0064] Specifically, one possible implementation is that the specific implementation process of this step S102 may include the following step AE:

[0065] Step A: Input the first frame images from the N video streams into the pedestrian detection model for prediction, and obtain the prediction results of the detection boxes for the pedestrian's type in the N first frame images.

[0066] In this implementation, the specific composition of the pedestrian detection model is not limited, and it can be trained using cross-entropy loss and ternary loss. Furthermore, this embodiment does not limit the type of each frame image in the N video streams. For example, each frame image can be a color image composed of the three primary colors of red (G), green (G), and blue (B), or it can be a grayscale image, etc. RGB will be used as an example in the following description.

[0067] For example: to utilize, such as Figure 3 Taking the example of 8 vehicle-mounted cameras acquiring 8 video streams, as shown below... Figure 4 As shown, the first eight images from the first frame can be simultaneously input into the pedestrian detection model. This input can be represented as (8, 3, 1920, 1080), where 8 represents the batch size (meaning eight images), 3 represents the number of image channels (e.g., RGB), 1920 represents the image width, and 1080 represents the image height. After inputting these images into the pedestrian detection model, the predicted bounding boxes for the "pedestrian" category can be obtained. Taking two pedestrians as an example, the result format can be represented as [x1, y1, x2, y2].

[0068] Step B: Using the detection bounding box prediction results for pedestrians' types, perform matting processing on the corresponding N first-frame images to obtain the matted images of the N first-frame images, which are used as the pedestrian detection results.

[0069] In this implementation, after obtaining the prediction results of the detection boxes for pedestrian types in the N first-frame images through step A, the prediction results for pedestrian types can be used to perform image matting on the corresponding N first-frame images to obtain the matted images of the N first-frame images as pedestrian detection results.

[0070] For example: Based on the above examples, such as Figure 4 As shown, after obtaining the predicted bounding box results [x1, y1, x2, y2] for the pedestrian's personality, we can further perform image matting on the corresponding original images to obtain the matted images of the first frame in 8 channels, and label them as cam1_crop0 (i.e. Figure 4 One of the target matting images shown in Cam1), cam1_crop1 (i.e. Figure 4 (See the second example of target matting under Cam1), cam2_crop0, ...

[0071] Step C: Input the matted images of the N first-frame images into the Person Re-Identification Feature Extractor (ReID) for feature extraction to obtain the matted features of the N first-frame images.

[0072] In this implementation, after obtaining the patching of the N first-frame images in step B as the pedestrian detection result, as follows: Figure 4 As shown, the matting of these N first-frame images can be further input into the ReID feature extractor for feature extraction to obtain the matting features of these N first-frame images.

[0073] For example: Based on the above examples, such as Figure 4 As shown, after obtaining N (i.e., 8) channels of the first frame image, the matting cam1_crop0 (i.e., Figure 4 One of the target matting images shown in Cam1), cam1_crop1 (i.e. Figure 4 As shown in the second example of target matting (Cam1_crop0), ..., this series of images can then be fed into the ReID feature extractor. The structure of the ReID feature extractor is not limited; it can consist of a simple pyramid network (trained using cross-entropy loss and ternary loss). Using the ReID feature extractor, the features of each crop image can be obtained and labeled as cam1_feature0, cam1_feature1, cam2_feature0, ...

[0074] Step D: Calculate the cosine similarity between the matting features of different first-frame images in the N streams, and select the highest cosine similarity. Based on the unique identifier of the pedestrian, construct the feature dictionary corresponding to the first-frame image in the N streams.

[0075] In this implementation, after obtaining the matting features of the N first-frame images through step C, as follows: Figure 4 As shown, the cosine similarity between the matting features of different first-frame images in the N-channel first-frame images can be further calculated, and the highest cosine similarity can be selected. Based on the unique identifier of the pedestrian, a feature dictionary corresponding to the first-frame image in the N-channel video stream can be constructed.

[0076] For example: Based on the above examples, such as Figure 4 As shown, after obtaining the features of each crop image and labeling them as cam1_feature0, cam1_feature1, cam2_feature0,..., the cosine similarity between cam{number}_feature{index} of different cameras can be calculated to confirm which pedestrian targets exist in multiple camera shots. After calculating the cosine similarity, the one with the highest cosine similarity can be selected and organized into a feature dictionary according to the unique identifier id of the person, which can be represented as {'id1':[cam1_feature0, cam2_feature2],'id2':[cam1_feature1, cam2_feature0]}.

[0077] Step E: Repeat steps A and D above to generate the feature dictionary, replacing the first frame image with the second frame image to construct the feature dictionary corresponding to the second frame image in the N video streams, as follows: Figure 4 As shown, based on the feature dictionary corresponding to the first frame image and the feature dictionary corresponding to the second frame image, the K-nearest neighbor algorithm can be used to determine the association dictionary; and so on, after calculating the association dictionary for multiple frames of images in N video streams, the tracking result for each pedestrian can be determined.

[0078] For example: Based on the above example, after constructing the feature dictionary corresponding to the second frame image, the K-nearest neighbor algorithm can be used to determine the associated dictionary {'time1_id1':'id2', 'time2_id2':'id1'} using the feature dictionary (1) corresponding to the first frame image and the feature dictionary (2) corresponding to the second frame image. After multi-frame processing, the trajectory of a pedestrian with a certain ID is thus determined (e.g., ...). Figure 4 The target tracking results shown are A and B, respectively.

[0079] S103: Based on the pedestrian tracking results, generate a bird's-eye view BEV pedestrian trajectory; and filter out the target pedestrian BEV trajectory from it.

[0080] In this embodiment, after obtaining the pedestrian tracking result in step S102, the pedestrian tracking result is further mapped to a trajectory in three-dimensional (3D) space based on the intrinsic parameters of the whole-vehicle panoramic camera composed of N on-board cameras on the target vehicle. Then, the pedestrian tracking result is mapped to the trajectory in three-dimensional space onto a two-dimensional plane through BEV projection to obtain the BEV pedestrian trajectory. The BEV trajectory of the pedestrian who may have the intention to open the trunk (defined here as the target pedestrian) is selected for subsequent step S104.

[0081] For example: when forming as Figure 5 The BEV trajectory diagram shown in the lower right corner displays two pedestrian BEV trajectories. The pedestrian BEV trajectory indicated by the purple arrow on the right can be selected as the target pedestrian BEV trajectory because its arrow points to the area in front of the trunk at the rear of the vehicle. This indicates that the pedestrian stepped off the vehicle from the right front and is likely intending to open the trunk. Conversely, the pedestrian BEV trajectory indicated by the blue arrow on the left points to the area in front of the vehicle's left front. This indicates that the pedestrian walked from the right rear of the vehicle to the left front and is unlikely to intend to open the trunk, therefore this pedestrian BEV trajectory cannot be selected as the target pedestrian BEV trajectory.

[0082] S104: Verify the identity of the target pedestrian using images of the target pedestrian contained in N video streams; and after the verification is successful, obtain the state map of the target pedestrian based on the pedestrian detection results.

[0083] In this embodiment, after determining the BEV trajectory of the target pedestrian in step S103, the images containing the target pedestrian in the N video streams can be used for image recognition processing to verify the identity of the target pedestrian. After the verification is successful, the state map of the target pedestrian is obtained based on the pedestrian detection result (which can be understood as the result of the target pedestrian being cut out on the original image based on the pedestrian detection model) to execute the subsequent step S105.

[0084] One possible implementation is to use images of the target pedestrian contained in N video streams to perform facial recognition on the pedestrian (the specific recognition method is not limited), obtain the facial recognition result, and then use the facial recognition result to verify the identity of the target pedestrian, thus forming a dual verification of "target pedestrian identity + terminal legitimacy," fundamentally solving security risks. That is, the vehicle will only respond if the target pedestrian's biometric features match and the operation command originates from a pre-bound legitimate device. This "human-machine-terminal" three-in-one security mechanism effectively prevents the risk of illegal copying or impersonation, forming a unique security barrier.

[0085] In this implementation, facial recognition of the target pedestrian can be achieved through a pre-built biometric module. This module can address the heterogeneous recognition problem in low-quality facial images by using neutral face learning and a progressive fusion network. The method can be divided into two stages: the first stage learns the inherent structured information of the face through adversarial learning; the second stage progressively fuses style differences from different source data using a symmetric network. Specific implementation details are not elaborated here.

[0086] S105: Input the BEV trajectory of the target pedestrian, the state diagram of the target pedestrian, and the scene text information corresponding to the landmark buildings within the preset range around the target vehicle, along with the prompt instruction, into the preset visual-language model (VLM) to obtain the model output of the determination result of whether the target pedestrian has the intention to open the trunk of the target vehicle; and automatically open the trunk of the target vehicle when it is determined that the target pedestrian has the intention to open the trunk of the target vehicle.

[0087] In this embodiment, after obtaining the target pedestrian's BEV trajectory in step S103 and the target pedestrian's state map in step S104, the target pedestrian's BEV trajectory, the target pedestrian's state map, and scene text information corresponding to landmark buildings within a preset range around the target vehicle (such as...) can be further processed. Figure 6 The system obtains the target vehicle's location information via GPS or other means, and after extracting text from landmarks within a 500-meter radius, it incorporates prompts such as "XX Bank; XX High-Speed ​​Railway Station; XX Shopping Mall..." into a preset VLM. This VLM can then output a determination of whether the pedestrian intends to open the target vehicle's trunk. If the VLM determines that the pedestrian intends to open the target vehicle's trunk, it automatically opens the trunk.

[0088] The specific structure of the preset VLM is not limited and can be trained using pre-training and supervised fine-tuning (SFT) based on actual conditions and empirical values. Figure 6As shown, this VLM may include a visual encoder, a projector, and a large language model (LLM).

[0089] Based on this, the implementation process of step S105, which involves "inputting the target pedestrian's BEV trajectory, the target pedestrian's state diagram, and the scene text information corresponding to landmarks within a preset range around the target vehicle, combined with the prompt instruction, into a preset visual-language model (WLM), and obtaining the model's output judgment result on whether the target pedestrian intends to open the target vehicle's trunk," can specifically include:

[0090] First, the BEV trajectory and state graph of the target pedestrian are stitched together to obtain a hypergraph. This hypergraph is then input into a visual encoder for feature extraction, yielding image-encoded features. Next, these image-encoded features are input into a projection layer for dimensionality reduction, resulting in dimensionality-reduced image-encoded features. Simultaneously, the scene text information corresponding to landmarks within a preset range around the target vehicle is encoded, yielding text-encoded features. Finally, the dimensionality-reduced image-encoded features and text-encoded features are combined with system prompts and traction information and input into the LLM (Low-Level Model) to obtain the model's output determining whether the target pedestrian intends to open the trunk of the target vehicle.

[0091] For example: Figure 6 As shown, based on the target pedestrian's BEV trajectory path (obtained in step S103), the target pedestrian's state map (obtained by slicing the target pedestrian from the original image using a pedestrian detection model), and landmarks within 500m of the current target vehicle's location (e.g., "XX Bank; XX High-Speed ​​Railway Station; XX Shopping Mall..."), the input to the VLM can be: "You are an assistant for determining a person's intent. Now, please determine whether the person intends to open the trunk based on the input image and scene information. The left side of the image shows the trajectory information under the BEV, and the right side shows the target person's current state map. Scene information: XX Department Store, XX Kindergarten. If the target person intends to open the trunk, please answer yes; otherwise, please answer no." The target pedestrian's BEV trajectory path and the target pedestrian's state map (i.e., the target pedestrian's actual state map) are used as image input, and the landmark information within 500m of the current target vehicle's location is used as text input. After processing this information, the VLM can output the determination result of whether the target pedestrian intends to open the target vehicle's trunk, such as yes or no.

[0092] Next, we will give a brief introduction to the training process of VLM, which can be divided into the pre-training stage and the SFT stage.

[0093] In pre-training, the model's input consists of images and text information, and various proxy tasks can be designed, such as common image description generation tasks, visual-language alignment tasks, visual question answering tasks, audio-visual question answering tasks, etc.

[0094] In the SFT stage, scene-related training data can be constructed. The image input can include a hypermap composed of sample pedestrian BEV trajectory paths and sample pedestrian state images. For example, the sample pedestrian BEV trajectory image size can be 640x480, and the sample pedestrian state image size can be 312x477. The sample pedestrian state image is resized along its long side and padded to a dynamic_width x 480. This hypermap is then stitched with the BEV trajectory image to obtain a resolution of (640 + dynamic_width, 480), which is then fed into the visual encoder for feature extraction. Additionally, text information can be constructed, and the VLM can be processed using a combination of system_prompt and traction information; the specific process will not be detailed here.

[0095] In summary, the automatic trunk opening method provided in this embodiment firstly unlocks the target vehicle when it is in a parked and locked state if the fingerprint verification of the target vehicle is successful. Then, the target vehicle is unlocked, and a full-vehicle panoramic camera composed of N on-board cameras on the target vehicle acquires N video streams around the target vehicle, where N is a positive integer greater than 0. Then, pedestrian detection and cross-camera pedestrian tracking are performed on each frame of the N video streams to obtain pedestrian detection results and pedestrian tracking results. Next, based on the pedestrian tracking results, BEV pedestrian trajectories are generated; and the BEV trajectories of target pedestrians (which can be understood as pedestrians whose final location is within a preset range near the trunk) are filtered out; then, the identity of the target pedestrian is verified using images of the target pedestrian contained in N video streams; after the verification is passed, the state map of the target pedestrian is obtained based on the pedestrian detection results, and the BEV trajectory of the target pedestrian, the state map of the target pedestrian, and the scene text information corresponding to the landmark buildings within the preset range around the target vehicle, combined with the prompt instruction, are input into the preset VLM to obtain the model output of the determination result of whether the target pedestrian has the intention to open the trunk of the target vehicle; and when it is determined that the target pedestrian has the intention to open the trunk of the target vehicle, the trunk of the target vehicle is automatically opened.

[0096] As can be seen, this application utilizes a full-vehicle panoramic camera system composed of N onboard cameras on the target vehicle to acquire N video streams around the target vehicle as a basis for judgment. Combined with VLM (Virtual Machine Model), it achieves coordinated processing of continuous user identity verification, dynamic tracking of movement trajectory, and intelligent analysis of behavioral intent. When it is determined that the target pedestrian has the intent to open the target vehicle's trunk, the trunk of the target vehicle is automatically opened. Compared with existing sensor-based assisted trunk opening solutions, this not only reduces the dependence on high-precision sensor hardware and significantly reduces costs, but also, after the target vehicle's device fingerprint verification is passed, this application superimposes continuous verification of the pedestrian's identity, thereby effectively distinguishing the operating permissions of different pedestrians and eliminating security risks.

[0097] Compared to existing vision-based trunk opening assistance solutions, this application not only eliminates the need for pedestrians to actively perform preset actions (such as leg sweeping), but also accurately determines the pedestrian's intention to open the vehicle's trunk through VLM, improving convenience and user experience. Furthermore, this application constructs a comprehensive monitoring network based on a combination of full-vehicle panoramic camera shooting, pedestrian detection, and cross-camera pedestrian tracking, ensuring that the needs of pedestrians can be captured in real time under any conditions. This eliminates monitoring blind spots (such as those limited by camera field of view and lighting conditions in vision-based trunk opening assistance solutions), significantly reducing the probability of false triggers and improving vehicle safety and pedestrian user experience.

[0098] Furthermore, this application embodiment also provides a vehicle trunk automatic opening device, including: a processor, a memory, and a system bus;

[0099] The processor and the memory are connected via the system bus;

[0100] The memory is used to store one or more programs, the one or more programs including instructions, which, when executed by the processor, cause the processor to perform any of the above-described implementations of the automatic opening method for the vehicle trunk.

[0101] refer to Figure 7 The diagram illustrates a suitable structural schematic for implementing the automatic trunk opening device in the embodiments of this application. The automatic trunk opening device in the embodiments of this application may include, but is not limited to, a PC terminal, a vehicle's intelligent tailgate control system, a cloud platform, a server, and a server cluster. Figure 7 The illustrated automatic trunk opening device is merely an example of an electronic device and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0102] like Figure 7As shown, the automatic trunk opening device may include a processor (e.g., a central processing unit, graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. When the automatic trunk opening device is powered on, the RAM 703 also stores various programs and data required for device operation. The processor 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0103] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, memory cards, hard drives, etc.; and communication devices 709. Communication device 709 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An automatic trunk opening device for a vehicle is shown, but it should be understood that it is not required to implement or have all of the shown devices. More or fewer devices may be implemented or included alternatively.

[0104] Furthermore, embodiments of this application also provide a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform any of the above-described methods for automatically opening the vehicle trunk.

[0105] Furthermore, this application embodiment also provides a computer program product, which, when run on a terminal device, causes the terminal device to execute any of the above-described methods for automatically opening the vehicle trunk.

[0106] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0107] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0108] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0109] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for automatically opening a vehicle trunk, characterized in that, include: When the target vehicle is in a parked and locked state, if the device fingerprint verification of the target vehicle is successful, the target vehicle is unlocked, and the N video streams around the target vehicle are obtained by using the N on-board cameras that make up the whole vehicle panoramic camera; where N is a positive integer greater than 0. Pedestrian detection and cross-camera pedestrian tracking are performed on each frame of the N video streams to obtain pedestrian detection results and pedestrian tracking results; Based on the pedestrian tracking results, a bird's-eye view BEV pedestrian trajectory is generated; And filter out the BEV trajectories of the target pedestrians; The identity of the target pedestrian is verified using images of the target pedestrian contained in the N video streams; And after the verification is passed, the state map of the target pedestrian is obtained based on the pedestrian detection results; The BEV trajectory of the target pedestrian, the state diagram of the target pedestrian, and the scene text information corresponding to the landmark buildings within a preset range around the target vehicle, combined with the prompt instruction, are input into a preset visual-language model (VLM) to obtain the model output a determination result on whether the target pedestrian has the intention to open the trunk of the target vehicle. And when it is determined that the target pedestrian intends to open the trunk of the target vehicle, the trunk of the target vehicle will be opened automatically.

2. The method according to claim 1, characterized in that, The pedestrian detection results are determined as follows: The pedestrian detection model is used to predict the first frame images of the N video streams to obtain the prediction results of the detection boxes for the pedestrian's type in the N first frame images. Using the predicted detection bounding boxes for pedestrian types, the corresponding N first-frame images are matted to obtain the matted images of the N first-frame images, which are used as pedestrian detection results.

3. The method according to claim 2, characterized in that, The pedestrian tracking results are determined as follows: The matted images of the N first-frame images are input into the ReID feature extractor for feature extraction to obtain the matted features of the N first-frame images; Calculate the cosine similarity between the matting features of different first-frame images in the N streams, and select the highest cosine similarity. Based on the unique identifier of the pedestrian, construct the feature dictionary corresponding to the first-frame image in the N streams. Repeat the aforementioned steps of generating feature dictionaries, replacing the first frame image with the second frame image to form the feature dictionary corresponding to the second frame image in the N video streams. Then, based on the feature dictionary corresponding to the first frame image and the feature dictionary corresponding to the second frame image, use the K-nearest neighbor algorithm to determine the associated dictionary. Similarly, after calculating the association dictionary for multiple frames of images in the N video streams, the tracking result for each pedestrian is determined.

4. The method according to claim 1, characterized in that, The step of generating a bird's-eye view BEV pedestrian trajectory based on the pedestrian tracking results includes: Based on the intrinsic parameters of the vehicle-mounted cameras consisting of N onboard cameras on the target vehicle, the pedestrian tracking results are mapped to a trajectory in three-dimensional space. By projecting the pedestrian tracking results onto a three-dimensional trajectory in BEV space, and then onto a two-dimensional plane, the BEV pedestrian trajectory is obtained.

5. The method according to claim 1, characterized in that, The step of verifying the identity of the target pedestrian using images of the target pedestrian contained in the N video streams includes: The target pedestrian's face is recognized by using images of the target pedestrian contained in the N video streams, and the face recognition result is used to verify the identity of the target pedestrian.

6. The method according to claim 1, characterized in that, The preset visual-language model (VLM) includes a visual encoder, a projection layer, and a large language model (LLM). The target pedestrian's BEV trajectory, the target pedestrian's state diagram, and scene text information corresponding to landmarks within a preset range around the target vehicle, combined with a prompt instruction, are input into the preset visual-language model (VLM) to obtain the model's output determining whether the target pedestrian intends to open the target vehicle's trunk. This includes: The BEV trajectory of the target pedestrian and the state map of the target pedestrian are stitched together to obtain a stitched hypermap; the hypermap is then input into the visual encoder for feature extraction to obtain image encoding features. The image coding features are input into the projection layer for dimensionality reduction to obtain the dimensionality-reduced image coding features. The scene text information corresponding to landmark buildings within a preset range around the target vehicle is encoded to obtain text encoding features; The reduced image encoding features and the text encoding features are input into the LLM through a combination of system prompt instructions and traction information to obtain the model output a determination result on whether the target pedestrian intends to open the trunk of the target vehicle.

7. The method according to any one of claims 1-6, characterized in that, The device fingerprint of the target vehicle includes a unique identifier of at least one of the following hardware: Bluetooth key, Near Field Communication (NFC), and Ultra Wideband (UWB).

8. The method according to any one of claims 1-6, characterized in that, The value of N is 8; the vehicle-mounted camera system consisting of N onboard cameras on the target vehicle includes two onboard cameras located in front of the target vehicle, two onboard cameras located in front left and front right of the target vehicle, two onboard cameras located in rear left and rear right of the target vehicle, and two onboard cameras located in rear view of the target vehicle.

9. An automatic trunk opening device for a vehicle, characterized in that, include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, the one or more programs including instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the method described in any one of claims 1-8.

11. A computer program product, characterized in that, The program product includes a computer program that, when executed by an electronic product, can implement the method described in any one of claims 1-8.

Images