Method for labeling attributes of a perception target for assisted driving and related device

Abstract

The embodiment of the application discloses a kind of attribute labeling methods and related devices for assisting driving perception target, method includes: collecting the original sensing data of assisting driving perception target;According to the original sensing data, obtain basic perception data;For the fine-grained attribute of assisting driving perception target, the assisting driving perception target is filtered to determine the candidate assisting driving perception target, and the local image data of the key area of candidate assisting driving perception target is extracted;Local image data and text are input into text-image alignment model for describing fine-grained attribute, to obtain image feature and text feature;The similarity of image feature and text feature is calculated, and the text that meets the preset condition is labeled as candidate assisting driving perception target.The efficiency and precision of labeling are improved by constructing the fine-grained attribute automatic labeling system of assisting driving perception target, and reliable data support is provided for the iteration of assisting driving perception model.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an attribute labeling method and related apparatus for assisted driving perception targets. Background Technology

[0002] With the rapid development of automotive intelligence and connectivity technologies, Advanced Driver Assistance Systems (ADAS) and autonomous driving technologies have become core development directions in the automotive industry. Current assisted driving systems require obtaining future trajectory or speed information of key targets affecting the vehicle's movement to enable reasonable path and speed planning based on the future trajectories of other road users. Besides using track IDs to predict the future trajectories and speeds of other obstacles, it also needs to combine the status of turn signals, brake lights, and other traffic lights to predict the future movement trends of other obstacles. The training, optimization, and rapid iteration of assisted driving perception models rely on massive amounts of high-precision labeled data as ground truth. The aforementioned attribute annotations are incremental attribute annotations and represent a core data requirement in the current perception model training process.

[0003] Currently, the industry's labeling solutions for such fine-grained attributes mainly include: 1. Manual labeling by human annotators: This method relies entirely on manual labor and is not automated. It is difficult to complete the incremental attribute labeling work on the existing massive training sets in the short term. 2. Semi-automatic labeling method: This method involves manually labeling a small batch of subdivided attribute data first, and then having the annotators review the model's output results. However, it still cannot achieve truly fully automated labeling. At the same time, the generalization ability of dedicated classification models is limited, and their adaptability across lighting, weather, and road scenes is poor. Continuous investment of manpower is required for data iteration and model optimization, making it difficult to control training costs.

[0004] Therefore, there is an urgent need for a fully automated, low-cost fine-grained attribute annotation scheme for assisted driving perception targets, which can quickly and efficiently complete the attribute annotation of driving data to meet the need for rapid iteration of assisted driving perception models. Summary of the Invention

[0005] In view of this, this application provides an attribute annotation method and related device for assisted driving perception targets. By constructing a fine-grained automatic annotation system for assisted driving perception targets through a multimodal large model, the method reduces the dependence on manual annotation, improves the efficiency and accuracy of annotation, and provides reliable data support for the iteration of assisted driving perception models.

[0006] In a first aspect, embodiments of this application provide an attribute annotation method for assisted driving perception targets, the method comprising: Collect raw sensor data of the environment in which the assisted driving perception target is located, wherein the assisted driving perception target includes multi-wheeled vehicles; Based on the original sensing data, basic perception data is obtained, which includes: the 3D detection box of the assisted driving perception target, vehicle identification, vehicle category, and the current vehicle pose data. The assisted driving perception targets are screened based on their fine-grained attributes to determine candidate assisted driving perception targets, and local image data of key areas of the candidate assisted driving perception targets are extracted. The local image data is input into the text-image alignment model to obtain image features; and the text used to describe the fine-grained attributes is input into the text-image alignment model to obtain text features. Calculate the similarity between the image features and the text features, and label the attributes corresponding to the text whose similarity meets the preset conditions as the candidate assisted driving perception targets.

[0007] In one possible embodiment, the fine-grained attributes include at least: a first attribute characterizing whether the two-wheeled vehicle is in a riding state, a second attribute characterizing whether the brake lights of the four-wheeled vehicle are on, a third attribute characterizing whether the turn signals of the four-wheeled vehicle are on, and a fourth attribute characterizing whether the doors of the four-wheeled vehicle are open; the step of filtering the assisted driving perception targets based on the fine-grained attributes of the assisted driving perception targets to determine candidate assisted driving perception targets includes: for the first attribute, filtering the two-wheeled vehicle as a candidate assisted driving perception target according to the vehicle category; for the second attribute, filtering vehicles in the lanes directly in front of the current vehicle and in the adjacent lanes to the left and right as candidate assisted driving perception targets based on the pose data of the current vehicle; for the third attribute, filtering vehicles within a preset distance, with an occlusion rate below a preset threshold, and exhibiting lateral deflection as candidate assisted driving perception targets, wherein the presence of lateral deflection means that the change in the heading angle of the 3D detection box of the four-wheeled vehicle meets a preset angle threshold; and for the fourth attribute, filtering vehicles adjacent to the future driving trajectory of the current vehicle as candidate assisted driving perception targets.

[0008] In one possible embodiment, labeling the candidate assisted driving perception target includes: when labeling the first attribute, transforming the 3D detection box of the two-wheeled vehicle to the world coordinate system based on the 3D detection box of the two-wheeled vehicle and the pose data of the current vehicle, and calculating the trajectory length, where the trajectory length refers to the cumulative movement distance of the center points of consecutive 3D detection boxes under the same vehicle identifier in the world coordinate system; if the trajectory length reaches a preset dynamic threshold, then labeling the two-wheeled vehicle as being in the riding state; if the trajectory length does not reach the preset dynamic threshold, then extracting local image data of the key areas of the two-wheeled vehicle, and through... The text-image alignment model obtains the first image feature, and the key region of the two-wheeled vehicle is the entire vehicle area. The first text is input into the text-image alignment model to obtain the text feature of the first text, which describes the first attribute, including: a moving two-wheeled vehicle ridden by a person and a stationary two-wheeled vehicle ridden by an unmanned person. The similarity between the first image feature and the text feature of the first text is calculated, and the attribute corresponding to the first text with the highest similarity is used as the annotation result of the first attribute of the four-wheeled vehicle. The first attribute is extended to all 3D detection boxes corresponding to the overall trajectory of the two-wheeled vehicle.

[0009] In one possible embodiment, the annotation of the candidate assisted driving perception target includes: when annotating the second attribute, extracting local image data of the key area of ​​the four-wheeled vehicle, extracting second image features through the text-image alignment model, wherein the key area of ​​the four-wheeled vehicle includes the taillight area of ​​the four-wheeled vehicle; inputting the second text into the text-image alignment model to obtain the text features of the second text, wherein the second text is used to describe the second attribute, including: brake lights on during the day, brake lights off during the day, brake lights on at night, and brake lights off at night; calculating the similarity between the second image features and the text features of the second text, and taking the attribute corresponding to the second text with the highest similarity as the annotation result of the second attribute of the four-wheeled vehicle.

[0010] In one possible embodiment, the annotation of the candidate assisted driving perception target includes: when annotating the third attribute, extracting local image data of the key area of ​​the four-wheeled vehicle, extracting third image features through the text-image alignment model, wherein the key area of ​​the four-wheeled vehicle includes the taillight area of ​​the four-wheeled vehicle; inputting the third text into the text-image alignment model to obtain the text features of the third text, wherein the third text is used to describe the third attribute, including: only the left turn signal is on during the day, only the right turn signal is on during the day, both left and right turn signals are on during the day, both left and right turn signals are off during the day, only the left turn signal is on at night, only the right turn signal is on at night, both left and right turn signals are on at night, and both left and right turn signals are off at night; calculating the similarity between the third image features and the text features of the third text, and taking the attribute corresponding to the third text with the highest similarity as the annotation result of the third attribute of the four-wheeled vehicle.

[0011] In one possible embodiment, the annotation of the candidate assisted driving perception target includes: when annotating the fourth attribute, extracting local image data of the key area of ​​the four-wheeled vehicle, extracting fourth image features through the text-image alignment model, wherein the key area of ​​the four-wheeled vehicle includes the door area of ​​the four-wheeled vehicle; inputting the fourth text into the text-image alignment model to obtain the text features of the fourth text, wherein the fourth text is used to describe the fourth attribute, including: at least one door on the left side of the vehicle is open, at least one door on the right side of the vehicle is open, and all doors of the vehicle are closed; calculating the similarity between the fourth image features and the text features of the fourth text, and taking the attribute corresponding to the fourth text with the highest similarity as the annotation result of the fourth attribute of the four-wheeled vehicle.

[0012] In one possible embodiment, the text-image alignment model is the CLIP model; the calculation of the similarity between the image features and the text features includes: determining a first feature vector of the image features; and determining a second feature vector of the text features; calculating the cosine similarity between the first feature vector and the second feature vector to determine the similarity between the image features and the text features.

[0013] Secondly, embodiments of this application provide an attribute annotation device for assisted driving perception targets. The device includes: a determination unit, an acquisition unit, an extraction unit, a feature extraction unit, and an annotation unit. The determination unit is used to determine the annotation requirements for fine-grained attributes of the assisted driving perception targets, including two-wheeled vehicles and four-wheeled vehicles. The acquisition unit is used to acquire raw sensor data of the environment in which the perception targets are located, and obtain basic perception data based on the raw sensor data. The basic perception data includes: the 3D detection box of the assisted driving perception targets, vehicle identification, vehicle category, and the pose data of the current vehicle. The extraction unit is used to... The fine-grained attributes of the assisted driving perception targets are used to filter the assisted driving perception targets, and local image data of key regions of the candidate assisted driving perception targets are extracted. The key regions refer to specific areas of the vehicle body of the candidate assisted driving perception targets. The feature extraction unit is used to input the local image data into a text-image alignment model to obtain image features; and to input the text used to describe the fine-grained attributes into the text-image alignment model to obtain text features. The annotation unit is used to calculate the similarity between the image features and the text features, and to annotate the attributes corresponding to the text whose similarity meets preset conditions as the candidate assisted driving perception targets.

[0014] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, the memory being used to store one or more programs and configured to be executed by the processor, the programs including instructions for performing the steps in the first aspect of embodiments of this application.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program for electronic data interchange, wherein the computer program causes a computer to perform some or all of the steps described in the first aspect of embodiments of this application.

[0016] Fifthly, embodiments of this application provide a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps described in the first aspect of embodiments of this application. The computer program product may be a software installation package.

[0017] As can be seen, this application provides a method and related apparatus for attribute annotation of assisted driving perception targets. The method includes: first, collecting raw sensor data of the environment in which the assisted driving perception target is located, including multi-wheeled vehicles; second, obtaining basic perception data based on the raw sensor data, including: 3D detection boxes of the assisted driving perception target, vehicle identification, vehicle category, and the current vehicle pose data; then, filtering assisted driving perception targets based on fine-grained attributes to determine candidate assisted driving perception targets, and extracting local image data of key regions of the candidate assisted driving perception targets; next, inputting the local image data into a text-image alignment model to obtain image features; and inputting the text used to describe the fine-grained attributes into the text-image alignment model to obtain text features; finally, calculating the similarity between the image features and the text features, and annotating the attributes corresponding to the text whose similarity meets preset conditions as candidate assisted driving perception targets. In this way, an automated annotation system for fine-grained attributes of assisted driving perception targets is constructed, reducing reliance on manual annotation, improving annotation efficiency and accuracy, and providing reliable data support for the iteration of assisted driving perception models. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the architecture of an attribute labeling system for assisted driving target perception provided in an embodiment of this application; Figure 2 This is a flowchart illustrating an attribute annotation method for a target perceived by assisted driving, provided in an embodiment of this application. Figure 3 This is a flowchart illustrating the process of labeling a first attribute for a candidate assisted driving perception target, provided in an embodiment of this application. Figure 4 This is a flowchart illustrating the process of labeling a second attribute for a candidate assisted driving perception target, provided in an embodiment of this application. Figure 5 This is a schematic diagram illustrating an application scenario for extracting image features according to an embodiment of this application; Figure 6 This is a flowchart illustrating the process of labeling a third attribute for a candidate assisted driving perception target, provided in an embodiment of this application. Figure 7 This is a flowchart illustrating the process of labeling a fourth attribute for a candidate assisted driving perception target, as provided in an embodiment of this application. Figure 8 This is a functional unit block diagram of an attribute labeling device for assisted driving target perception provided in an embodiment of this application; Figure 9 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0021] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but in some embodiments includes steps or units not listed, or in some embodiments includes other steps or units inherent to these processes, methods, products, or apparatuses.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] In the embodiments of this application, "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone; A and B exist simultaneously; B exists alone. Among them, A and B can be singular or plural.

[0024] In this embodiment, the symbol " / " can indicate that the preceding and following objects are in an "or" relationship. Alternatively, the symbol " / " can also represent a division sign, i.e., performing a division operation. For example, A / B can mean A divided by B.

[0025] In the embodiments of this application, "at least one item" or its similar expression refers to any combination of these items, including any combination of a single item or a plurality of items. "One or more" means one or more, while "multiple" means two or more. For example, "at least one item" of a, b, or c can represent the following seven cases: a, b, c; a and b; a and c; b and c; a, b, and c. Each of a, b, and c can be an element or a set containing one or more elements.

[0026] In the embodiments of this application, "equal to" can be used with "greater than" and is applicable to technical solutions used when "greater than" is used; it can also be used with "less than" and is applicable to technical solutions used when "less than" is used. When "equal to" is used with "greater than", it is not used with "less than"; when "equal to" is used with "less than", it is not used with "greater than".

[0027] The following describes the relevant content, concepts, meanings, technical issues, technical solutions, and beneficial effects involved in the embodiments of this application.

[0028] CLIP (Contrastive Language-Image Pre-training) is a general pre-training model based on the contrastive learning paradigm. It uses massive amounts of image and text pairing data collected from the Internet for cross-modal joint pre-training to achieve feature alignment between images and text in a unified semantic space.

[0029] With the rapid development of automotive intelligence and connectivity technologies, Advanced Driver Assistance Systems (ADAS) and autonomous driving technologies have become core development directions in the automotive industry. Current assisted driving systems require obtaining future trajectory or speed information of key targets affecting the vehicle's movement to enable reasonable path and speed planning based on the future trajectories of other road users. Besides using track IDs to predict the future trajectories and speeds of other obstacles, it also needs to combine the status of turn signals, brake lights, and other traffic lights to predict the future movement trends of other obstacles. The training, optimization, and rapid iteration of assisted driving perception models rely on massive amounts of high-precision labeled data as ground truth. The aforementioned attribute annotations are incremental attribute annotations and represent a core data requirement in the current perception model training process. Currently, the industry's labeling solutions for such fine-grained attributes mainly include: 1. Manual labeling by human annotators: This method relies entirely on manual labor and is not automated. It is difficult to complete the incremental attribute labeling work on the existing massive training sets in the short term. 2. Semi-automatic labeling method: This method involves manually labeling a small batch of subdivided attribute data first, and then having the annotators review the model's output results. However, it still cannot achieve truly fully automated labeling. At the same time, the generalization ability of dedicated classification models is limited, and their adaptability across lighting, weather, and road scenes is poor. Continuous investment of manpower is required for data iteration and model optimization, making it difficult to control training costs.

[0030] Therefore, there is an urgent need for a fully automated, low-cost fine-grained attribute annotation scheme for assisted driving perception targets, which can quickly and efficiently complete the attribute annotation of driving data to meet the need for rapid iteration of assisted driving perception models.

[0031] To address the aforementioned issues, this application provides an attribute annotation method and related apparatus for assisted driving perception targets. By constructing a fine-grained automated annotation system for assisted driving perception targets through a multimodal large model, it reduces reliance on manual annotation, improves annotation efficiency and accuracy, and provides reliable data support for the iteration of assisted driving perception models.

[0032] First, combined Figure 1 The attribute labeling system for assisted driving perception targets in the embodiments of this application will be described. Figure 1 This is a schematic diagram of the architecture of an attribute labeling system for assisted driving target perception provided in an embodiment of this application, such as... Figure 1As shown, the attribute annotation system 100 for assisted driving target perception includes a processor 110, a sensor module 120, and an on-board storage unit 130. The processor 110 is electrically connected to both the sensor module 120 and the on-board storage unit 130. Specifically, the processor 110 is electrically connected to the vehicle's central control system 140 to acquire real-time information such as the vehicle's pose data and driving status. It can also be electrically connected to the on-board communication unit 150 to perform batch transmission of annotated data, model weight updates, and cloud-based collaborative annotation operations.

[0033] The processor 110 collects raw sensor data of the environment in which the assisted driving perception target is located through the sensor module 120. The assisted driving perception target includes multi-wheeled vehicles. The processor 110 calculates basic perception data based on the collected raw sensor data. The basic perception data includes: 3D detection boxes of the assisted driving perception target, vehicle identification, vehicle category, and the current vehicle pose data. The processor 110 filters assisted driving perception targets based on fine-grained attributes to determine candidate assisted driving perception targets and extracts local image data of key areas of the candidate assisted driving perception targets. The processor 110 inputs the local image data into a text-image alignment model to obtain image features. At the same time, the text used to describe the fine-grained attributes is input into the text-image alignment model to obtain text features. The processor 110 calculates the similarity between the image features and the text features, and labels the attributes corresponding to the text whose similarity meets the preset conditions as candidate assisted driving perception targets, and synchronously stores the final labeling results in the vehicle storage unit 130.

[0034] Specifically, the processor 110 can be an automotive-grade intelligent driving domain controller SOC, an in-vehicle edge computing chip, or a cloud-based high-performance computing processor; it integrates a multi-sensor fusion processing unit to perform operations such as time synchronization, spatial calibration, and basic perception data calculation of LiDAR point cloud and image data; it has a built-in AI acceleration computing unit that supports local deployment and real-time inference of pre-trained text-image alignment models, multimodal feature encoding, similarity calculation, and other AI algorithm operations, and can efficiently complete attribute labeling calculations for various perception targets such as multi-wheeled vehicles; it has multi-interface adaptation capabilities, including a MIPICSI interface, an in-vehicle Ethernet interface, and an LVDS interface connected to the sensor module 120, an eMMC and UFS high-speed storage interface connected to the in-vehicle storage unit 130, CAN, CANFD, and LIN interfaces connected to the vehicle central control system 140 and the in-vehicle bus, and a PCIe and USB high-speed communication interface connected to the in-vehicle communication unit 150. The sensor module 120 includes a lidar, an automotive-grade image sensor (vehicle-mounted camera), and an inertial measurement unit (IMU). The lidar, which can be a mechanical rotating type or a solid-state lidar, is installed on the roof, front, and sides of the vehicle to collect temporal 3D point cloud data of the driving scene, enabling accurate identification of the spatial position of small traffic participants such as multi-wheeled vehicles. The image sensor, using a high dynamic range vehicle-mounted camera, is installed on the front, rear, and sides of the vehicle, and synchronizes with the lidar in time and spatial calibration. It is used to collect continuous frame 2D image data of the driving scene, clearly capturing details of key areas of multi-wheeled vehicles and motor vehicles. The inertial measurement unit is rigidly connected to the vehicle body and is used to collect the vehicle's six-axis pose data, providing a pose reference for basic perception data calculation. The raw sensing data collected by the sensor module 120 is transmitted to the processor 110 in real time, providing full raw data support for attribute annotation. The vehicle storage unit 130 uses automotive-grade high-reliability, high-capacity storage chips to store raw sensor data, calculated basic perception data, weight files of pre-trained text-image alignment models, and final attribute annotation results output by the processor 110. It supports high-speed data reading and writing and long-term reliable storage, and can meet the storage needs of batch annotation of massive driving scenario data.

[0035] Specifically, please refer to Figure 2 , Figure 2 This is a flowchart illustrating an attribute annotation method for assisted driving perception targets provided in an embodiment of this application. Figure 2 The methods in [the document] are applied to, for example... Figure 1 The processor 110 shown includes the following steps in its method: S210 collects raw sensor data of the environment in which the target is perceived in assisted driving.

[0036] Among them, the target of assisted driving perception includes multi-wheeled vehicles; the raw sensing data is the time-series multi-source sensing data in real driving scenarios collected by the vehicle-mounted multi-sensor group, including at least lidar point cloud data, continuous frame image data, and vehicle inertial navigation pose data, which fully covers the spatial, texture, and motion information of various traffic participants in the driving scenario, and can accurately capture the complete feature information of various perception targets such as multi-wheeled vehicles.

[0037] S220 obtains basic sensing data based on the original sensing data.

[0038] The basic perception data includes: 3D bounding boxes of the perceived target, vehicle identifiers, vehicle categories, and the current vehicle's pose data. The basic perception data is prior ground truth data obtained by preprocessing the raw sensor data using mature automated annotation tools. The 3D bounding boxes contain three-dimensional spatial information such as the spatial coordinates, size, and heading angle of the perceived target. The vehicle identifier is used to uniquely identify the same perceived target and associate it with its continuous motion trajectory. The vehicle category is the coarse-grained classification result of the perceived target. The current vehicle's pose data is the dynamic information such as the vehicle's position and attitude in the world coordinate system, providing basic data support for subsequent full-process attribute annotation.

[0039] S230, filter assisted driving perception targets based on fine-grained attributes to determine candidate assisted driving perception targets.

[0040] Among them, fine-grained attributes are the incremental attribute annotations required for training the assisted driving perception model. They are the core information for predicting the future movement trends of traffic participants and supporting downstream driving decisions and vehicle control. Candidate assisted driving perception targets are the core traffic participants that have a direct impact on the driving safety of the vehicle under the corresponding fine-grained attributes. By targeted screening, invalid interference targets can be eliminated, reducing the computational power consumption of subsequent model inference and improving the execution efficiency of the annotation process and the pertinence of attribute judgment.

[0041] S240, extract local image data of key areas of candidate assisted driving perception targets.

[0042] The key region refers to the specific area of ​​the vehicle body where the features of the corresponding fine-grained attributes are concentrated on the candidate assisted driving perception target. By using a transformation matrix from 3D coordinates to 2D pixel coordinates constructed based on the vehicle pose and camera intrinsic parameters, the 3D detection box of the candidate target is mapped to the 2D image with the corresponding timestamp. The local image data of the corresponding region is then cropped, which can effectively remove irrelevant background interference, strengthen the core features of the attributes, and improve the accuracy of subsequent attribute determination.

[0043] S250, input local image data into the text-image alignment model to obtain image features; and input text used to describe fine-grained attributes into the text-image alignment model to obtain text features.

[0044] The text-image alignment model is a pre-trained multimodal large-scale model with cross-modal image and text feature encoding and semantic alignment capabilities, requiring no additional fine-tuning or retraining for specific attribute labeling tasks. Image features are high-dimensional feature vectors obtained by the model through deep encoding of local image data, while text features are feature vectors of the same dimension as image features obtained by the model through encoding fine-grained attribute description text. Both provide the core data foundation for subsequent semantic matching judgment. Specifically, the text-image alignment model is the CLIP model.

[0045] S260, calculate the similarity between image features and text features, and label the attributes corresponding to texts whose similarity meets the preset conditions as candidate assisted driving perception targets.

[0046] The similarity is used to characterize the semantic matching degree between local image content and fine-grained attribute description text. The preset condition is to select the candidate description text with the highest similarity to the image features, and use the fine-grained attribute corresponding to the text as the final annotation result of the candidate assisted driving perception target, thus completing the fully automated attribute annotation of the perception target.

[0047] Specifically, in one possible embodiment, calculating the similarity between image features and text features includes: determining a first feature vector of the image features; and determining a second feature vector of the text features; calculating the cosine similarity between the first feature vector and the second feature vector to determine the similarity between the image features and the text features.

[0048] In this embodiment, the first feature vector and the second feature vector are vectors with completely identical dimensions, both being 512-dimensional, ensuring that they can be used to calculate the matching degree. The degree of matching between the two vectors is calculated using cosine similarity, thereby characterizing the semantic fit between the image content and the attribute description text. The core formula for calculating cosine similarity is: Similarity = cosθ = A B / |A|×|B|; The calculation result ranges from -1 to 1. A value closer to 1 indicates a higher degree of matching between the image and text features, meaning the image content closely matches the attributes described in the text. B is the dot product of the first and second feature vectors, |A| is the magnitude of the first feature vector, and |B| is the magnitude of the second feature vector. Finally, after calculating the similarity values ​​corresponding to all attribute description texts, the attribute corresponding to the text with the highest similarity is selected as the final annotation result for the perceived target. In this embodiment, by using a pre-trained CLIP model combined with a cosine similarity image-text matching mechanism, it is not necessary to train a dedicated classification model separately for different sub-attribute annotation tasks, thus reducing the cost and cycle of the annotation scheme. At the same time, the standardized cosine similarity calculation method eliminates the interference of irrelevant factors on the matching results. While improving the efficiency of attribute determination, it ensures the accuracy and generalization of the annotation results in complex driving scenarios, and efficiently completes the automated annotation of fine-grained attributes of assisted driving perception targets.

[0049] As can be seen, the attribute annotation method for assisted driving perception targets provided in this application firstly collects raw sensor data of the environment in which the assisted driving perception target is located, including multi-wheeled vehicles; secondly, it calculates basic perception data containing prior truth values ​​based on the raw sensor data, including the 3D detection box of the perception target, vehicle identification, vehicle category, and the pose data of the current vehicle; then, it filters perception targets based on fine-grained attributes of the assisted driving perception targets, determines candidate assisted driving perception targets, and extracts local image data of key regions of the candidate targets; next, it completes feature encoding of the local image data and the fine-grained attribute description text respectively through a text-image alignment model to obtain the corresponding image features and text features; finally, it calculates the similarity between the image features and the text features, and completes attribute annotation for the candidate assisted driving perception targets based on the matching results. In this way, by building a fully automated attribute labeling system through pre-trained multimodal large models, high-precision and high-efficiency labeling of fine-grained attributes of assisted driving perception targets can be achieved without manual intervention and training of dedicated classification models. This significantly reduces the labeling cost of the entire process and can quickly adapt to the incremental attribute labeling needs of massive driving data, providing high-quality ground truth data support for the rapid iteration of assisted driving perception models.

[0050] In one possible embodiment, the fine-grained attributes include at least: a first attribute characterizing whether a two-wheeled vehicle is in a riding state, a second attribute characterizing whether the brake lights of a four-wheeled vehicle are on, a third attribute characterizing whether the turn signals of a four-wheeled vehicle are on, and a fourth attribute characterizing whether the doors of a four-wheeled vehicle are open; the fine-grained attributes of the assisted driving perception targets are used to filter assisted driving perception targets to determine candidate assisted driving perception targets, including: for the first attribute, selecting two-wheeled vehicles as candidate assisted driving perception targets based on vehicle type; for the second attribute, selecting vehicles in the lanes directly in front of the current vehicle and in the adjacent lanes to the left and right based on the current vehicle's pose data as candidate assisted driving perception targets; for the third attribute, selecting vehicles within a preset distance, with an occlusion rate below a preset threshold, and exhibiting lateral deflection as candidate assisted driving perception targets, where lateral deflection means that the change in the heading angle of the 3D detection box of the four-wheeled vehicle meets a preset angle threshold; and for the fourth attribute, selecting vehicles adjacent to the future driving trajectory of the current vehicle as candidate assisted driving perception targets.

[0051] The four attributes are as follows: First, the riding status of the two-wheeled vehicle, which is the core indicator for determining whether the two-wheeled vehicle is carrying a passenger. Its status determines the accuracy of predicting the future trajectory of the two-wheeled vehicle and is a key reference information for assisted driving path planning and obstacle avoidance decisions in urban road scenarios. Second, the brake light status of the four-wheeled vehicle, which is used to determine the braking intention of the vehicle in front and is the core basis for longitudinal speed control and following distance maintenance of the vehicle, which is related to driving safety in following other vehicles and intersection scenarios. Third, the turn signal status of the four-wheeled vehicle, which is used to determine the lateral lane changing and turning intentions of surrounding vehicles and is a key reference information for decision-making and planning in lane changing, overtaking, and intersection scenarios. Fourth, the door status of the four-wheeled vehicle, which is used to determine the risk of doors opening on roadside vehicles and is a key reference information for collision avoidance warnings in urban roads, parking lots, and other scenarios.

[0052] Specifically, for the first attribute of two-wheeled vehicle target selection, based on the coarse classification results of vehicle categories in the basic perception data, all perceived targets belonging to the category of two-wheeled vehicles are directly extracted as candidate targets. This can cover various types of two-wheeled vehicles such as electric bicycles and ordinary motorcycles, ensuring that the riding status of all two-wheeled vehicles is labeled without omission. For the second attribute of target selection, based on the current vehicle's pose data and lane line perception results, four-wheeled vehicle targets in the lane directly in front of the vehicle and the two adjacent lanes on the left and right are identified. Through directional lane range filtering, vehicle targets in oncoming lanes and non-adjacent lanes to the side, which do not directly affect the vehicle's driving, can be eliminated. For the target selection based on the third attribute, the preset distance refers to the effective perception range around the vehicle. In this embodiment, it is set to a range of 50 meters around the vehicle by default, and can be adaptively adjusted according to different driving scenarios such as highways and urban roads. The occlusion rate refers to the proportion of the target vehicle area obscured by other obstacles. The preset threshold is set to 30% by default, and only vehicle targets with an occlusion rate of less than 30% are retained to ensure that the vehicle's turn signal area can be clearly captured, avoiding subsequent attribute misjudgments due to occlusion. The presence of lateral deflection refers to the change in the heading angle of the 3D detection frame of the four-wheeled vehicle meeting a preset angle threshold. In this embodiment, the preset angle threshold is set to 10°, that is, when the change in the heading angle of the target vehicle in consecutive frames is ≥10°, it is determined that there is lateral deflection. For the target selection based on the fourth attribute, the vehicle's driving trajectory within a preset future time period is predicted based on the vehicle's driving speed, heading angle, and lane information, and four-wheeled vehicle targets adjacent to this trajectory are selected. In this embodiment, the preset future time period is set to 5 seconds, and the proximity judgment criterion is that the shortest distance between the vehicle and the vehicle's future trajectory is less than 2 meters.

[0053] As can be seen, in this embodiment, by formulating targeted screening rules for core targets of interest with different fine-grained attributes, the perception targets that have a direct impact on the driving safety of the vehicle are accurately identified as candidate targets. This not only achieves comprehensive coverage of core targets of interest but also effectively eliminates invalid interfering targets, reduces the computational power consumption of subsequent model inference, and improves the execution efficiency of the entire annotation process. At the same time, the targeted screening rules ensure that the attribute feature regions of the candidate targets are clear and collectable, effectively reducing the probability of mislabeling and omission in subsequent attribute determination and improving the accuracy of the annotation results.

[0054] In one possible implementation, when annotating the first attribute, please refer to... Figure 3 , Figure 3 This is a flowchart illustrating the process of labeling a first attribute for a candidate assisted driving perception target, as provided in an embodiment of this application. Figure 3 As shown, it includes the following steps: S31, calculate the length of the two-wheeled vehicle's trajectory.

[0055] When labeling the first attribute, the 3D detection frame of the two-wheeled vehicle is transformed to the world coordinate system based on the vehicle's 3D bounding box and the current vehicle's pose data. The trajectory length is then calculated. The trajectory length refers to the cumulative distance the center points of consecutive 3D detection frames under the same vehicle identifier have moved in the world coordinate system. The world coordinate system is a unified global spatial coordinate system, which can eliminate the interference of the vehicle's own motion on the target displacement calculation and ensure the accuracy of the trajectory length.

[0056] Specifically, by associating consecutive 3D detection frames of the same two-wheeled vehicle target with the same vehicle identifier (tracking ID), the linear distance between the center points of adjacent frame detection frames in the world coordinate system is calculated sequentially. The total distance values ​​of all adjacent frames are then summed to obtain the complete trajectory length of the two-wheeled vehicle. By calculating the trajectory length, the moving and stationary states of the two-wheeled vehicle can be quickly distinguished, providing a core basis for efficient initial screening of the riding status.

[0057] S32, determine whether the length of the two-wheeled vehicle trajectory has reached the preset dynamic threshold; if yes, proceed to step S33; if no, proceed to step S34.

[0058] The preset dynamic threshold is a critical trajectory length value used to distinguish between the moving and stationary states of a two-wheeled vehicle. It can be adaptively adjusted based on road conditions, sensor accuracy, and vehicle speed in the actual driving scenario. For example, in congested urban road scenarios, where vehicle speed is slow and the two-wheeled vehicle's displacement distance is short, the preset dynamic threshold can be set to 3 meters; in ordinary urban road scenarios, the preset dynamic threshold can be set to 5 meters; and in highway scenarios, where the two-wheeled vehicle's speed is high and the displacement distance is long, the preset dynamic threshold can be set to 10 meters. By adapting the dynamic threshold settings to match the movement characteristics of two-wheeled vehicles in different driving scenarios, the problem of misjudgment and omission with fixed thresholds is avoided, improving the accuracy of initial screening for moving and stationary states and scenario adaptability.

[0059] S33 indicates that the two-wheeled vehicle is in riding mode.

[0060] If the trajectory length reaches a preset dynamic threshold, it indicates that the two-wheeled vehicle is in a continuously moving state. This state can only be achieved by human riders and there is no possibility of unmanned autonomous movement. Therefore, the two-wheeled vehicle can be directly identified and labeled as being in a riding state without the need for subsequent image feature extraction and model inference, thus reducing the computational power consumption of model inference and significantly improving the efficiency of the labeling process.

[0061] S34, Obtain the first image features.

[0062] Specifically, by extracting local image data of the key areas of the two-wheeled vehicle, the first image feature is obtained based on the text-image alignment model. The key areas of the two-wheeled vehicle are the entire vehicle area.

[0063] Specifically, from the continuous trajectory of the stationary two-wheeled vehicle, the 3D detection box closest to the vehicle is selected. The image region of the two-wheeled vehicle corresponding to this frame has the highest resolution, which maximizes the accuracy of subsequent attribute determination. Based on the vehicle's pose and camera intrinsic parameters, a transformation matrix from 3D coordinates to 2D pixel coordinates is constructed. The 3D detection box is then transformed into the 2D image with the corresponding timestamp. Local image data of the entire two-wheeled vehicle region is cropped and input into the image encoder of the text-image alignment model. The output is the first image feature with fixed dimensions.

[0064] S35, Obtain the text features of the first text.

[0065] The first text is input into the text-image alignment model to obtain the text features of the first text. The first text is used to describe the first attribute, including: a mobile two-wheeled vehicle ridden by a person and a stationary two-wheeled vehicle ridden by an unmanned person.

[0066] S36, determine the annotation result of the first attribute of the two-wheeled vehicle.

[0067] The similarity between the first image feature and the text feature of the first text is calculated. The attribute corresponding to the first text with the highest similarity is used as the annotation result of the first attribute of the two-wheeled vehicle. The similarity is used to characterize the semantic matching degree between the local image content of the two-wheeled vehicle and the text description. Specifically, cosine similarity is used for calculation, and the calculation result ranges from [-1, 1]. The closer the value is to 1, the higher the matching degree between the image and the text. For example, if the similarity between the first image feature and the text feature of "a moving two-wheeled vehicle ridden by someone" is 0.82, and the similarity between the first image feature and the text feature of "a stationary two-wheeled vehicle without riders" is 0.28, then the two-wheeled vehicle is determined to be in a riding state.

[0068] S37 extends the first attribute to all 3D detection boxes corresponding to the overall trajectory of the two-wheeled vehicle.

[0069] As can be seen, in this embodiment, the riding state of the two-wheeled vehicle is first rapidly screened by calculating the trajectory length and determining the dynamic threshold, and the riding state of the dynamic two-wheeled vehicle is directly labeled. Then, the riding state of the stationary two-wheeled vehicle is determined by encoding the image and text features and matching similarity through a text-image alignment model. Finally, the labeling results are extended to all 3D detection boxes of the entire trajectory of the two-wheeled vehicle to complete the labeling. In this way, while reducing the computational power consumption of labeling and improving the labeling efficiency, high-precision labeling of the riding state of the two-wheeled vehicle is achieved in all scenarios, providing high-quality data support for the training and optimization of the assisted driving two-wheeled vehicle trajectory prediction model and obstacle avoidance decision model.

[0070] In one possible implementation, when annotating the second attribute, please refer to... Figure 4 , Figure 4This is a flowchart illustrating the process of labeling a second attribute for a candidate assisted driving perception target, as provided in an embodiment of this application. Figure 4 As shown, it includes the following steps: S41, Extract the second image features.

[0071] Specifically, when labeling the second attribute, local image data of the key areas of the four-wheeled vehicle are extracted, and the second image features are extracted through a text-image alignment model. The key areas of the four-wheeled vehicle include the taillight area.

[0072] Specifically, please refer to Figure 5 , Figure 5 This is a schematic diagram illustrating an application scenario for extracting image features according to an embodiment of this application, such as... Figure 5 As shown, a transformation matrix from 3D coordinates to 2D pixel coordinates is constructed based on the vehicle pose and camera intrinsic parameters. The 3D detection boxes of the selected candidate target vehicles are transformed into 2D images with corresponding timestamps. The heading angle of the vehicle's rear is locked by combining the heading angle of the 3D detection boxes. The core region 51 where the taillights are located is cropped to obtain local image data 52 of the taillight region. The cropped local image data 52 is input into the image encoder of the text-image alignment model 53, and the output is a second image feature 54 with a fixed dimension aligned with the text feature dimension. Specifically, as shown... Figure 5 In the scene shown, the second image feature 54 corresponds to the feature of brake lights being on during the day.

[0073] S42, Obtain the text features of the second text.

[0074] The second text is input into the text-image alignment model to obtain the text features of the second text. The second text is used to describe the second attribute, including: brake lights on during the day, brake lights off during the day, brake lights on at night, and brake lights off at night.

[0075] Specifically, four sets of descriptive texts corresponding to different lighting scenarios and different brake light states are sequentially input into the text encoder of the text-image alignment model, and four sets of text features with the same dimension as the second image features are output. The four sets of texts cover all mutually exclusive working conditions of brake light states in day and night scenarios, with no scene omissions or semantic overlaps.

[0076] S43, determine the annotation results of the second attribute of the four-wheeled vehicle.

[0077] The process involves calculating the similarity between the second image features and the text features of the second text. The attribute corresponding to the second text with the highest similarity is used as the annotation result for the second attribute of the four-wheeled vehicle. The similarity is used to characterize the semantic matching degree between the image content of the vehicle's taillight area and the text describing the brake light status. In this embodiment, cosine similarity, consistent with the overall scheme, is used for calculation. The calculation result ranges from -1 to 1. The closer the value is to 1, the higher the matching degree between the image content and the corresponding text description. For example, in a daytime strong light following scenario, if the cosine similarity between the second image feature and the text feature "brake lights on during the day" is 0.79, and the similarity with the other three sets of texts is all below 0.3, then the vehicle's brake lights are determined to be on. For vehicle targets acquired from multiple consecutive frames, the average similarity across multiple frames can be calculated. The attribute corresponding to the text with the highest average similarity is used as the final annotation result, avoiding misjudgments caused by strong light reflection or temporary occlusion in a single frame image, further improving the stability of the annotation results.

[0078] As can be seen, in this embodiment, the core features of brake light status annotation are first extracted by creating a local image of the vehicle's taillight region and encoding the image features. Then, text feature encoding is completed using brake light status description text covering the entire scene. Finally, brake light status annotation is completed based on image-text feature similarity matching. Thus, by focusing on image extraction of core feature regions and providing accurate text descriptions that fit the scene, a dedicated brake light classification model can be trained. This allows for adaptive adaptation to complex driving scenarios with varying day and night lighting conditions. While reducing annotation computational costs, it improves the accuracy and scene generalization of brake light status annotation, providing high-quality data support for training decision-making models for assisted driving.

[0079] In one possible implementation, when annotating a third attribute, please refer to... Figure 6 , Figure 6 This is a flowchart illustrating the process of labeling a third attribute for a candidate assisted driving perception target, as provided in an embodiment of this application. Figure 6 As shown, it includes the following steps: S61, extract the features of the third image.

[0080] When labeling the third attribute, local image data of the key areas of the four-wheeled vehicle are extracted, and the third image features are extracted through a text-image alignment model. The key areas of the four-wheeled vehicle include the taillight area.

[0081] Specifically, based on the vehicle's pose and the heading angle of the 3D detection bounding boxes of candidate targets within the camera, the spatial orientation of the target vehicle relative to the vehicle is determined, accurately locating the core area where the turn signals are located: for vehicles traveling in the same direction in front of the vehicle, the taillight turn signal areas on the left and right sides of the rear are located; for vehicles traveling in the opposite direction or intersecting laterally, the front turn signal areas on the left and right sides of the front of the vehicle are located. After spatial mapping is completed through a 3D coordinate to 2D pixel coordinate transformation matrix, local image data of the core area of ​​the turn signals is cropped. Considering that turn signals have the periodic flashing characteristics specified by national standards, single-frame images are prone to missed detections when the turn signals are just in an off state. Therefore, a sequence of local image data of the turn signal area of ​​6 consecutive frames under the same vehicle identification is extracted and input into the image encoder of the text-image alignment model to obtain a third image feature sequence corresponding to multiple frames and aligned with the text feature dimensions.

[0082] S62, Obtain the text features of the third text.

[0083] The third text is input into the text-image alignment model to obtain the text features of the third text. The third text is used to describe the third attributes, including: only the left turn signal is on during the day, only the right turn signal is on during the day, both left and right turn signals are on during the day, both left and right turn signals are off during the day, only the left turn signal is on at night, only the right turn signal is on at night, both left and right turn signals are on at night, and both left and right turn signals are off at night.

[0084] Specifically, eight sets of descriptive texts, each corresponding to different lighting scenarios and different turn signal operating states, are sequentially input into the text encoder of the text-image alignment model, and the output is eight sets of text features of the same dimension as the third image features. The eight sets of texts cover daytime and nighttime full lighting scenarios, as well as four mutually exclusive turn signal operating conditions: left turn, right turn, hazard warning, and fully off. There are no scene omissions or semantic overlaps, and the turn signal status is fully covered in all driving scenarios such as lane changing, turning, and emergency stopping.

[0085] S63, determine the annotation results of the third attribute of the four-wheeled vehicle.

[0086] Specifically, the similarity between the third image features and the text features of the third text is calculated, and the attribute corresponding to the third text with the highest similarity is used as the annotation result of the third attribute of the four-wheeled vehicle.

[0087] Specifically, the matching degree of image and text features is calculated using cosine similarity, consistent with the overall scheme. The calculation result ranges from -1 to 1, with a value closer to 1 indicating a higher semantic matching degree between the image content and the corresponding text description. For a sequence of third image features across multiple consecutive frames, the cosine similarity between each frame's image features and eight sets of text features is calculated. The average of the multiple frame similarities is taken, and the attribute corresponding to the text with the highest average is used as the final annotation result for the vehicle's third attribute. For example, in a lane-changing scenario on urban roads, the average similarity between six consecutive frame image features and "only the left turn signal is on during the day" is 0.81, while the average similarity with the other seven sets of text is below 0.36, thus determining that the vehicle's left turn signal is on. For vehicles making emergency stops at intersections, the average similarity between multiple consecutive frame features and "both left and right turn signals are on simultaneously during the day" is the highest, thus determining that the vehicle's hazard lights are on. Through the multi-frame similarity averaging judgment logic, the periodic flashing characteristics of turn signals can be further adapted, avoiding misjudgments caused by a single frame image being off, and ensuring the stability of the annotation results.

[0088] As can be seen, in this embodiment, firstly, based on the temporal flickering characteristics and feature distribution of the turn signal status annotation, a continuous multi-frame local image sequence of the vehicle's turn signal area is locked and extracted to complete the standardized encoding of the multi-frame image features; then, through refined descriptive text covering all day and night scenarios and all turn signal operating conditions, the corresponding text features are encoded; finally, based on the average similarity matching of the multi-frame image and text features, the automatic annotation of the vehicle's turn signal status is completed. Thus, by extracting the flickering characteristics of the turn signal from a continuous frame sequence and covering the recognition requirements of all operating conditions with scenario-based text descriptions, there is no need to train a dedicated turn signal classification model. This allows for adaptive adaptation to complex driving scenarios with different day and night lighting, different vehicle types, and different driving conditions. While reducing annotation development costs, it significantly improves the accuracy, stability, and scenario adaptability of turn signal status annotation.

[0089] In one possible implementation, when annotating the fourth attribute, please refer to... Figure 7 , Figure 7 This is a flowchart illustrating the process of labeling a fourth attribute for a candidate assisted driving perception target, as provided in an embodiment of this application. Figure 7 As shown, it includes the following steps: S71, extract the fourth image features.

[0090] When labeling the fourth attribute, local image data of the key areas of the four-wheeled vehicle are extracted, and the fourth image features are extracted through a text-image alignment model. The key areas of the four-wheeled vehicle include the door area of ​​the four-wheeled vehicle.

[0091] Specifically, based on the vehicle's pose and the heading angle of the 3D detection bounding boxes of candidate targets within the camera, the spatial orientation of the target vehicle relative to the vehicle and the visible range of the vehicle's sides are determined, accurately locating the core areas on the left and right sides of the vehicle where the doors are located. After spatial mapping is completed through a 3D coordinate to 2D pixel coordinate transformation matrix, combined with the vehicle's body size ratio, local image data of the door installation area is cropped to fully cover the complete opening and closing boundaries of the front and rear doors. For vehicles in core risk scenarios such as stationary vehicles on the roadside or slow-moving vehicles, the image corresponding to the 3D detection bounding box closest to the vehicle with the lowest occlusion rate is selected first to ensure that the details of the door area are clearly discernible. For targets with slight occlusion, based on the vehicle body outline and national standards for door installation, a fixed area where the door is located is locked and cropped to avoid loss of core features due to partial occlusion. The cropped local image data of the door area is input into the image encoder of the text-image alignment model, and the output is a fixed-dimensional fourth image feature aligned with the text feature dimension.

[0092] S72, Obtain the text features of the fourth text.

[0093] The fourth text is input into the text-image alignment model to obtain the text features of the fourth text. The fourth text is used to describe the fourth attribute, including: at least one door on the left side of the vehicle is open, at least one door on the right side of the vehicle is open, and all doors of the vehicle are closed.

[0094] Specifically, the three sets of descriptive texts corresponding to different door opening and closing states and different risk levels are sequentially input into the text encoder of the text-image alignment model, and the output is three sets of text features with the same dimension as the fourth image features. The three sets of texts cover all mutually exclusive working conditions of vehicle door states, with no scene omissions or semantic overlaps, and correspond to three core scenarios: left door opening risk, right door opening risk, and no door opening risk.

[0095] S73, determine the annotation results of the fourth attribute of a four-wheeled vehicle.

[0096] Specifically, the similarity between the fourth image features and the text features of the fourth text is calculated, and the attribute corresponding to the fourth text with the highest similarity is used as the annotation result of the fourth attribute of the four-wheeled vehicle.

[0097] Specifically, the semantic matching degree of image and text features is calculated using cosine similarity, consistent with the overall scheme. The calculation result ranges from -1 to 1, with a value closer to 1 indicating a higher degree of matching between the image content and the corresponding text description. For vehicle targets captured in multiple consecutive frames, the cosine similarity between the image features of each frame and the three sets of text features is calculated separately. The average of the similarities across multiple frames is taken, and the attribute corresponding to the text with the highest average is used as the final annotation result. This avoids misjudgments caused by pedestrian occlusion, shooting angle deviation, or temporary obstacle occlusion in a single frame image, further improving the stability and reliability of the annotation results. For example, in a roadside parking scenario on an urban road, if the cosine similarity between the fourth image feature of the target vehicle and the text feature "at least one door on the right side of the vehicle is open" is 0.87, and the similarity with the other two sets of text is less than 0.3, then the vehicle's right door is determined to be open. If the average similarity of the target vehicle across multiple frames matches "all vehicle doors are closed" to the highest degree, then the vehicle's doors are determined to be closed.

[0098] As can be seen, in this embodiment, the core risk scenarios and feature distribution of the vehicle door status are first identified and local images of the vehicle door area are extracted to complete standardized image feature encoding. Then, descriptive text covering all working conditions of the vehicle door and distinguishing the opening and closing directions is used to complete the encoding of corresponding text features. Finally, based on image-text feature similarity matching, the automatic annotation of the vehicle door status is completed. In this way, by focusing on feature extraction of the core risk areas of the vehicle door and providing refined text descriptions that meet the requirements of collision avoidance, different vehicle door status recognition needs can be adaptively adapted without training a dedicated vehicle door status classification model. This reduces the computational cost of annotation while improving the accuracy and scenario adaptability of vehicle door status annotation.

[0099] This application embodiment can divide the electronic device into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0100] and Figure 2 The implementation is consistent with the previous one; please refer to [link / reference]. Figure 8 , Figure 8This is a functional unit block diagram of an attribute annotation device for assisted driving perception targets provided in this application embodiment. The attribute annotation device 800 for assisted driving perception targets includes: a determination unit 810, an acquisition unit 820, an extraction unit 830, a feature extraction unit 840, and an annotation unit 850. The determination unit 810 is used to determine the annotation requirements for fine-grained attributes of the assisted driving perception targets, including two-wheeled vehicles and four-wheeled vehicles. The acquisition unit 820 is used to acquire raw sensor data of the environment in which the perception targets are located, and obtain basic perception data based on the raw sensor data. The basic perception data includes: the 3D detection box of the assisted driving perception target, vehicle identification, vehicle category, and the current vehicle pose. The system includes: a data extraction unit 830, used to filter assisted driving perception targets based on fine-grained attributes to determine candidate assisted driving perception targets, and extract local image data of key regions of candidate assisted driving perception targets, where key regions refer to specific areas of the vehicle body of the candidate assisted driving perception targets; a feature extraction unit 840, used to input local image data into a text-image alignment model to obtain image features; and input text describing fine-grained attributes into a text-image alignment model to obtain text features; and a labeling unit 850, used to calculate the similarity between image features and text features, and label the attributes corresponding to text whose similarity meets preset conditions as candidate assisted driving perception targets.

[0101] In one possible embodiment, the fine-grained attributes include at least: a first attribute characterizing whether a two-wheeled vehicle is in a riding state, a second attribute characterizing whether the brake lights of a four-wheeled vehicle are on, a third attribute characterizing whether the turn signals of a four-wheeled vehicle are on, and a fourth attribute characterizing whether the doors of a four-wheeled vehicle are open; the extraction unit 830 is specifically used to: for the first attribute, select two-wheeled vehicles as candidate driver assistance perception targets based on vehicle type; for the second attribute, select vehicles in the lanes directly in front of the current vehicle and in the adjacent lanes to the left and right based on the current vehicle's pose data as candidate driver assistance perception targets; for the third attribute, select vehicles within a preset distance, with an occlusion rate below a preset threshold, and exhibiting lateral deflection as candidate driver assistance perception targets, where lateral deflection means that the change in the heading angle of the 3D detection box of the four-wheeled vehicle meets a preset angle threshold; and for the fourth attribute, select vehicles adjacent to the future driving trajectory of the current vehicle as candidate driver assistance perception targets.

[0102] In one possible embodiment, the annotation unit 850 is used to annotate candidate assisted driving perception targets. Specifically, when annotating the first attribute, the annotation unit 850 converts the 3D detection box of the two-wheeled vehicle to the world coordinate system based on the 3D detection box of the two-wheeled vehicle and the pose data of the current vehicle, and calculates the trajectory length. The trajectory length refers to the cumulative movement distance of the center point of consecutive 3D detection boxes under the same vehicle identifier in the world coordinate system. If the trajectory length reaches a preset dynamic threshold, the two-wheeled vehicle is labeled as being in a riding state. If the trajectory length does not reach the preset dynamic threshold, local image data of the key area of ​​the two-wheeled vehicle is extracted, and the first image feature is obtained through a text-image alignment model. The key area of ​​the two-wheeled vehicle is the entire vehicle area of ​​the two-wheeled vehicle. The first text is input into the text-image alignment model to obtain the text features of the first text. The first text is used to describe the first attribute, including: a moving two-wheeled vehicle ridden by a person and a stationary two-wheeled vehicle ridden by an unmanned person. The similarity between the first image feature and the text features of the first text is calculated, and the attribute corresponding to the first text with the highest similarity is used as the annotation result of the first attribute of the four-wheeled vehicle. The first attribute is extended to all 3D detection boxes corresponding to the overall trajectory of the two-wheeled vehicle.

[0103] In one possible embodiment, the annotation unit 850 is specifically used to: when annotating the second attribute, extract local image data of the key area of ​​the four-wheeled vehicle, extract the second image features through a text-image alignment model, the key area of ​​the four-wheeled vehicle includes the taillight area of ​​the four-wheeled vehicle; input the second text into the text-image alignment model to obtain the text features of the second text, the second text is used to describe the second attribute, including: brake lights on during the day, brake lights off during the day, brake lights on at night, brake lights off at night; calculate the similarity between the second image features and the text features of the second text, and take the attribute corresponding to the second text with the highest similarity as the annotation result of the second attribute of the four-wheeled vehicle.

[0104] In one possible embodiment, the annotation unit 850 is specifically used to: when annotating the third attribute, extract local image data of the key area of ​​the four-wheeled vehicle, extract the third image features through a text-image alignment model, the key area of ​​the four-wheeled vehicle includes the taillight area of ​​the four-wheeled vehicle; input the third text into the text-image alignment model to obtain the text features of the third text, the third text is used to describe the third attribute, including: only the left turn signal is on during the day, only the right turn signal is on during the day, both left and right turn signals are on during the day, both left and right turn signals are off during the day, only the left turn signal is on at night, only the right turn signal is on at night, both left and right turn signals are on at night, and both left and right turn signals are off at night; calculate the similarity between the third image features and the text features of the third text, and take the attribute corresponding to the third text with the highest similarity as the annotation result of the third attribute of the four-wheeled vehicle.

[0105] In one possible embodiment, the annotation unit 850 is specifically used to: when annotating the fourth attribute, extract local image data of the key area of ​​the four-wheeled vehicle, extract the fourth image features through a text-image alignment model, the key area of ​​the four-wheeled vehicle includes the door area of ​​the four-wheeled vehicle; input the fourth text into the text-image alignment model to obtain the text features of the fourth text, the fourth text is used to describe the fourth attribute, including: at least one door on the left side of the vehicle is open, at least one door on the right side of the vehicle is open, and all doors of the vehicle are closed; calculate the similarity between the fourth image features and the text features of the fourth text, and take the attribute corresponding to the fourth text with the highest similarity as the annotation result of the fourth attribute of the four-wheeled vehicle.

[0106] In one possible embodiment, the text-image alignment model is the CLIP model; the annotation unit 850 is specifically used to: determine a first feature vector of the image features; and determine a second feature vector of the text features; calculate the cosine similarity between the first feature vector and the second feature vector to determine the similarity between the image features and the text features.

[0107] It is understood that since the method embodiments and the device embodiments are different presentations of the same technical concept, the content of the method embodiment section in this application should be adapted to the device embodiment section in a synchronous manner, and will not be repeated here.

[0108] Figure 9 This is a structural block diagram of an electronic device provided in an embodiment of this application. For example... Figure 9 As shown, the electronic device 900 may include one or more of the following components: a processing module 901 and a memory 902 coupled to the processing module 901, wherein the memory 902 may store one or more computer programs, which may be configured to implement the methods described in the examples above when executed by one or more processing modules 901. The electronic device 900 may be as follows: Figure 1 The processor 110 shown.

[0109] The processing module 901 may include one or more processing cores. The processing module 901 connects to various parts within the electronic device 900 using various interfaces and lines. It executes various functions and processes data of the electronic device 900 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 902, and by calling data stored in the memory 902. Optionally, the processing module 901 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processing module 901 may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. It is understood that the aforementioned modem may also not be integrated into the processing module 901 and may be implemented separately through a communication chip.

[0110] The memory 902 may include random access memory (RAM) or read-only memory (ROM). The memory 902 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 902 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the above-described method examples. The data storage area may also store data created during the use of the electronic device 900.

[0111] It is understood that the electronic device 900 may include more or fewer structural elements than those shown in the above block diagram, such as a power module, physical buttons, WiFi (Wireless Fidelity) module, speaker, Bluetooth module, sensor, etc., without limitation.

[0112] This application also provides a computer storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements some or all of the steps of any of the methods described in the above method embodiments.

[0113] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments.

[0114] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0115] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and there may be other division methods in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0116] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0117] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically comprised separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or in the form of hardware plus software functional units.

[0118] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute partial steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes: a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, volatile memory, or non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM), etc., which are various media capable of storing program code.

[0119] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can easily conceive of variations or substitutions without departing from the spirit and scope of the present invention, and various modifications and alterations can be made, including combinations of the different functions and implementation steps described above, as well as software and hardware implementation methods, all of which are within the protection scope of the present invention.

Claims

1. A method for attribute labeling of a target perceived in assisted driving, characterized in that, The method includes: Collect raw sensor data of the environment in which the assisted driving perception target is located. The assisted driving perception target includes multi-wheeled vehicles, including two-wheeled vehicles and four-wheeled vehicles. Based on the original sensing data, basic perception data is obtained, which includes: the 3D detection box of the assisted driving perception target, vehicle identification, vehicle category, and the current vehicle pose data. The assisted driving perception targets are screened based on fine-grained attributes to determine candidate assisted driving perception targets, and local image data of key areas of the candidate assisted driving perception targets are extracted. The fine-grained attributes include at least: a first attribute for characterizing whether the two-wheeled vehicle is in a riding state, a second attribute for characterizing whether the brake lights of the four-wheeled vehicle are on, a third attribute for characterizing whether the turn signals of the four-wheeled vehicle are on, and a fourth attribute for characterizing whether the doors of the four-wheeled vehicle are open. The local image data is input into the text-image alignment model to obtain image features; and the text used to describe the fine-grained attributes is input into the text-image alignment model to obtain text features. Determine a first feature vector of the image features; and determine a second feature vector of the text features; Calculate the cosine similarity between the first feature vector and the second feature vector to determine the similarity between the image features and the text features, and label the attributes corresponding to the text whose similarity meets the preset conditions as the candidate assisted driving perception targets.

2. The method according to claim 1, characterized in that, The step of filtering the assisted driving perception targets based on fine-grained attributes to determine candidate assisted driving perception targets includes: Based on the first attribute, the two-wheeled vehicle is selected as the candidate assisted driving perception target according to the vehicle category; For the second attribute, based on the current vehicle's pose data, vehicles in the lanes directly in front of the current vehicle and in the adjacent lanes to the left and right are selected as candidate assisted driving perception targets. For the third attribute, vehicles within a preset distance, with an occlusion rate lower than a preset threshold and exhibiting lateral deflection are selected as candidate assisted driving perception targets. The presence of lateral deflection means that the change in the heading angle of the 3D detection frame of the four-wheeled vehicle meets a preset angle threshold. Based on the fourth attribute, vehicles adjacent to the current vehicle's future driving trajectory are selected as candidate assisted driving perception targets.

3. The method according to claim 2, characterized in that, The step of labeling the candidate assisted driving perception targets includes: When the first attribute is labeled, the 3D detection frame of the two-wheeled vehicle is converted to the world coordinate system based on the 3D detection frame of the two-wheeled vehicle and the pose data of the current vehicle, and the trajectory length is calculated. The trajectory length refers to the cumulative movement distance of the center point of consecutive 3D detection frames under the same vehicle identifier in the world coordinate system. If the trajectory length reaches a preset dynamic threshold, the two-wheeled vehicle is marked as being in the riding state. If the trajectory length does not reach the preset dynamic threshold, then local image data of the key area of ​​the two-wheeled vehicle is extracted, and the first image feature is obtained through the text-image alignment model. The key area of ​​the two-wheeled vehicle is the whole vehicle area of ​​the two-wheeled vehicle. The first text is input into the text-image alignment model to obtain the text features of the first text. The first text is used to describe the first attribute, including: a mobile two-wheeled vehicle ridden by a person and a stationary two-wheeled vehicle ridden by an unmanned person. Calculate the similarity between the first image features and the text features of the first text, and take the attribute corresponding to the first text with the highest similarity as the annotation result of the first attribute of the two-wheeled vehicle; The first attribute is extended to all 3D detection boxes corresponding to the overall trajectory of the two-wheeled vehicle.

4. The method according to claim 2, characterized in that, The step of labeling the candidate assisted driving perception targets includes: When the second attribute is labeled, local image data of the key area of ​​the four-wheeled vehicle is extracted, and the second image features are extracted through the text-image alignment model. The key area of ​​the four-wheeled vehicle includes the taillight area of ​​the four-wheeled vehicle. The second text is input into the text-image alignment model to obtain the text features of the second text. The second text is used to describe the second attribute, including: brake lights on during the day, brake lights off during the day, brake lights on at night, and brake lights off at night. Calculate the similarity between the second image features and the text features of the second text, and take the attribute corresponding to the second text with the highest similarity as the annotation result of the second attribute of the four-wheeled vehicle.

5. The method according to claim 2, characterized in that, The step of labeling the candidate assisted driving perception targets includes: When the third attribute is labeled, local image data of the key area of ​​the four-wheeled vehicle is extracted, and the third image features are extracted through the text-image alignment model. The key area of ​​the four-wheeled vehicle includes the taillight area of ​​the four-wheeled vehicle. The third text is input into the text-image alignment model to obtain the text features of the third text. The third text is used to describe the third attribute, including: only the left turn signal is on during the day, only the right turn signal is on during the day, both left and right turn signals are on during the day, both left and right turn signals are off during the day, only the left turn signal is on at night, only the right turn signal is on at night, both left and right turn signals are on at night, and both left and right turn signals are off at night. Calculate the similarity between the third image features and the text features of the third text, and take the attribute corresponding to the third text with the highest similarity as the annotation result of the third attribute of the four-wheeled vehicle.

6. The method according to claim 2, characterized in that, The step of labeling the candidate assisted driving perception targets includes: When the fourth attribute is labeled, local image data of the key area of ​​the four-wheeled vehicle is extracted, and the fourth image feature is extracted through the text-image alignment model. The key area of ​​the four-wheeled vehicle includes the door area of ​​the four-wheeled vehicle. The fourth text is input into the text-image alignment model to obtain the text features of the fourth text. The fourth text is used to describe the fourth attribute, including: at least one door on the left side of the vehicle is open, at least one door on the right side of the vehicle is open, and all doors of the vehicle are closed. Calculate the similarity between the fourth image feature and the text feature of the fourth text, and take the attribute corresponding to the fourth text with the highest similarity as the annotation result of the fourth attribute of the four-wheeled vehicle.

7. An attribute labeling device for assisted driving target perception, characterized in that, The device includes: a data acquisition unit, an acquisition unit, a filtering unit, a feature extraction unit, and a labeling unit; wherein, The acquisition unit is used to acquire raw sensor data of the environment in which the assisted driving perception target is located. The assisted driving perception target includes multi-wheeled vehicles, which include two-wheeled vehicles and four-wheeled vehicles. The acquisition unit is used to obtain basic perception data based on the original sensing data. The basic perception data includes: the 3D detection box of the assisted driving perception target, the vehicle identification, the vehicle category, and the current vehicle pose data. The filtering unit is used to filter the assisted driving perception target based on the fine-grained attributes of the assisted driving perception target to determine candidate assisted driving perception targets, and extract local image data of key areas of the candidate assisted driving perception targets. The fine-grained attributes include at least: a first attribute for characterizing whether the two-wheeled vehicle is in a riding state, a second attribute for characterizing whether the brake lights of the four-wheeled vehicle are on, a third attribute for characterizing whether the turn signals of the four-wheeled vehicle are on, and a fourth attribute for characterizing whether the doors of the four-wheeled vehicle are open. The feature extraction unit is used to input the local image data into the text-image alignment model to obtain image features; and to input the text used to describe the fine-grained attributes into the text-image alignment model to obtain text features; The annotation unit is used to determine a first feature vector of the image features and a second feature vector of the text features. Calculate the cosine similarity between the first feature vector and the second feature vector to determine the similarity between the image features and the text features, and label the attributes corresponding to the text whose similarity meets the preset conditions as the candidate assisted driving perception targets.

8. An electronic device, characterized in that, It includes a processor and a memory, the memory being used to store one or more programs and configured to be executed by the processor, the programs including instructions for performing the steps of the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, A computer program for storing electronic data interchange is provided, wherein the computer program causes a computer to perform the method as described in any one of claims 1-6.

Images