Multi-view three-dimensional target fusion labeling method and device and computer equipment

Abstract

The application relates to a multi-view three-dimensional target fusion labeling method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a plurality of single-end labeling data and reference labeling data in the plurality of single-end labeling data; according to a reference timestamp corresponding to the reference labeling data, the single-end timestamps corresponding to the plurality of single-end labeling data are matched and screened to obtain a set of to-be-fused labeling data; according to the reference timestamp, the labeling data in the set of to-be-fused labeling data are time-aligned to form time-aligned labeling data; according to the labeling frame information of the time-aligned labeling data, multi-view labeling frames are constructed; and multi-view three-dimensional target fusion labeling is performed according to the multi-view labeling frames. The scheme can not only improve the accuracy of the labeling information after fusion, but also improve the reliability of the scheme when applied to multi-scene labeling tasks.

Description

Technical Field

[0001] This application relates to the field of automated / semi-automated annotation engineering technology, and in particular to a multi-view 3D target fusion annotation method, device, computer equipment, storage medium and computer program product. Background Technology

[0002] With the development of deep learning technology, data-driven models and methods are widely used for perception and understanding of complex real-world scenarios. However, this has led to a surge in demand for labeled data suitable for supervised training. To ensure the accuracy of supervised information, most existing annotation processes rely on human annotators. However, manual annotation is extremely costly, and as the difficulty and complexity of perception tasks increase, the required level of expertise among human annotators also rises. Therefore, utilizing automated annotation technology to assist or extend the manual annotation process has become a key technical aspect of annotation engineering.

[0003] In recent years, a series of automated or semi-automated annotation technologies have emerged to meet the target perception needs of practical industries. Addressing the inevitable occlusion and truncation of target objects from different viewing angles, a significant amount of research and engineering work has focused on target detection tasks from multiple perspectives. With the development of multi-agent collaborative perception tasks, the need for multi-view collaborative annotation has also arisen.

[0004] However, for multi-view information fusion tasks, traditional technical solutions mostly focus on feature fusion for specific perception task models. For example, they generate bird's-eye view fusion features by fusing visual features from multiple perspectives and then use these features to achieve 3D object detection; or they improve 3D object detection performance by fusing point cloud features from multiple perspectives in a raster format. The feature-level fusion in these traditional technical solutions relies on the model's implicit ability to express the relative pose relationships between perspectives, which makes their reliability insufficient when transferred to multi-scene annotation tasks. Summary of the Invention

[0005] Therefore, it is necessary to provide a more reliable multi-view 3D target fusion annotation method, device, computer equipment, computer-readable storage medium, and computer program product to address the aforementioned technical problems.

[0006] Firstly, this application provides a multi-view 3D target fusion annotation method. The method includes:

[0007] Acquire multiple single-ended annotation data, and the baseline annotation data among the multiple single-ended annotation data;

[0008] Based on the benchmark timestamps corresponding to the benchmark annotation data, the single-end timestamps corresponding to the multiple single-end annotation data are matched and filtered to obtain a set of annotation data to be fused.

[0009] Based on the reference timestamp, the labeled data in the set of labeled data to be fused are time-aligned to form time-aligned labeled data;

[0010] Based on the annotation box information corresponding to the time-aligned annotation data, a multi-view annotation box is constructed;

[0011] Based on the multi-view annotation box, generate a multi-view 3D target fusion annotation result.

[0012] In one embodiment, the step of matching and filtering the single-end timestamps corresponding to the plurality of single-end annotation data according to the reference timestamps corresponding to the reference annotation data to obtain the set of annotation data to be fused includes:

[0013] If the timestamp difference between the single-end timestamp and the reference timestamp is less than the preset data sampling period, then multiple single-end labeled data will be combined to form a set of labeled data to be fused. The single-end timestamp is the timestamp corresponding to multiple single-end labeled data, and the reference timestamp is the timestamp corresponding to the reference labeled data.

[0014] If the timestamp difference between the single-end timestamps corresponding to multiple single-end labeled data and the reference timestamp corresponding to the reference labeled data is not less than the preset data sampling period, then the labeled data at the current moment is determined.

[0015] Generate candidate annotation data for the current time step, wherein the candidate annotation data includes the annotation data from the previous time step and the annotation data from the next time step;

[0016] Based on the baseline timestamp corresponding to the baseline annotation data and the preset data sampling period, the candidate timestamps corresponding to the candidate annotation data are matched and filtered to obtain a set of annotation data to be fused.

[0017] In one embodiment, the step of performing time alignment on the labeled data in the set of labeled data to be fused, based on the reference timestamp, to form time-aligned labeled data includes:

[0018] Based on the temporal information of the single-end labeled data in the labeled data set to be fused, the first labeled data and the second labeled data are extracted from the labeled data set to be fused, wherein the timestamps of the first labeled data and the second labeled data are adjacent;

[0019] Linear interpolation is performed based on the first annotation data, the second annotation data, and the reference timestamp to obtain the time-aligned annotation data corresponding to the reference timestamp.

[0020] In one embodiment, constructing a multi-view annotation box based on the annotation box information corresponding to the time-aligned annotation data includes:

[0021] The coordinate system is one for the annotation box information corresponding to the time alignment annotation data;

[0022] According to the pre-defined annotation dictionary of the observed object, the annotation box information after coordinate system one is matched and classified. The annotation dictionary is formed by combining several descriptive fields of a single observed object.

[0023] Based on the matching and classification results, the bounding box information in a single category is combined to form a multi-view bounding box.

[0024] In one embodiment, the step of combining the bounding box information in a single category to form a multi-view bounding box based on the matching classification result includes:

[0025] Retrieve the attribute information and attribute values ​​corresponding to the annotation box information in a single category;

[0026] The attribute information corresponding to the annotation box information in a single category is merged to obtain the merged attribute information;

[0027] The attribute values ​​corresponding to the annotation box information in a single category are merged to obtain the merged attribute value;

[0028] The fusion attribute information and the fusion attribute values ​​are integrated to obtain a multi-view annotation box.

[0029] In one embodiment, fusing the attribute values ​​corresponding to the annotation box information in a single category to obtain the fused attribute value includes:

[0030] Obtain the distance information corresponding to the labeled information box in a single category, the distance information being used to characterize the distance between the image acquisition device and the observed object;

[0031] Based on the distance information, generate the confidence score corresponding to the labeled information box in a single category;

[0032] Based on the confidence level and the preset time-synchronized confidence decay parameter, the attribute values ​​corresponding to the bounding box information in a single category are fused to obtain the fused attribute value. The time-synchronized confidence decay parameter is used to characterize the impact of linear interpolation on the confidence level of the labeled data.

[0033] Secondly, this application also provides a multi-view 3D target fusion annotation device. The device includes:

[0034] The data acquisition module is used to acquire multiple single-ended annotation data and the baseline annotation data among the multiple single-ended annotation data;

[0035] The matching and filtering module is used to match and filter the single-end timestamps corresponding to the multiple single-end annotation data according to the benchmark timestamps corresponding to the benchmark annotation data, so as to obtain a set of annotation data to be fused.

[0036] The time alignment module is used to perform time alignment on the labeled data in the set of labeled data to be fused according to the reference timestamp, so as to form time-aligned labeled data;

[0037] The spatial alignment module is used to construct multi-view annotation boxes based on the annotation box information corresponding to the time alignment annotation data;

[0038] The annotation fusion module is used to generate multi-view 3D target fusion annotation results based on the multi-view annotation boxes.

[0039] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0040] Acquire multiple single-ended annotation data, and the baseline annotation data among the multiple single-ended annotation data;

[0041] Based on the benchmark timestamps corresponding to the benchmark annotation data, the single-end timestamps corresponding to the multiple single-end annotation data are matched and filtered to obtain a set of annotation data to be fused.

[0042] Based on the reference timestamp, the labeled data in the set of labeled data to be fused are time-aligned to form time-aligned labeled data;

[0043] Based on the annotation box information corresponding to the time-aligned annotation data, a multi-view annotation box is constructed;

[0044] Based on the multi-view annotation box, generate a multi-view 3D target fusion annotation result.

[0045] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0046] Acquire multiple single-ended annotation data, and the baseline annotation data among the multiple single-ended annotation data;

[0047] Based on the benchmark timestamps corresponding to the benchmark annotation data, the single-end timestamps corresponding to the multiple single-end annotation data are matched and filtered to obtain a set of annotation data to be fused.

[0048] Based on the reference timestamp, the labeled data in the set of labeled data to be fused are time-aligned to form time-aligned labeled data;

[0049] Based on the annotation box information corresponding to the time-aligned annotation data, a multi-view annotation box is constructed;

[0050] Based on the multi-view annotation box, generate a multi-view 3D target fusion annotation result.

[0051] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0052] Acquire multiple single-ended annotation data, and the baseline annotation data among the multiple single-ended annotation data;

[0053] Based on the benchmark timestamps corresponding to the benchmark annotation data, the single-end timestamps corresponding to the multiple single-end annotation data are matched and filtered to obtain a set of annotation data to be fused.

[0054] Based on the reference timestamp, the labeled data in the set of labeled data to be fused are time-aligned to form time-aligned labeled data;

[0055] Based on the annotation box information corresponding to the time-aligned annotation data, a multi-view annotation box is constructed;

[0056] Based on the multi-view annotation box, generate a multi-view 3D target fusion annotation result.

[0057] This application provides a method, apparatus, computer device, storage medium, and computer program product for multi-view 3D target fusion annotation. The solution first performs a matching verification on multiple collected single-end annotation data sets. This verification is done by matching and filtering the collected single-end annotation data against a determined baseline annotation data set, leveraging the temporal continuity of the collected data to mitigate time asynchrony issues caused by hardware time synchronization errors. After obtaining the set of annotation data to be fused through matching and filtering, the solution further aligns the data set to be fused with time based on the baseline timestamp, forming time-aligned annotation data. Post-annotation processing ensures that the annotation times of the data in the set are the same, thereby reducing the spatial fusion burden of multi-end data annotation frames. Furthermore, the scheme constructs multi-view annotation boxes based on the annotation box information corresponding to the time-aligned annotation data, and then generates multi-view 3D target fusion annotation results based on the multi-view annotation boxes. This scheme can effectively address the unavoidable spatiotemporal asynchrony problem in multi-end data acquisition and annotation engineering through spatiotemporal synchronization of multi-end annotations. It realizes the fusion of multiple single-view 3D target annotations based on multiple single-end annotation data, and fully considers the confidence difference caused by the observation distance of different views and the additional uncertainty introduced in the preceding spatiotemporal synchronization process. This not only improves the accuracy of the fused annotation information, but also improves the reliability of the scheme when it is applied to multi-scene annotation tasks. Attached Figure Description

[0058] Figure 1 This is an application environment diagram of a multi-view 3D target fusion annotation method in one embodiment;

[0059] Figure 2 This is a flowchart illustrating a multi-view 3D target fusion annotation method in one embodiment;

[0060] Figure 3 This is a flowchart illustrating the timing matching verification and re-matching sub-steps in one embodiment;

[0061] Figure 4 This is a flowchart illustrating a multi-view 3D target fusion annotation method in another embodiment;

[0062] Figure 5 This is a structural block diagram of a multi-view 3D target fusion annotation device in one embodiment;

[0063] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0065] With the development of multi-agent collaborative perception tasks, the need for multi-view collaborative annotation has emerged in traditional technical solutions to address the inevitable occlusion and truncation of target objects from different viewing angles. Compared to single-end, single-view annotation engineering, the technical challenge introduced by multi-view 3D target collaborative annotation lies in the coordination between multiple single-end annotations. Therefore, automated multi-view collaborative annotation mainly focuses on the automated fusion annotation of single-end annotation results. For multi-view information fusion tasks, existing technical solutions are mostly feature fusion for specific perception task models; for example, generating bird's-eye view fusion features through the fusion of visual features from multiple perspectives and using them to achieve 3D target detection; or improving 3D target detection performance by fusing point cloud features from multiple perspectives in a raster format. However, feature-level fusion relies on the model's implicit ability to express the relative pose relationships between perspectives, and its reliability and interpretability are insufficient when transferred to multi-scene annotation tasks.

[0066] To ensure the interpretability of automated fusion of multi-view annotations and to fully utilize existing single-view automated annotation technologies, this application proposes an automated fusion scheme based on single-view 3D annotation results. This scheme can be used for automated fusion of multi-view manual annotations and can also be cascaded with existing single-view automated annotation methods to assist in multi-view 3D target collaborative annotation projects. Specifically, this application addresses the asynchronous temporal and spatial dimensions in multi-view data acquisition and annotation projects by automatically achieving the fusion annotation of multi-view single-view 3D targets. This allows for the automated generation of multi-view collaborative annotations using only single-view 3D target annotation results, significantly reducing manual annotation costs, contributing to cost reduction and efficiency improvement in multi-view collaborative annotation, and supporting the development of multi-view target perception technology.

[0067] The multi-view 3D target fusion annotation method provided in this application can be applied to, for example... Figure 1In the application environment shown, image acquisition device 102 communicates with server 104 via a network, and terminal 106 also communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated on server 104 or placed on the cloud or other network servers. In this application environment, multiple image acquisition devices 102 pre-deployed in a specific scene first acquire real-time images of that scene; in this embodiment, all pre-deployed image acquisition devices 102 are single-view image acquisition devices, so the real-time images acquired by the image acquisition devices 102 are all single-view real-time images. The acquired single-view real-time images are uploaded to server 104, and the observed objects in the single-view real-time images are labeled using a pre-set target labeling program on the server, forming single-end labeled data. During the process of server 104 integrating a number of single-end labeled data, a specific image acquisition device 102 is selected as the main acquisition device from all the single-end labeled data. The labeled data acquired and labeled by the main acquisition device can be recorded as the reference labeled data. After server 104 completes the collection and processing of annotation data, it filters the collection time of the baseline data. Server 104 will construct a sampling time range based on the preset data collection cycle and the timestamp of the main pole annotation data. That is, using the specified main pole timestamp as the baseline, if the difference between the single-end timestamp of the other pole data in the same group (i.e., the single-end annotation data) and the baseline timestamp is less than the data collection cycle, then the data in that group satisfies the time sequence matching requirement; otherwise, multiple single-end annotation data need to be re-filtered. After determining that the time sequence matching requirement is met, server 104 integrates the single-end annotation data in the same group to form a set of annotation data to be fused. Further, server 104 will perform further time alignment at the annotation level based on the baseline timestamp, addressing minor time asynchrony in the set of annotation data to be fused, to obtain time-aligned annotation data. Then, server 104 will fuse the annotation box information corresponding to the time-aligned annotation data to form multi-view annotation boxes, and based on these multi-views, perform multi-view 3D target fusion annotation in any real-time image. Finally, server 104 obtains the multi-view 3D target fusion annotation results and sends them to terminal 106 for visualization display in the terminal 106's visual operation interface. The image acquisition device 102 can include, but is not limited to, cameras, webcams, and scanners. Terminal 106 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0068] In one embodiment, such as Figure 2 As shown, a multi-view 3D target fusion annotation method is provided, which is then applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0069] Step 202: Obtain multiple single-ended annotation data and the baseline annotation data from the multiple single-ended annotation data.

[0070] In this embodiment, multiple single-end labeled data are obtained by labeling observed objects from a single perspective. Single-end labeled data refers to the data content obtained by labeling observed objects in real-time images acquired by an image acquisition device with a fixed perspective. In the embodiment, the labeling can be automated using an automated labeling model; or, by utilizing the correlation of temporally continuous data, manually labeling preceding frames and inferring and predicting subsequent potential target labels based on this. The reference labeled data in the embodiment is the data content obtained by labeling observed objects in real-time images acquired by the main pole image acquisition device, and can also be referred to as main pole labeling data. Furthermore, the main pole and its relationship with other poles in the embodiment can be determined during the deployment of the image acquisition device or specified during the data acquisition and processing stage.

[0071] For example, firstly, the embodiment trains an automated annotation model based on a small batch of manually labeled data and generated virtual data, and stores the model locally on the server. When the server receives real-time images acquired and uploaded by the image acquisition device, it inputs the real-time images into the automated annotation model to annotate the observed objects in the images, forming multiple single-viewpoint, single-end annotation data. After the server obtains a certain number of single-end annotation data, based on the selected master acquisition device during the real-time image acquisition process, the real-time image acquired by that device is labeled as the reference image. The reference image is then input into the automated annotation model, and the output annotation data is also labeled as reference annotation data or master annotation data.

[0072] Step 204: Based on the benchmark timestamps corresponding to the benchmark annotation data, match and filter the single-end timestamps corresponding to multiple single-end annotation data to obtain the set of annotation data to be merged.

[0073] In this context, a timestamp is a sequence of characters or encoded information used to identify when a specific event occurs. More specifically, in this embodiment, the baseline timestamp describes the acquisition time of the real-time image corresponding to the baseline annotation data. Similar to the baseline timestamp, the single-end timestamp describes the acquisition time of the real-time image corresponding to single-view single-end annotation data. The set of annotation data to be fused is formed by combining several numbers of single-end annotation data; and all single-end annotation data in this set of annotation data to be fused should be annotation data that has passed the multi-end data temporal matching verification.

[0074] For example, after obtaining multiple single-end labeled data and baseline labeled data, the server further needs to optimize the asynchronous acquisition time of the aforementioned labeled data. Specifically, in this embodiment, multi-end data temporal matching verification and re-matching are used to optimize the re-matching of labeled data with significant asynchronous acquisition times. The main purpose of multi-end data temporal matching verification is to mitigate the time asynchrony problem caused by hardware time synchronization errors by utilizing the temporal continuity of the acquired data, and to ensure that the acquisition times of the same group of multi-end data describing the same scene are basically synchronized, thereby reducing the burden of time alignment for labeling of the same group of multi-end data. More specifically, in this embodiment, to mitigate the time error caused by synchronization error based on the temporal continuity of the acquired data, the server can construct a sampling time range based on a preset data sampling period and the baseline timestamp corresponding to the baseline labeled data determined in the aforementioned steps. The time corresponding to other labeled data is then filtered using the formed sampling time range.

[0075] In this embodiment, because a multi-pole data acquisition system involves multiple steps such as data acquisition, data transmission, and data storage when collecting a set of data describing the same scene, asynchronous data acquisition is unavoidable in engineering practice, even with hardware time synchronization. This is specifically reflected in the timestamp differences within the same data set. Based on this situation, the matching and filtering process in this embodiment refers to a time-series matching requirement where the timestamp difference within the same data set is less than a specified data acquisition period. For example, using a specified master pole timestamp as a benchmark, if the difference between the timestamps of the remaining poles in the same group and this benchmark timestamp is less than 100 milliseconds (ms), then the data set satisfies the time-series matching requirement; otherwise, the time-series matching requirement for multi-terminal data is not met.

[0076] Step 206: Based on the reference timestamp, perform time alignment on the labeled data in the dataset to be fused to form time-aligned labeled data.

[0077] Among them, time alignment of labeled data is a further time alignment at the labeling level for the slight time asynchrony of the same group of data that meets the time sequence matching. Its main purpose is to use the time sequence of data labeling to deal with the problem that the labeling time is not completely matched due to slight asynchrony. By post-labeling processing, it is ensured that the labeling time of the same group of data is the same, thereby reducing the burden of spatial fusion of labeling boxes of multiple data in the same group.

[0078] For example, taking the implementation scenario of 3D target fusion annotation in a traffic scene as an example, since most of the targets to be perceived in a traffic scene, such as pedestrians and vehicles, are in motion, even slight temporal asynchrony will cause the same object to shift in the world coordinate system, and the manual annotations based on it will also shift accordingly, resulting in an inherent spatial misalignment of the annotations. Although the annotation time alignment of matching data essentially deals with the spatial displacement of the annotations, it is important to clarify that the root cause of this displacement is the asynchronous time of the annotations. Therefore, the resulting spatial position deviation between multiple annotations is holistic and can be addressed through overall temporal alignment. Specifically, in this embodiment, the timestamp of the main annotation of the data set to be fused, i.e., the data in the same group, is used as the benchmark, and the timestamps of the other annotations are aligned to it. The spatial unification of annotations under different timestamps is achieved by linear interpolation of the annotation spatial position.

[0079] Step 208: Construct multi-view annotation boxes based on the annotation box information corresponding to the time-aligned annotation data.

[0080] Here, a bounding box refers to a method of selecting and labeling observed objects in an image using rectangular boxes; furthermore, the bounding box information describes the relevant attribute information of the bounding box. The multi-view bounding box in this embodiment is a fusion annotation method for multi-view 3D targets.

[0081] For example, in this embodiment, spatial fusion is performed on spatially misaligned bounding boxes of time-aligned data within the same group. If each single-end annotation is accurate and time-aligned at the annotation level, the multi-end data bounding boxes should be substantially consistent in spatial position. However, due to various factors, including but not limited to differences in perspective between single-end annotations and spatial uncertainties introduced by linear interpolation for time alignment, the same object may still exhibit spatial misalignment under different annotations at multiple ends. This manifests as the bounding boxes of the same object at the same time in the same coordinate system not being completely overlapping. Unlike the spatial position deviation caused by asynchronous annotation time, which is global, the spatial position deviation of the bounding boxes is individual, meaning that the deviation is different for each object. Therefore, spatial fusion of bounding boxes needs to be achieved at the individual level. To achieve this goal, in this embodiment, bounding box spatial fusion includes three main steps: bounding box spatial coordinate system one, multi-end bounding box spatial matching and gap filling, and multi-end bounding box spatial fusion.

[0082] Step 210: Generate multi-view 3D target fusion annotation results based on the multi-view annotation boxes.

[0083] For example, in the embodiment, after fusing multiple bounding box information and constructing a multi-view bounding box, the fusion annotation of multi-view single-end 3D targets can be automatically performed, so that multi-view collaborative annotation can be automatically generated using only the single-end 3D target annotation results.

[0084] The multi-view 3D target fusion annotation method in this embodiment first performs a matching verification on multiple collected single-end annotation data. The verification method involves matching and filtering the collected single-end annotation data against a determined baseline annotation data, leveraging the temporal continuity of the collected data to mitigate the time asynchrony problem caused by hardware time synchronization errors. After obtaining the set of annotation data to be fused through matching and filtering, the method further aligns the set of annotation data to be fused according to the baseline timestamp, forming time-aligned annotation data. Post-annotation processing ensures that the annotation times of the data in the set are the same, thereby reducing the spatial fusion burden of multi-end data annotation boxes. Furthermore, the method constructs multi-view annotation boxes based on the annotation box information corresponding to the time-aligned annotation data, and then generates multi-view 3D target fusion annotation results based on the multi-view annotation boxes. This method can effectively address the unavoidable spatiotemporal asynchrony problem in multi-end data acquisition and annotation engineering by achieving spatiotemporal synchronization of multi-end annotations. It achieves the fusion of multiple single-view 3D target annotations based on multiple single-end annotation data, and fully considers the confidence difference caused by the observation distance of different views and the additional uncertainty introduced in the preceding spatiotemporal synchronization process. This not only improves the accuracy of the fused annotation information, but also improves the reliability of the method when it is applied to multi-scene annotation tasks.

[0085] In one embodiment, to form a set of labeled data to be fused, the temporal matching verification and re-matching in the method mainly consists of two parts: temporal matching judgment and temporal re-matching. For example... Figure 3 As shown, in the method, based on the sampling time range, the single-end timestamps corresponding to multiple single-end labeled data are matched and filtered to obtain the set of labeled data to be fused, including:

[0086] Step 212: If the timestamp difference between the single-end timestamp and the reference timestamp is less than the preset data sampling period, then the multiple single-end labeled data are combined to form a set of labeled data to be fused. The single-end timestamp is the timestamp corresponding to the multiple single-end labeled data, and the reference timestamp is the timestamp corresponding to the reference labeled data.

[0087] For example, in an embodiment, using the specified master pole timestamp as a reference, if the difference between the timestamps of the data of the other poles in the same group and the reference timestamp is less than the data acquisition period, then the data in that group satisfies the time sequence matching; further, in an embodiment, single-end labeled data that satisfy the time sequence matching can be integrated to form a set of labeled data to be fused.

[0088] Step 214: If the timestamp difference between the single-end timestamps corresponding to multiple single-end labeled data and the reference timestamps corresponding to the reference labeled data is not less than the preset data sampling period, then the labeled data at the current moment is determined.

[0089] For example, in this embodiment, when single-ended labeled data fails to meet temporal matching requirements, a re-matching step is required. First, in this embodiment, it is necessary to identify single-ended labeled data that does not meet temporal matching requirements; for example, rod I. a Data collection timestamp t a With main rod I ego Timestamp T ego The difference is greater than the data collection period, where the timestamp t a This refers to the timestamp of the current moment, timestamp T. ego This is the base timestamp.

[0090] Step 216: Generate candidate annotation data for the current time step. The candidate annotation data includes the annotation data from the previous time step and the annotation data from the next time step.

[0091] For example, in this embodiment, the timing of data acquisition is utilized to obtain data from rod I. a At time t a Preorder t a-1 and subsequent t a+1Data that matches the timing of the main data is selected during the acquisition period and replaced with the original data to achieve time-series rematching of multi-terminal data. Since the matching of time-series data is determined based on the specified data acquisition cycle, if the data at a certain moment does not match the timing of the main data, then at least one of its preceding and following data must match the timing of the main data.

[0092] Step 218: Based on the baseline timestamp corresponding to the baseline annotation data and the preset data sampling period, match and filter the candidate timestamps corresponding to the candidate annotation data to obtain the set of annotation data to be fused.

[0093] For example, the embodiment starts from rod I a At time t a Preorder t a-1 and subsequent t a+1 Data that matches the time sequence of the master data during data acquisition is selected and used to replace the original data as candidate annotation data. The re-matching process stops when the candidate timestamps corresponding to the candidate annotation data match the time sequence. This candidate annotation data replaces the corresponding annotation data that does not match the time sequence, thus constructing a set of annotation data to be fused. In this embodiment, after the data time sequence matching verification and re-matching steps, data in the same group describing the same scene can guarantee basic time sequence matching at the data level, laying the foundation for time alignment of subsequent data annotation.

[0094] In one embodiment, the method aligns the single-ended labeled data in the dataset to be fused with time based on a reference timestamp to form time-aligned labeled data. This may include the following steps:

[0095] Step 1: Based on the time sequence information of the single-end labeled data in the labeled data set to be merged, extract the first labeled data and the second labeled data from the labeled data set to be merged. The timestamps of the first labeled data and the second labeled data are adjacent.

[0096] Step 2: Perform linear interpolation based on the first annotation data, the second annotation data, and the reference timestamp to obtain the time-aligned annotation data corresponding to the reference timestamp.

[0097] The first and second labeled data refer to a set of labeled data at adjacent times. In this embodiment, the time sequence information refers to the sequence information formed between the labeled data based on their collection time or timestamp order.

[0098] For example, in this embodiment, the timestamp of the main anchor annotation in the same group of data in the dataset to be fused is used as a reference, and the timestamps of the other end annotations are aligned to it, achieving spatial uniformity of annotations under different timestamps through linear interpolation of annotation spatial positions. Specifically, if the main anchor I... e At time t e Data was collected, and a rod I to be fused and labeled is still in use. m At time t m The corresponding data was collected; considering the complex engineering environment in this embodiment, without loss of generality, the data collection time t is... e <t m Furthermore, corresponding annotations L were generated through manual annotation. e and L m , where L={l 1 ,…,l n ,…,l N}, where N is the number of annotations for a single object. The goal of annotation time alignment is to obtain rod I. m At time t e The label L below me To achieve this goal, we first index I from the labeled data. m At time t m-1 The label L m-1 , where t m-1 It is rod I m In t m Theoretically, there exists a time t from the previous data collection time. m-1 <t e <t m Then use rod I m The label L of itself at the two time points before and after m and L m-1 Linear interpolation estimation at t e Time markers For the nth element have:

[0099]

[0100] Thus, the same scene at time t is obtained. e The main rod below is marked with L. e and aligned rod I m Time alignment mark This laid the foundation for the spatial fusion of data annotation frames from multiple terminals.

[0101] In one embodiment, the process of constructing multi-view annotation boxes based on the annotation box information corresponding to the time-aligned annotation data may include the following steps:

[0102] Step 1: Systematize the coordinate system of the annotation box information corresponding to the time-aligned annotation data.

[0103] For example, the process of spatial fusion of bounding boxes in matching data first requires unifying the spatial coordinate system of the bounding boxes. Specifically, each time-aligned single-end label exists in its own coordinate system. However, in order to perform spatial fusion of multi-end labels, the spatial coordinate systems of each single-end label need to be unified, and the spatial fusion of the bounding boxes needs to be performed in the same coordinate system. For example, in a roadside holographic perception task, a virtual world coordinate system is used as the unified spatial coordinate system. Through calibration data obtained in the hardware synchronization scheme, each single-end label is transformed to this coordinate system, forming a unified coordinate system.

[0104] Step 2: Based on the pre-defined annotation dictionary of the observed object, match and classify the annotation box information after coordinate system one. The annotation dictionary is formed by combining several descriptive fields of a single observed object.

[0105] For example, using spatial coordinate system one, it can be assumed that single-ended bounding boxes in the same coordinate system can be matched. Typically, there are multiple targets in a traffic scenario, making bounding box matching a typical problem of matching between sets. In this embodiment, the Hungarian matching algorithm can be used. The Hungarian algorithm is based on the sufficiency proof of Hall's theorem and is the most common algorithm for bipartite graph matching problems. Its core lies in finding augmenting paths; it is an algorithm that uses augmenting paths to find the maximum matching in a bipartite graph. For spatial matching problems of bounding boxes, spatial Euclidean distance can be used as the basis for matching to achieve spatial matching between two sets of bounding boxes.

[0106] More specifically, a list data structure is maintained based on the main pole annotation content, where each element is a target annotation, and individual target annotations are stored in dictionary form. For multi-end annotations, an end annotation is introduced each time, excluding the main pole annotations. The Hungarian matching algorithm is used to match its annotation box with the main pole. For successfully matched annotation boxes, they are recorded in the matching annotation field of the corresponding annotation dictionary. For unmatched annotation boxes, their physical meaning is that the box is a newly introduced road end annotation, and the annotation box information is directly added to the list in dictionary form for subsequent matching. By traversing the annotation boxes of each end, matching and filling in gaps between multi-end annotations can be achieved.

[0107] Step 3: Based on the matching and classification results, combine the bounding box information in a single category to form a multi-view bounding box.

[0108] For example, after spatial matching and gap filling of multi-device annotation boxes, a list of annotation boxes to be merged is formed for annotation data in the same group or the same category. Each element is a dictionary, representing the annotation of the same object on multiple / single devices in this scenario. If the matching annotation field in a single annotation dictionary is empty, it means that the object is only annotated on one device and does not need to be merged; if the field contains at least one matching annotation dictionary, it means that the object is annotated on multiple devices and annotation fusion is required.

[0109] In one embodiment, the spatial fusion process for multi-view bounding boxes may include attribute fusion and numerical fusion. Furthermore, the process of combining bounding box information from a single category to form multi-view bounding boxes based on the matching and classification results may include the following steps:

[0110] Step 1: Obtain the attribute information and attribute values ​​corresponding to the annotation box information in a single category.

[0111] Step 2: Merge the attribute information corresponding to the annotation box information in a single category to obtain merged attribute information.

[0112] Step 3: Merge the attribute values ​​corresponding to the annotation box information in a single category to obtain the merged attribute value.

[0113] Step four: Integrate the fusion attribute information and fusion attribute values ​​to obtain multi-view annotation boxes.

[0114] For example, in this embodiment, attribute fusion refers to the fusion of abstract attributes describing the target, such as category. In this embodiment, this fusion typically does not result in discrepancies under accurate manual annotation; therefore, only one attribute needs to be retained as the fused annotation attribute. If discrepancies arise, a multi-end voting method can be used to address them. In this embodiment, numerical fusion refers to the fusion of numerical attributes describing the target, such as spatial location and three-dimensional dimensions. This fusion may be affected by different observation perspectives during manual annotation, causing differences. It is crucial for the spatial fusion of annotation boxes. The key issue lies in how to balance the primary and secondary relationships among multiple annotations during fusion, that is, how to define the confidence level of each single-end annotation in the fusion definition.

[0115] In the technical solution of this application, the confidence level of annotation is considered from two aspects. First, a basic assumption is that the accuracy of manual annotation of objects in space decreases with increasing observation distance. This leads to a heuristic method for evaluating annotation confidence, which estimates the confidence level of the annotation based on the distance between the annotated object and the corresponding rod in the world coordinate system. The greater the rod-object distance, the lower the confidence level; that is, the annotation confidence level is inversely proportional to the rod-object distance. Second, another assumption proposed in the solution is that the spatial uncertainty introduced by the annotation time alignment process reduces the annotation confidence level. Therefore, it is necessary to further reduce the annotation confidence level of annotations that have undergone the time alignment step. Based on the aforementioned assumptions, in one embodiment, the method fuses the attribute values ​​corresponding to the annotation box information in a single category to obtain fused attribute values, which may include the following steps:

[0116] Step 1: Obtain the distance information corresponding to the labeled information box in a single category. The distance information is used to characterize the distance between the image acquisition device and the observed object.

[0117] Step 2: Based on the distance information, generate the confidence score corresponding to the labeled information box in a single category.

[0118] Step 3: Based on the confidence level and the preset time-synchronized confidence decay parameter, the attribute values ​​corresponding to the bounding box information in a single category are fused to obtain the fused attribute value. The time-synchronized confidence decay parameter is used to characterize the impact of linear interpolation on the confidence level of the labeled data.

[0119] For example, in the embodiment, for the same scene that has been time-synchronized at time t e The main rod below is marked with L. e and aligned rod I m Time synchronization annotation Without loss of generality, we will use one object k as an example to describe the spatial fusion of the annotation frame. Specifically, the observed object k is within the main rod annotation L. e Individuals in the data are labeled as L e (k), similarly, its time-aligned rod I m Label Individuals in the data are labeled as Firstly, based on individual annotation The following calculation formula is given:

[0120]

[0121] Calculating each individual annotation that needs to be merged in the annotation process yields the result for the same scene at time t. e Spatial fusion annotation below

[0122] Furthermore, considering the additional uncertainty introduced by interpolation during the time alignment process, the label confidence can be further adjusted during the fusion process. That is, for the time-aligned label L... em (k), with a confidence level of Where ω∈[0,1] is the newly introduced time synchronization confidence decay parameter, reflecting the impact of time linear interpolation on the labeled confidence, and thus:

[0123]

[0124] The specific value of the time synchronization confidence decay parameter ω can be set and adjusted according to the actual data collected. The example implements the fusion of multiple single-view artificial 3D target annotations based on viewpoint confidence, fully considering the confidence differences caused by different viewpoint observation distances and the additional uncertainties introduced by the preceding spatiotemporal synchronization steps, thus improving the accuracy of the fused annotations.

[0125] Refer to the instruction manual. Figure 4 Taking the multi-view 3D target fusion annotation in a road traffic scene as an example, the multi-view 3D target fusion annotation method in this application is described in detail below:

[0126] Step 1: Multi-terminal data time sequence matching verification and re-matching.

[0127] Multi-terminal data timing matching verification and re-matching mainly consists of two parts: timing matching judgment and timing re-matching. Because a multi-pole data acquisition system involves multiple steps such as data acquisition, data transmission, and data storage when collecting a set of data describing the same scene, even with hardware time synchronization, asynchronous data acquisition is unavoidable in engineering practice. This is specifically reflected in the timestamp differences within the same data set. Based on this situation, the timing matching of multi-terminal data defined in this embodiment refers to the timestamp difference within the same data set being less than a specified data acquisition period. For example, using a specified master pole timestamp as a benchmark, if the difference between the timestamps of the remaining poles in the same group and this benchmark timestamp is less than the data acquisition period, such as 100ms, then the data set satisfies timing matching; otherwise, it does not satisfy the timing matching requirement for multi-terminal data.

[0128] Data that satisfies temporal matching requires no additional processing, while data that does not satisfy temporal matching needs a re-matching step. Specifically, in the same group of data, if rod I... a Data collection timestamp t a With main rod I ego Timestamp T ego If the difference is greater than the data acquisition cycle, then the timing of data acquisition is utilized to start from rod I. a At time t a Preorder t a-1and subsequent t a+1 Data that matches the timing of the main data is selected during the acquisition period and replaced with the original data to achieve time-series rematching of multi-terminal data. Since the matching of time-series data is determined based on a data acquisition cycle with specific rules, if the data at a certain moment does not match the timing of the main data, then there must be one of its preceding and following data that matches the timing of the main data.

[0129] Step 2: Match the data annotation time alignment.

[0130] Since most perceptible targets in traffic scenarios, such as pedestrians and vehicles, are in motion, even slight temporal asynchrony can cause the same object to shift in the world coordinate system. Consequently, the manually labeled objects will also shift, resulting in an inherent spatial misalignment of the overall labels. While label time alignment in matching data essentially deals with spatial displacement of labels, it's crucial to understand that the root cause of this displacement is the asynchrony of label time. Therefore, the resulting spatial positional deviation between multiple labels is holistic and can be addressed through overall temporal alignment. Specifically, using the main label timestamp of the same data set as a benchmark, the timestamps of other labels are aligned to it, achieving spatial uniformity of labels at different timestamps through linear interpolation of label spatial positions.

[0131] In the embodiment, if the main rod I e At time t e Data was collected, and a rod I to be fused and labeled is still in use. m At time t m The relevant data was collected. Considering the complex engineering environment in this embodiment, without loss of generality, the data collection time is assumed to be t. e <t m Furthermore, corresponding annotations L were generated through manual annotation. e and L m , where L={l 1 ,…,l n ,…,l N}, where N is the number of annotations for a single object. The goal of annotation time alignment is to obtain rod I. m At time t e The label L below me To achieve this goal, we first index I from the labeled data. m At time t m-1 The label L m-1 , where t m-1 It is rod I m In t m Theoretically, there exists a time t from the previous data collection time. m-1 <te <t m Then use rod I m The label L of itself at the two time points before and after m and L m-1 Linear interpolation estimation at t e Time markers For the nth element have:

[0132]

[0133] Thus, the same scene at time t is obtained. e The main rod below is marked with L. e and aligned rod I m Time alignment mark This laid the foundation for the spatial fusion of data annotation frames from multiple terminals.

[0134] Step 3: Spatial fusion of matching data bounding boxes. In this embodiment, spatial fusion of matching data bounding boxes includes three main steps:

[0135] (a) Spatial coordinate system one for annotation boxes. Each time-aligned single-end annotation exists in its own coordinate system. To perform spatial fusion of multi-end annotations, it is first necessary to unify the spatial coordinate systems of each single-end annotation and perform spatial fusion of the annotation boxes in the same coordinate system. In the roadside holographic perception task, a virtual world coordinate system is used as the unified spatial coordinate system. Through the calibration data obtained in the hardware synchronization scheme, each single-end annotation is transformed to this coordinate system to form coordinate system one.

[0136] (b) Spatial Matching and Filling of Multi-End Label Boxes. A list data structure is maintained based on the main pole label content, where each element is a label for a target, and the labels for a single target are stored in dictionary form. For multi-end labels, each time an end label is introduced from outside the main pole labels, the label box matching between it and the main pole is achieved through the Hungarian matching algorithm. For successfully matched label boxes, they are recorded in the matching label field of the corresponding label dictionary. For unmatched label boxes, their physical meaning is that the box is a newly introduced road end label, and the label box information is directly added to the list in dictionary form for subsequent matching. By traversing the label boxes of each end, matching and filling of gaps between multi-end labels can be achieved.

[0137] (c) Multi-end bounding box spatial fusion. Annotation fusion mainly includes two types: attribute fusion and numerical fusion. Attribute fusion refers to the fusion of abstract attributes describing the target, such as category. Under accurate manual annotation, this fusion usually does not result in discrepancies; therefore, only one attribute needs to be retained as the fused annotation attribute. If discrepancies arise, a multi-end voting method can be used to address them. Numerical fusion refers to the fusion of numerical attributes describing the target, such as spatial location and three-dimensional dimensions. This fusion may be affected by different observation perspectives during manual annotation, causing differences. It is the key to bounding box spatial fusion. The key issue is how to balance the primary and secondary relationships among the multi-end annotations during fusion, that is, how to define the confidence level of each single-end annotation in the fusion definition.

[0138] For the same scene that has already been time-synchronized at time t e The main rod below is marked with L. e and aligned rod I m Time synchronization annotation Without loss of generality, we will use an object k as an example to describe the spatial fusion of the annotation box. Specifically, object k is within the main rod annotation L. e Individuals in the data are labeled as L e (k), similarly, its time-aligned rod I m Label Individuals in the data are labeled as First, based on individual annotations, calculate the spatial distance between the object and its corresponding rod, denoted as d. e (k) and Then, based on the pole-object distance, the spatial fusion of individual bounding boxes can be achieved to form their fused annotations. For the nth element have:

[0139]

[0140] Perform the above calculations on each individual annotation that needs to be merged in the annotation, and you can obtain the same scene at time t. e Spatial fusion annotation below

[0141] Furthermore, considering the additional uncertainty introduced by interpolation during the time alignment process, the label confidence can be further adjusted during the fusion process. That is, for the time-aligned label L... em (k), with a confidence level of Where ω∈[0,1] is the newly introduced time synchronization confidence decay parameter, reflecting the impact of time linear interpolation on the labeled confidence, and thus:

[0142]

[0143] The specific value of the confidence decay parameter ω for time synchronization can be set and adjusted according to the actual data collected.

[0144] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0145] Based on the same inventive concept, this application also provides a multi-view 3D target fusion annotation apparatus for implementing the multi-view 3D target fusion annotation method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method. Therefore, the specific limitations in one or more embodiments of the multi-view 3D target fusion annotation apparatus provided below can be found in the limitations of the multi-view 3D target fusion annotation method described above, and will not be repeated here.

[0146] In one embodiment, such as Figure 5 As shown, a multi-view 3D target fusion annotation device 500 is provided, including: a data acquisition module 501, a matching and filtering module 502, a time alignment module 503, a spatial alignment module 504, and an annotation fusion module 505, wherein:

[0147] The data acquisition module 501 is used to acquire multiple single-ended annotation data and the baseline annotation data from the multiple single-ended annotation data;

[0148] The matching and filtering module 502 is used to match and filter the single-end timestamps corresponding to multiple single-end annotation data according to the benchmark timestamps corresponding to the benchmark annotation data to obtain the set of annotation data to be merged.

[0149] The time alignment module 503 is used to perform time alignment on the labeled data in the dataset to be fused based on the reference timestamp, so as to form time-aligned labeled data.

[0150] The spatial alignment module 504 is used to construct multi-view annotation boxes based on the annotation box information corresponding to the time-aligned annotation data.

[0151] The annotation fusion module 505 is used to generate multi-view 3D target fusion annotation results based on multi-view annotation boxes.

[0152] In one embodiment, the matching and filtering module 502 is further configured to: combine multiple single-end labeled data to form a set of labeled data to be fused when the timestamp difference between the single-end timestamp and the reference timestamp is less than a preset data sampling period; and determine the labeled data at the current moment when the timestamp difference between the single-end timestamps corresponding to the multiple single-end labeled data and the reference timestamp corresponding to the reference labeled data is not less than a preset data sampling period; generate candidate labeled data for the labeled data at the current moment, including labeled data from the previous moment and labeled data from the next moment; and match and filter the candidate timestamps corresponding to the candidate labeled data according to the reference timestamps corresponding to the reference labeled data and the preset data sampling period to obtain the set of labeled data to be fused.

[0153] In one embodiment, the time alignment module 503 is further configured to extract first annotation data and second annotation data from the annotation data set to be fused based on the time sequence information of the single-end annotation data in the annotation data set to be fused, wherein the timestamps of the first annotation data and the second annotation data are adjacent; and to perform linear interpolation operation based on the first annotation data, the second annotation data, and the reference timestamp to obtain the time-aligned annotation data corresponding to the reference timestamp.

[0154] In one embodiment, the spatial alignment module 504 is further configured to perform coordinate system one on the annotation box information corresponding to the time alignment annotation data; match and classify the annotation box information after coordinate system one according to the annotation dictionary preset by the observed object, wherein the annotation dictionary is formed by combining several descriptive fields of a single observed object; and combine the annotation box information in a single category according to the matching and classification results to form a multi-view annotation box.

[0155] In one embodiment, the spatial alignment module 504 is further configured to obtain the attribute information and attribute values ​​corresponding to the annotation box information in a single category; fuse the attribute information corresponding to the annotation box information in a single category to obtain fused attribute information; fuse the attribute values ​​corresponding to the annotation box information in a single category to obtain fused attribute values; and integrate the fused attribute information and fused attribute values ​​to obtain multi-view annotation boxes.

[0156] In one embodiment, the spatial alignment module 504 is further configured to obtain distance information corresponding to the labeled information boxes in a single category, the distance information being used to characterize the distance between the image acquisition device and the observed object; generate confidence scores corresponding to the labeled information boxes in a single category based on the distance information; and fuse the attribute values ​​corresponding to the labeled information boxes in a single category based on the confidence scores and a preset time-synchronized confidence attenuation parameter to obtain fused attribute values, the time-synchronized confidence attenuation parameter being used to characterize the impact of linear interpolation on the confidence scores of the labeled data.

[0157] Each module in the aforementioned multi-view 3D target fusion annotation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0158] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores single-ended annotation data or multi-ended annotation data formed after fusion. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a multi-view 3D target fusion annotation method.

[0159] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0160] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0161] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0162] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0163] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0164] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multi-view 3D target fusion annotation method, characterized in that, The method includes: Acquire multiple single-ended annotation data, and the baseline annotation data among the multiple single-ended annotation data; Based on the benchmark timestamps corresponding to the benchmark annotation data, the single-end timestamps corresponding to the multiple single-end annotation data are matched and filtered to obtain a set of annotation data to be fused. Based on the reference timestamp, the labeled data in the set of labeled data to be fused are time-aligned to form time-aligned labeled data; Based on the annotation box information corresponding to the time-aligned annotation data, a multi-view annotation box is constructed; Based on the multi-view annotation boxes, generate multi-view 3D target fusion annotation results; The step of matching and filtering the single-end timestamps corresponding to the multiple single-end annotation data according to the benchmark timestamps corresponding to the benchmark annotation data to obtain the set of annotation data to be fused includes: If the timestamp difference between the single-end timestamp and the reference timestamp is less than the preset data sampling period, then multiple single-end labeled data will be combined to form a set of labeled data to be fused. The single-end timestamp is the timestamp corresponding to multiple single-end labeled data, and the reference timestamp is the timestamp corresponding to the reference labeled data. If the timestamp difference between the single-end timestamps corresponding to multiple single-end labeled data and the reference timestamp corresponding to the reference labeled data is not less than the preset data sampling period, then the labeled data at the current moment is determined. Generate candidate annotation data for the current time step, wherein the candidate annotation data includes the annotation data from the previous time step and the annotation data from the next time step; Based on the baseline timestamp corresponding to the baseline annotation data and the preset data sampling period, the candidate timestamps corresponding to the candidate annotation data are matched and filtered to obtain a set of annotation data to be fused.

2. The method according to claim 1, characterized in that, The step of performing time alignment on the labeled data in the set of labeled data to be fused, based on the reference timestamp, to form time-aligned labeled data includes: Based on the temporal information of the single-end labeled data in the labeled data set to be fused, the first labeled data and the second labeled data are extracted from the labeled data set to be fused, wherein the timestamps of the first labeled data and the second labeled data are adjacent; Linear interpolation is performed based on the first annotation data, the second annotation data, and the reference timestamp to obtain the time-aligned annotation data corresponding to the reference timestamp.

3. The method according to claim 1, characterized in that, The step of constructing multi-view annotation boxes based on the annotation box information corresponding to the time-aligned annotation data includes: The coordinate system is one for the annotation box information corresponding to the time alignment annotation data; According to the pre-defined annotation dictionary of the observed object, the annotation box information after coordinate system one is matched and classified. The annotation dictionary is formed by combining several descriptive fields of a single observed object. Based on the matching and classification results, the bounding box information in a single category is combined to form a multi-view bounding box.

4. The method according to claim 3, characterized in that, The step of combining the bounding box information from a single category to form a multi-view bounding box based on the matching and classification results includes: Retrieve the attribute information and attribute values ​​corresponding to the annotation box information in a single category; The attribute information corresponding to the annotation box information in a single category is merged to obtain the merged attribute information; The attribute values ​​corresponding to the annotation box information in a single category are merged to obtain the merged attribute value; The fusion attribute information and the fusion attribute values ​​are integrated to obtain a multi-view annotation box.

5. The method according to claim 4, characterized in that, The process of fusing the attribute values ​​corresponding to the annotation box information in a single category to obtain the fused attribute values ​​includes: Obtain the distance information corresponding to the labeled information box in a single category, the distance information being used to characterize the distance between the image acquisition device and the observed object; Based on the distance information, generate the confidence score corresponding to the labeled information box in a single category; Based on the confidence level and the preset time-synchronized confidence decay parameter, the attribute values ​​corresponding to the bounding box information in a single category are fused to obtain the fused attribute value. The time-synchronized confidence decay parameter is used to characterize the impact of linear interpolation on the confidence level of the labeled data.

6. A multi-view 3D target fusion annotation device, characterized in that, The device includes: The data acquisition module is used to acquire multiple single-ended annotation data and the baseline annotation data among the multiple single-ended annotation data; The matching and filtering module is used to match and filter the single-end timestamps corresponding to the multiple single-end annotation data according to the benchmark timestamps corresponding to the benchmark annotation data, so as to obtain a set of annotation data to be fused. The time alignment module is used to perform time alignment on the labeled data in the set of labeled data to be fused according to the reference timestamp, so as to form time-aligned labeled data; The spatial alignment module is used to construct multi-view annotation boxes based on the annotation box information corresponding to the time alignment annotation data; The annotation fusion module is used to generate multi-view 3D target fusion annotation results based on the multi-view annotation boxes; The step of matching and filtering the single-end timestamps corresponding to the multiple single-end annotation data according to the benchmark timestamps corresponding to the benchmark annotation data to obtain the set of annotation data to be fused includes: If the timestamp difference between the single-end timestamp and the reference timestamp is less than the preset data sampling period, then multiple single-end labeled data will be combined to form a set of labeled data to be fused. The single-end timestamp is the timestamp corresponding to multiple single-end labeled data, and the reference timestamp is the timestamp corresponding to the reference labeled data. If the timestamp difference between the single-end timestamps corresponding to multiple single-end labeled data and the reference timestamp corresponding to the reference labeled data is not less than the preset data sampling period, then the labeled data at the current moment is determined. Generate candidate annotation data for the current time step, wherein the candidate annotation data includes the annotation data from the previous time step and the annotation data from the next time step; Based on the baseline timestamp corresponding to the baseline annotation data and the preset data sampling period, the candidate timestamps corresponding to the candidate annotation data are matched and filtered to obtain a set of annotation data to be fused.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images