Multi-modal data labeling method, device and system based on AI and image fusion

Abstract

The application discloses a multi-modal data labeling method, device and system based on AI and image fusion, and particularly relates to the technical field of data labeling.The difficulty grading mechanism of multi-dimensional parameter quantitative analysis of signal-to-noise ratio, information entropy and contrast ratio and scene adaptive weight distribution is used to realize the objectivization, standardized grading and differential process treatment of the fusion image labeling difficulty, solve the problems of waste of computing resources, low efficiency of simple samples and insufficient accuracy of difficult samples caused by the parameterless analysis and unified processing of all samples of the prior art, and the corresponding processing strategies are matched for different difficulty samples, and the computing and artificial resources are reasonably distributed, so that the labeling accuracy of complex scenes is ensured, and the execution efficiency and resource utilization of the overall labeling process are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of data annotation technology, and more specifically, to a multimodal data annotation method, apparatus, and system based on AI and image fusion. Background Technology

[0002] With the widespread application of multi-source heterogeneous image perception technology in scenarios such as autonomous driving, remote sensing observation, security monitoring, and industrial inspection, multimodal images such as visible light, infrared, and depth maps have become core data sources for environmental perception and target analysis. Different modal images have complementary information in terms of illumination adaptability, texture representation, and spatial structure perception. How to perform unified, standardized, and reusable annotation processing on multimodal images to form high-quality, highly consistent labeled datasets has become a key aspect of the engineering application of multi-source data.

[0003] However, existing multimodal data annotation technologies still have the following shortcomings in practical applications: First, existing annotation methods lack systematic parametric analysis and differential processing mechanisms. They do not perform quantitative analysis of fused images based on parameters such as signal-to-noise ratio, information entropy, and contrast, making it impossible to achieve objective classification of annotation difficulty. Using a uniform processing flow for all annotated samples can easily lead to resource waste and insufficient annotation accuracy. Secondly, cross-modal geometric consistency and feature consistency checks were not performed on the candidate annotations that were inversely mapped to each original modality. At the same time, temporal consistency checks for continuous frame sequences were lacking, making it impossible to effectively determine the reliability of the annotation results.

[0004] To address this, a multimodal data annotation method, device, and system based on AI and image fusion have been developed. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a multimodal data annotation method, apparatus, and system based on AI and image fusion.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A multimodal data annotation method based on AI and image fusion includes the following steps: Generate fused image: Extract multimodal image data, and based on the preset annotation task requirements, use an adaptive multi-scale feature fusion network to generate a fused image optimized for the annotation task; Image fusion analysis and difficulty classification: The signal-to-noise ratio, information entropy difference, and contrast parameters of the fused image are extracted, and the comprehensive difficulty value is calculated by combining the weights corresponding to the scene type. The annotation difficulty level of the fused image is determined based on the comprehensive difficulty value range. Multi-task AI initial annotation and inverse mapping: The pre-trained multi-task AI annotation model is used to complete the initial annotation on the fused image. Then, the annotation results are back-projected to each original modality image through geometric transformation to generate candidate annotations for each modality. Consistency verification analysis: For fused images with a labeling difficulty level of "difficult", the geometric parameters and feature parameters of candidate labels under different modalities are collected, and the single-frame consistency index is calculated; for scenes with continuous frame sequences, temporal consistency verification is triggered and corrected to obtain the final consistency index, and the confidence level of the labeling results is determined based on the final consistency index. Annotation fusion and final output: The high-confidence AI initial annotation results are merged with the low-confidence annotation results after manual review and correction, and the complete multimodal image annotation results are output according to the preset standard format.

[0007] Specifically, the process of obtaining the fused image is as follows: The registered modal images are decomposed into multiple scales, and high-frequency detail features and low-frequency semantic features are extracted at each scale. Based on the preset annotation task type, the fusion weights are dynamically assigned to features of different modalities at each scale; Within the same scale, the corresponding features of different modalities are weighted and summed according to the weights assigned to them at that scale to generate the fused features at the corresponding scale. Multi-scale reconstruction is performed on the fusion features at all scales to generate a fusion image optimized for the current annotation task.

[0008] Specifically, the process of obtaining the overall difficulty value is as follows: The fused image is divided into r sub-blocks, the noise variance and signal variance are calculated, and the signal-to-noise ratio is obtained by formula. Quantize the image to Each gray level has several gray levels; statistics are collected for each gray level. Pixel frequency The total number of pixels is The probability of grayscale appearing is Using the formula Obtain the information entropy value The information entropy difference is obtained by calculating the absolute difference between the information entropy value and a pre-set information entropy threshold. ; Calculate the average pixel brightness of the entire image, and obtain the contrast value by taking the square root of the average of the squared deviations of the pixel brightness from the mean; After normalizing the signal-to-noise ratio, information entropy difference, and contrast value, the formula is used. The overall difficulty value is obtained after weighted calculation. ,in , , These are the corresponding matching weight factors; Two sets of comprehensive difficulty value intervals are preset, and each set of comprehensive difficulty value intervals corresponds to a difficulty level of the fused image. The comprehensive difficulty value is matched with the corresponding comprehensive difficulty value interval to obtain the difficulty level of the fused image. The difficulty levels for image annotation fusion include general and difficult.

[0009] Specifically, the process of obtaining the single-frame consistency index is as follows: Geometric parameters of candidate labels under different modalities are extracted, including intersection-union ratio, centroid distance similarity, and area ratio. After comprehensive processing, geometric estimates are obtained. Feature parameters of candidate labels under different modalities are extracted, including mutual information value, structural similarity value, and cross-correlation value. After comprehensive processing, feature estimates are obtained. After normalizing the geometric and feature estimates, the single-frame consistency index is calculated using a pre-built formula.

[0010] Specifically, the process of obtaining the geometric estimate is as follows: The bounding boxes located on the original modal image after inverse mapping are used as candidate boxes; Obtain the coordinates of the top left and bottom right corners of the two candidate boxes respectively, calculate the pixel area occupied by each box, obtain the area of ​​the overlapping part of the two areas, calculate the total area of ​​the union of the two areas, and calculate the ratio of the overlapping area to the total area of ​​the union to obtain the intersection-union ratio. Calculate the pixel coordinates of the center points of two candidate boxes and calculate the straight-line distance between the two points; obtain the diagonal length of the image, divide the straight-line distance between the center points by the diagonal length of the image to get the distance ratio, and subtract the distance ratio from 1 to get the center point distance similarity; Calculate the area of ​​each candidate box separately, and use the smaller area value as the numerator and the larger area value as the denominator to calculate the area ratio. After normalizing the intersection-union ratio, center point distance, and area ratio, the formula is used. Geometric valuation is obtained by weighting. ,in , , These are the corresponding preset weighting factors.

[0011] Specifically, the process of obtaining feature estimates is as follows: Quantize the image pixels a and b within the two candidate boxes into K gray levels, and calculate the joint histogram of the gray levels of the two image blocks to obtain the joint probability of each pair of gray levels. Statistically calculate their respective marginal probability distributions and According to the definition of mutual information in information theory, the original value of mutual information can be obtained using the formula. The theoretical maximum value corresponding to the preset mutual information original value. Using the formula The mutual information value is calculated; Set two candidate bounding box image blocks and The number of pixels is ; Statistical analysis of the mean brightness of image patches x and y , ; The standard deviation of contrast is obtained using a pre-constructed formula. , Structural covariance ; Using formula The structural similarity values ​​are obtained, where, , These are pre-defined constants; Calculate the average value of pixels within the two candidate bounding box image blocks, and then subtract the average value from each pixel value to obtain a pixel matrix with zero mean. Calculate the sum of the dot products of the two zero-mean matrices, and calculate the square root of the sum of the squares of each matrix. Divide the sum of the dot products by the product of the square roots of the two matrices to obtain the cross-correlation value. The feature estimates are obtained by comprehensively processing the mutual information value, structural similarity value, and cross-correlation value respectively.

[0012] Specifically, the process of obtaining the eventual consistency index from the time-series consistency checksum is as follows: A multi-target tracking algorithm is used to associate the same target in the current frame and the previous and next s frames, and the single-frame consistency index of the target in each frame is extracted to form a sequence C; Calculate the mean of sequence C and variance ,like and If the consistency index of the current frame is used as the final consistency index, then the consistency index of the current frame will be used as the final consistency index. Otherwise, if the timing consistency is insufficient, the exponent of the current frame needs to be reduced, and the final consistency exponent is obtained using a pre-built formula. The final consistency index is compared with a preset threshold. If it is greater than the threshold, it is judged as high confidence and the AI ​​initial annotation result is directly output; if it is less than or equal to the threshold, it is judged as low confidence and submitted for manual review.

[0013] Specifically, the multimodal data annotation device based on AI and image fusion includes: Visible light and infrared cameras: acquire raw modal images based on different imaging principles, and add acquisition timestamps and sensor calibration parameters to each frame of image; Signal generator: Using the timestamp of one mode as a reference, timestamp matching is performed on the data of the other modes; GPU server for multi-task model: performs end-to-end inference on fused images and outputs initial annotation results such as bounding boxes, class labels and segmentation masks; Inverse mapping projection device: Using sensor calibration parameters, the annotation results on the fused image are back-projected onto the coordinate system of each original modality image through geometric transformation to generate candidate annotations.

[0014] Specifically, multimodal data annotation systems based on AI and image fusion include: Image generation module: Extracts multimodal image data, and based on the preset annotation task requirements, uses an adaptive multi-scale feature fusion network to generate a fused image optimized for the annotation task; Image analysis and classification module: Extracts the signal-to-noise ratio, information entropy difference, and contrast parameters of the fused image, calculates the comprehensive difficulty value by combining the weights corresponding to the scene type, and determines the annotation difficulty level of the fused image based on the comprehensive difficulty value range; The annotation and mapping module uses a pre-trained multi-task AI annotation model to complete the initial annotation on the fused image, and then uses geometric transformation to back-project the annotation results to each original modality image to generate candidate annotations for each modality. Consistency verification module: For fused images with a labeling difficulty level of "difficult", it collects the geometric parameters and feature parameters of candidate labels under different modalities and calculates the single-frame consistency index; for scenes with continuous frame sequences, it triggers temporal consistency verification and corrects to obtain the final consistency index, and determines the confidence level of the labeling results based on the final consistency index. The annotation fusion output module merges the high-confidence AI initial annotation results with the low-confidence annotation results after manual review and correction, and outputs the complete multimodal image annotation results according to the preset standard format.

[0015] The technical effects and advantages of this invention are as follows: This invention achieves an objective, standardized, and differentiated process for merging image annotation difficulty through a difficulty grading mechanism that combines multi-dimensional parameter quantification analysis of signal-to-noise ratio, information entropy, and contrast with scene-adaptive weight allocation. It solves the problems of waste of computing resources, low efficiency for simple samples, and insufficient accuracy for difficult samples caused by the lack of parameterized analysis and uniform processing of all samples in existing technologies. It matches corresponding processing strategies for samples of different difficulty levels, rationally allocates computing and manual resources, and significantly improves the execution efficiency and resource utilization of the overall annotation process while ensuring the annotation accuracy of complex scenes. This invention achieves accurate determination of the reliability of annotation results, targeted screening and manual correction of low-confidence regions by cross-modal geometric and feature dual-dimensional consistency verification, continuous frame temporal consistency correction, and standardized fusion of high and low confidence annotations. It solves the defects of existing technologies such as lack of consistency verification, low credibility of annotation results, high cost of full manual review, and uncontrollable annotation quality. By replacing manual sampling with automated verification and full correction with targeted correction, it effectively improves the spatial consistency, temporal stability and overall accuracy of multimodal annotation results, and finally outputs a standardized, high-quality multimodal annotation dataset. Attached Figure Description

[0016] Figure 1 This is a flowchart of the multimodal data annotation method based on AI and image fusion of the present invention.

[0017] Figure 2 This is a schematic diagram of the multimodal data annotation system based on AI and image fusion of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1

[0020] like Figure 1 As shown, the steps of the multimodal data annotation method based on AI and image fusion are as follows: Generate fused images: Extract multimodal image data and, based on pre-defined annotation task requirements (such as object detection and semantic segmentation), use an adaptive multi-scale feature fusion network to generate fused images optimized for annotation tasks; The registered modal images are decomposed into multiple scales, and high-frequency detail features and low-frequency semantic features are extracted at each scale. Based on the preset annotation task type, the fusion weights are dynamically assigned to features of different modalities at each scale; Within the same scale, the corresponding features of different modalities (such as visible light, infrared, and depth) are weighted and summed according to the weights assigned to them at that scale to generate fused features at the corresponding scale. Multi-scale reconstruction of fused features at all scales is performed by a multi-scale feature decoder to generate a fused image optimized for the current annotation task. Image fusion analysis and difficulty classification: Extract fused image data (such as signal-to-noise ratio, information entropy, and contrast), set a set of weight combinations corresponding to different scene types (city / highway / night), identify the scene type corresponding to the fused image to determine the weight combination, and perform comprehensive analysis on the fused image data in combination with the determined weight combination to obtain the comprehensive difficulty value. Based on the pre-set mapping rules, determine the image difficulty level corresponding to the fused image. Specifically: The fused image is divided into r sub-blocks. The local variance and gradient magnitude of each sub-block are calculated. Variance thresholds and gradient magnitude thresholds are set for each sub-block. Sub-blocks with variance and gradient values ​​less than the variance threshold and gradient values ​​less than the gradient threshold are designated as flat blocks. The mean variance of the flat blocks is used as the noise variance. Calculate the overall variance of all pixels in the entire image. The signal variance is obtained by subtracting the noise variance from the total variance. Using the formula Obtain the signal-to-noise ratio value ; To elaborate further, the signal-to-noise ratio (SNR) measures the intensity contrast between the effective signal and random noise in an image. During annotation, noise blurs target edges and obscures details and textures, resulting in unclear boundaries and small targets being obscured. The lower the SNR, the harder it is for annotators to distinguish between "real targets and noise particles," leading to more repeated confirmations during segmentation or bounding box selection, and a decrease in localization accuracy, especially in semantic segmentation tasks that require fine contour delineation. Quantize the image to Each gray level has several gray levels; statistics are collected for each gray level. Pixel frequency The total number of pixels is The probability of grayscale appearing is Using the formula Obtain the information entropy value The information entropy difference is obtained by calculating the absolute difference between the information entropy value and a pre-set information entropy threshold. ; To elaborate further, the information entropy difference quantifies the deviation of the current image grayscale distribution from the ideal labelable state. The larger the information entropy difference, the more extreme the image grayscale distribution is—either too uniform, making it difficult to distinguish the target from the background; or too cluttered, causing the outline of the foreground object to be submerged in dense texture; both will significantly increase the difficulty of annotation. Calculate the average brightness of all pixels in the image. Then, iterate through each pixel and calculate the square of the difference between each pixel and the mean. Average and square root these squared deviations for all pixels to obtain the contrast value. ; To elaborate further, contrast characterizes the overall intensity of the difference between light and dark areas in an image, directly affecting the separability of targets from the background. High contrast results in sharp target outlines and distinct layers, allowing annotators to quickly determine boundaries. Low contrast, however, makes objects appear as a blurry gray area, requiring repeated dragging and adjustment of the window width and level to barely distinguish them, especially for small targets in low visibility conditions where they are almost invisible. Therefore, the lower the contrast, the greater the visual load on the target separation process, leading to a simultaneous increase in the probability of missed and incorrect annotations, and conversely, a greater increase in the overall difficulty. After normalizing the signal-to-noise ratio, information entropy difference, and contrast value, the formula is used. The overall difficulty value is obtained after weighted calculation. ,in , , These are the corresponding matching weight factors; Two sets of comprehensive difficulty value intervals are preset, and each set of comprehensive difficulty value intervals corresponds to a difficulty level of the fused image. The comprehensive difficulty value is matched with the corresponding comprehensive difficulty value interval to obtain the difficulty level of the fused image. Among them, the difficulty levels of image annotation fusion include general and difficult; Multi-task AI initial annotation and inverse mapping: The multi-task AI annotation model is used to complete the initial annotation on the fused image (outputting bounding boxes, categories and segmentation masks), and then the annotation results are back-projected to the original modal image (e.g. visible light image, infrared image or depth map) through geometric transformation to generate candidate annotations for each modality. A multi-task AI annotation model is pre-built, and the generated fused image is input into the pre-built multi-task AI annotation model to output the initial annotation corresponding to the fused image; This model is an end-to-end multi-task model built on a deep learning architecture. The backbone network adopts a general deep feature extraction architecture, and the backend has multiple parallel task branches, each corresponding to a labeling task. The model is pre-trained on a large-scale general multimodal dataset, possessing general semantic feature extraction and labeling capabilities, and has undergone preliminary adaptation using a small amount of scene data. The annotation results on the fused image are mapped back to the image coordinate system of each original modality through geometric transformation, and corresponding candidate annotations are obtained on each original modality image; Consistency verification analysis: Identify fused images with a labeling level of difficulty, and collect the geometric parameters and feature parameters of the corresponding candidate labels for each fused image under different modalities. Perform comprehensive analysis on the geometric parameters and feature parameters respectively to obtain geometric estimates and feature estimates. Multiply the geometric estimates and feature estimates by their respective preset weighting factors and sum them to obtain the single-frame consistency index. For application scenarios with continuous frame sequences, a timing consistency check is triggered. Among them, geometric parameters include intersection-union ratio, center point distance, and area ratio, while feature parameters include mutual information, structural similarity, and cross-correlation; Specifically: The bounding boxes located on the original modal image after inverse mapping are used as candidate boxes; Obtain the coordinates of the top-left and bottom-right corners of the two candidate boxes, calculate the pixel area occupied by each box, obtain the area of ​​the overlapping portion of the two areas, calculate the total area of ​​the union of the two areas, and calculate the ratio of the overlapping area to the total area of ​​the union to obtain the intersection-union ratio (IU). ; Calculate the pixel coordinates of the center points of two candidate bounding boxes, and then calculate the straight-line distance between the two points using the Euclidean distance formula. Obtain the diagonal length of the image, divide the straight-line distance between the center points by the diagonal length of the image to get the distance ratio, and subtract the distance ratio from 1 to obtain the center point distance similarity. ; Calculate the area of ​​each candidate box separately, and use the smaller area value as the numerator and the larger area value as the denominator to calculate the area ratio. ; After normalizing the intersection-union ratio, center point distance, and area ratio, the formula is used. Geometric valuation is obtained by weighting. ,in , , These are the corresponding preset weighting factors; Quantize the image pixels a and b within the two candidate boxes into K gray levels, and calculate the joint histogram of the gray levels of the two image blocks to obtain the joint probability of each pair of gray levels. (The joint probability represents the gray level in a as) And the gray level in b is (The proportion of pixel pairs to the total number of pixel pairs), and their respective edge probability distributions are calculated. and According to the definition of mutual information in information theory, using the formula Obtain the original value of mutual information The theoretical maximum value corresponding to the preset mutual information original value. For a gray level K, the maximum possible mutual information is Using the formula Calculate the mutual information value ; Set two candidate bounding box image blocks and The number of pixels is ; Then use the formula Obtain the average brightness values ​​of image patch x and y. , ; Using formula Obtain the standard deviation of contrast , ; Using formula Obtain structural covariance ; Using formula Obtain structural similarity values ,in, , These are pre-defined constants; Calculate the average value of pixels within the two candidate bounding box image blocks, and then subtract the average value from each pixel value to obtain a pixel matrix with zero mean. Calculate the sum of the dot products of the two zero-mean matrices, and then calculate the square root of the sum of the squares of each matrix. Divide the sum of the dot products by the product of the square roots of the two matrices to obtain the cross-correlation value. ; After normalizing the mutual information value, structural similarity value, and cross-correlation value respectively, the formula is used. The feature estimate is obtained by weighted calculation. ,in , , These are the corresponding preset weighting factors; The ideal values ​​corresponding to the preset geometric estimate and the feature estimate are respectively labeled as , ; After normalizing the geometric estimate and eigenvalue, we input them into the formula. The consistency index is obtained after weighted calculation. ,in , These are the corresponding preset weighting factors; The timing consistency check is specifically as follows: Multi-target tracking algorithms (such as SORT and DeepSORT) are used to associate the same target in the current frame and the preceding and following s frames (e.g., s=2, i.e., the preceding 2 frames and the following 2 frames, for a total of 2s+1 frames), and the single-frame consistency index of the target in each frame is extracted to form a sequence C. Using formula , The mean of each sequence was obtained. and variance ; Set time series mean threshold Time series variance threshold ,like and If the consistency index of the current frame is used as the final consistency index, then the consistency index of the current frame will be used as the final consistency index. Otherwise, if the timing consistency is insufficient, the exponent of the current frame needs to be reduced, and the correction formula is as follows: The final consistency index is obtained. ;in, This is a time-series correction factor, with a value range of (0,1]. The consistency index of the current frame; the formula for calculating the timing correction factor is: ; The final consistency index is compared with the corresponding preset threshold. When the final consistency index is greater than the threshold, it is judged as high confidence and the AI ​​initial annotation result is directly output. If the confidence level is less than or equal to the threshold, it is determined to be low confidence and submitted for manual review. Annotation fusion and final output: The target annotations judged as high confidence in the consistency check (directly from the AI ​​initial annotation) are merged with the low confidence target annotations after manual review and correction. The complete multimodal image data annotation results are output according to the preset standard format (such as COCO, KITTI or custom JSON structure). Each target annotation includes a bounding box, category label, segmentation mask and corresponding confidence field.

[0021] The above formulas are all dimensionless calculations. Dimensionless calculations can be performed using various methods such as standardization, which will not be elaborated here. The formulas are derived from software simulations based on a large amount of collected data, and the preset parameters in the formulas can be set by those skilled in the art according to the actual situation.

[0022] Example 2 Please see Figure 2 As shown, based on the multimodal data annotation method based on AI and image fusion provided in Embodiment 1 of this application, Embodiment 2 of this application proposes a multimodal data annotation system based on AI and image fusion. Embodiment 2 is merely a preferred embodiment of Embodiment 1, and the implementation of Embodiment 2 will not affect the separate implementation of Embodiment 1.

[0023] Specifically, Embodiment 2 of this application provides a multimodal data annotation system based on AI and image fusion, comprising: Image generation module: Extracts multimodal image data, and based on the preset annotation task requirements, uses an adaptive multi-scale feature fusion network to generate a fused image optimized for the annotation task; Image analysis and classification module: Extracts the signal-to-noise ratio, information entropy difference, and contrast parameters of the fused image, calculates the comprehensive difficulty value by combining the weights corresponding to the scene type, and determines the annotation difficulty level of the fused image based on the comprehensive difficulty value range; The annotation and mapping module uses a pre-trained multi-task AI annotation model to complete the initial annotation on the fused image, and then uses geometric transformation to back-project the annotation results to each original modality image to generate candidate annotations for each modality. Consistency verification module: For fused images with a labeling difficulty level of "difficult", it collects the geometric parameters and feature parameters of candidate labels under different modalities and calculates the single-frame consistency index; for scenes with continuous frame sequences, it triggers temporal consistency verification and corrects to obtain the final consistency index, and determines the confidence level of the labeling results based on the final consistency index. The annotation fusion output module merges the high-confidence AI initial annotation results with the low-confidence annotation results after manual review and correction, and outputs the complete multimodal image annotation results according to the preset standard format.

[0024] Device components: Visible light camera ≥1080P / 25fps, infrared camera, depth sensor; Function: Acquire raw modal images based on different imaging principles, and add acquisition timestamps and sensor calibration parameters to each frame of image.

[0025] External trigger signal generator / PTP master clock, synchronization accuracy ≤10ms; Function: Using the timestamp of one modality as a reference, timestamp matching is performed on the data of the other modalities to ensure that the time synchronization error of cross-modal data is controlled within a single frame acquisition interval.

[0026] GPU servers equipped with pre-trained multi-task models; Function: Performs end-to-end inference on the fused image and outputs initial annotation results such as bounding boxes, category labels, and segmentation masks.

[0027] Inverse mapping projection device; Function: Using sensor calibration parameters, the annotation results on the fused image are back-projected onto the coordinate system of each original modality image through geometric transformation to generate candidate annotations.

[0028] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, ATA hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state ATA hard disk.

[0029] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0030] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0031] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0032] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0033] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0034] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable ATA hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0035] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. The multimodal data annotation method based on AI and image fusion includes the following steps: Generate fused image: Extract multimodal image data, and based on the preset annotation task requirements, use an adaptive multi-scale feature fusion network to generate a fused image optimized for the annotation task; Image fusion analysis and difficulty classification: The signal-to-noise ratio, information entropy difference, and contrast parameters of the fused image are extracted, and the comprehensive difficulty value is calculated by combining the weights corresponding to the scene type. The annotation difficulty level of the fused image is determined based on the comprehensive difficulty value range. Multi-task AI initial annotation and inverse mapping: The pre-trained multi-task AI annotation model is used to complete the initial annotation on the fused image. Then, the annotation results are back-projected to each original modality image through geometric transformation to generate candidate annotations for each modality. Consistency verification analysis: For fused images with a labeling difficulty level of "difficult", the geometric parameters and feature parameters of candidate labels under different modalities are collected, and the single-frame consistency index is calculated; for scenes with continuous frame sequences, temporal consistency verification is triggered and corrected to obtain the final consistency index, and the confidence level of the labeling results is determined based on the final consistency index. Annotation fusion and final output: The high-confidence AI initial annotation results are merged with the low-confidence annotation results after manual review and correction, and the complete multimodal image annotation results are output according to the preset standard format.

2. The multimodal data annotation method based on AI and image fusion according to claim 1, characterized in that, The specific process for obtaining the fused image is as follows: The registered modal images are decomposed into multiple scales, and high-frequency detail features and low-frequency semantic features are extracted at each scale. Based on the preset annotation task type, the fusion weights are dynamically assigned to features of different modalities at each scale; Within the same scale, the corresponding features of different modalities are weighted and summed according to the weights assigned to them at that scale to generate the fused features at the corresponding scale. Multi-scale reconstruction is performed on the fusion features at all scales to generate a fusion image optimized for the current annotation task.

3. The multimodal data annotation method based on AI and image fusion according to claim 1, characterized in that, The specific process for obtaining the overall difficulty value is as follows: The fused image is divided into r sub-blocks, the noise variance and signal variance are calculated, and the signal-to-noise ratio is obtained by formula. Quantize the image to Each gray level has several gray levels; statistics are collected for each gray level. Pixel frequency The total number of pixels is The probability of grayscale appearing is Using the formula Obtain the information entropy value The information entropy difference is obtained by calculating the absolute difference between the information entropy value and a pre-set information entropy threshold. ; Calculate the average pixel brightness of the entire image, and obtain the contrast value by taking the square root of the average of the squared deviations of the pixel brightness from the mean; After normalizing the signal-to-noise ratio, information entropy difference, and contrast value, the formula is used. The overall difficulty value is obtained after weighted calculation. ,in , , These are the corresponding matching weight factors; Two sets of comprehensive difficulty value intervals are preset, and each set of comprehensive difficulty value intervals corresponds to a difficulty level of the fused image. The comprehensive difficulty value is matched with the corresponding comprehensive difficulty value interval to obtain the difficulty level of the fused image. The difficulty levels for image annotation fusion include general and difficult.

4. The multimodal data annotation method based on AI and image fusion according to claim 1, characterized in that, The process of obtaining the single-frame consistency index is as follows: Geometric parameters of candidate labels under different modalities are extracted, including intersection-union ratio, centroid distance similarity, and area ratio. After comprehensive processing, geometric estimates are obtained. Feature parameters of candidate labels under different modalities are extracted, including mutual information value, structural similarity value, and cross-correlation value. After comprehensive processing, feature estimates are obtained. After normalizing the geometric and feature estimates, the single-frame consistency index is calculated using a pre-built formula.

5. The multimodal data annotation method based on AI and image fusion according to claim 4, characterized in that, The specific process of obtaining the geometric estimate is as follows: The bounding boxes located on the original modal image after inverse mapping are used as candidate boxes; Obtain the coordinates of the top left and bottom right corners of the two candidate boxes respectively, calculate the pixel area occupied by each box, obtain the area of ​​the overlapping part of the two areas, calculate the total area of ​​the union of the two areas, and calculate the ratio of the overlapping area to the total area of ​​the union to obtain the intersection-union ratio. Calculate the pixel coordinates of the center points of two candidate boxes and calculate the straight-line distance between the two points; obtain the diagonal length of the image, divide the straight-line distance between the center points by the diagonal length of the image to get the distance ratio, and subtract the distance ratio from 1 to get the center point distance similarity; Calculate the area of ​​each candidate box separately, and use the smaller area value as the numerator and the larger area value as the denominator to calculate the area ratio. After normalizing the intersection-union ratio, center point distance, and area ratio, the formula is used. Geometric valuation is obtained by weighting. ,in , , These are the corresponding preset weighting factors.

6. The multimodal data annotation method based on AI and image fusion according to claim 4, characterized in that, The specific process for obtaining feature estimates is as follows: Quantize the image pixels a and b within the two candidate boxes into K gray levels, and calculate the joint histogram of the gray levels of the two image blocks to obtain the joint probability of each pair of gray levels. Statistically calculate their respective marginal probability distributions and According to the definition of mutual information in information theory, the original value of mutual information can be obtained using the formula. The theoretical maximum value corresponding to the preset mutual information original value. Using the formula The mutual information value is calculated; Set two candidate bounding box image blocks and The number of pixels is ; Statistical analysis of the mean brightness of image patches x and y , ; The standard deviation of contrast is obtained using a pre-constructed formula. , Structural covariance ; Using formula The structural similarity values ​​are obtained, where, , These are pre-defined constants; Calculate the average value of pixels within the two candidate bounding box image blocks, and then subtract the average value from each pixel value to obtain a pixel matrix with zero mean. Calculate the sum of the dot products of the two zero-mean matrices, and calculate the square root of the sum of the squares of each matrix. Divide the sum of the dot products by the product of the square roots of the two matrices to obtain the cross-correlation value. The feature estimates are obtained by comprehensively processing the mutual information value, structural similarity value, and cross-correlation value respectively.

7. The multimodal data annotation method based on AI and image fusion according to claim 1, characterized in that, The specific process of obtaining the eventual consistency index from the timing consistency checksum is as follows: A multi-target tracking algorithm is used to associate the same target in the current frame and the previous and next s frames, and the single-frame consistency index of the target in each frame is extracted to form a sequence C; Calculate the mean of sequence C and variance ,like and If the consistency index of the current frame is used as the final consistency index, then the consistency index of the current frame will be used as the final consistency index. Otherwise, if the timing consistency is insufficient, the exponent of the current frame needs to be reduced, and the final consistency exponent is obtained using a pre-built formula. The final consistency index is compared with a preset threshold. If it is greater than the threshold, it is judged as high confidence and the AI ​​initial annotation result is directly output. If the score is less than or equal to the threshold, it is considered a low confidence level and is submitted for manual review.

8. A multimodal data annotation device based on AI and image fusion, applied to the multimodal data annotation method based on AI and image fusion as described in any one of claims 1-7, characterized in that, include: Visible light and infrared cameras: acquire raw modal images based on different imaging principles, and add acquisition timestamps and sensor calibration parameters to each frame of image; Signal generator: Using the timestamp of one mode as a reference, timestamp matching is performed on the data of the other modes; GPU server for multi-task model: performs end-to-end inference on fused images and outputs initial annotation results such as bounding boxes, class labels and segmentation masks; Inverse mapping projection device: Using sensor calibration parameters, the annotation results on the fused image are back-projected onto the coordinate system of each original modality image through geometric transformation to generate candidate annotations.

9. A multimodal data annotation system based on AI and image fusion, applied to the multimodal data annotation method based on AI and image fusion as described in any one of claims 1-7, characterized in that, include: Image generation module: Extracts multimodal image data, and based on the preset annotation task requirements, uses an adaptive multi-scale feature fusion network to generate a fused image optimized for the annotation task; Image analysis and classification module: Extracts the signal-to-noise ratio, information entropy difference, and contrast parameters of the fused image, calculates the comprehensive difficulty value by combining the weights corresponding to the scene type, and determines the annotation difficulty level of the fused image based on the comprehensive difficulty value range; The annotation and mapping module uses a pre-trained multi-task AI annotation model to complete the initial annotation on the fused image, and then uses geometric transformation to back-project the annotation results to each original modality image to generate candidate annotations for each modality. Consistency verification module: For fused images with a labeling difficulty level of "difficult", it collects the geometric parameters and feature parameters of candidate labels under different modalities and calculates the single-frame consistency index; for scenes with continuous frame sequences, it triggers temporal consistency verification and corrects to obtain the final consistency index, and determines the confidence level of the labeling results based on the final consistency index. The annotation fusion output module merges the high-confidence AI initial annotation results with the low-confidence annotation results after manual review and correction, and outputs the complete multimodal image annotation results according to the preset standard format.

Images