Intelligent image data annotation method and system based on multi-modal semantic fusion

Abstract

The invention is suitable for the technical field of image annotation and intelligent transportation, and provides an intelligent image data annotation method and system based on multi-modal semantic fusion, and the method comprises the following steps: obtaining image data, point cloud data, voice data and environment information of the same time and space; performing semantic segmentation on the image data to obtain image segmentation information; performing alignment processing on the point cloud data and the image data to generate a 3D point cloud aligned with image pixels; according to the 3D point cloud, deducing the geometric contour of the shielded target to obtain point cloud geometric information; fusing the image segmentation information and the point cloud geometric information in combination with environment information to obtain image fusion information; and in combination with the voice data, marking visible targets and shielded targets in the image data with behavior semantics according to the image fusion information. According to the method, multi-modal data such as images, voices and point clouds are fused, so that semantic annotation can be accurately carried out on the occluded target in a complex scene.

Description

Technical Field

[0001] The present invention belongs to the field of image annotation and intelligent transportation technology, and in particular relates to an intelligent image data annotation method and system based on multimodal semantic fusion. Background Art

[0002] In the fields of autonomous driving and intelligent transportation, high-precision and robust environmental perception is a core capability for ensuring safe driving. Achieving this capability relies heavily on large-scale, high-quality annotated image datasets for training key models such as object detection, semantic segmentation, and behavior prediction.

[0003] However, traditional image data annotation technology faces the following problems when meeting the special needs of autonomous driving scenarios: existing annotation methods are mostly limited to a single image modality and lack joint semantic understanding of multi-source data; and it is difficult to accurately annotate complex scenes, such as multi-target occlusion. Summary of the Invention

[0004] The purpose of the present invention is to provide an intelligent image data annotation method based on multimodal semantic fusion, aiming to solve the above technical problems.

[0005] The present invention is implemented as follows: an intelligent image data annotation method based on multimodal semantic fusion, comprising the following steps:

[0006] Acquire image data, point cloud data, voice data, and environmental information at the same time and space;

[0007] Based on the deep learning model, the visible objects in the image data and the occluded areas between multiple objects are identified, and the image data is semantically segmented to obtain image segmentation information;

[0008] Align the point cloud data with the image data to generate a 3D point cloud aligned with the image pixels;

[0009] According to the 3D point cloud, the geometric outline of the occluded target is inferred to obtain the point cloud geometric information;

[0010] Combined with environmental information, the image segmentation information is fused with the point cloud geometry information to obtain image fusion information;

[0011] Combined with speech data, the visible and occluded objects in the image data are annotated with behavioral semantics based on image fusion information.

[0012] Preferably, the step of aligning the point cloud data with the image data to generate a 3D point cloud aligned with the image pixels specifically includes:

[0013] Calculate the extrinsic parameter matrix between image data and point cloud data through feature point matching method;

[0014] Convert the point cloud data into the coordinate system of the image data through the external parameter matrix;

[0015] Based on the coordinate system of the image data, the point cloud data is projected onto the image plane through the intrinsic parameter matrix of the image data to generate a 3D point cloud aligned with the image pixels.

[0016] Preferably, the step of inferring the geometric outline of the occluded object based on the 3D point cloud and obtaining the point cloud geometric information specifically includes:

[0017] Perform clustering processing on the 3D point cloud to obtain clustering results;

[0018] Based on the clustering results, identify the 3D point cloud clusters of the occluded targets in the occluded area;

[0019] According to the 3D point cloud cluster of the occluded target, the geometric outline of the occluded target is inferred and the point cloud geometric information is obtained.

[0020] Preferably, the step of fusing the image segmentation information with the point cloud geometry information in combination with the environmental information to obtain the image fusion information specifically includes:

[0021] Based on the dynamic weight fusion network, the fusion weight of image segmentation information and point cloud geometric information is dynamically adjusted according to the environmental information;

[0022] Image segmentation information and point cloud geometric information are fused according to their fusion weights to obtain image fusion information.

[0023] Preferably, the step of labeling visible objects and occluded objects in the image data with behavioral semantics based on the image fusion information in combination with the speech data specifically includes:

[0024] Extract the speech text from the speech data and convert the speech text into speech semantic vectors;

[0025] Based on the image fusion information and speech semantic vectors, visible objects and occluded objects in the image data are annotated with behavioral semantics.

[0026] Another object of the present invention is to provide an intelligent image data annotation system based on multimodal semantic fusion, which is used to implement the above-mentioned intelligent image data annotation method, and includes:

[0027] Data acquisition module, used to obtain image data, point cloud data, voice data and environmental information in the same time and space;

[0028] The image segmentation module is used to identify visible objects in image data and occluded areas between multiple objects based on a deep learning model, perform semantic segmentation on the image data, and obtain image segmentation information;

[0029] The data alignment module is used to align the point cloud data with the image data to generate a 3D point cloud aligned with the image pixels;

[0030] The point cloud processing module is used to infer the geometric outline of the occluded target based on the 3D point cloud and obtain the point cloud geometric information;

[0031] The information fusion module is used to combine the environmental information, fuse the image segmentation information with the point cloud geometry information, and obtain the image fusion information;

[0032] The image annotation module is used to combine voice data and annotate visible and occluded objects in the image data with behavioral semantics based on image fusion information.

[0033] Preferably, the data alignment module includes:

[0034] an extrinsic parameter matrix determination unit, configured to calculate an extrinsic parameter matrix between image data and point cloud data by a feature point matching method;

[0035] A coordinate system conversion unit, used to convert point cloud data into the coordinate system of image data through an external parameter matrix;

[0036] The point cloud projection unit is used to project the point cloud data onto the image plane through the intrinsic parameter matrix of the image data based on the coordinate system of the image data, and generate a 3D point cloud aligned with the image pixels.

[0037] Preferably, the point cloud processing module includes:

[0038] Point cloud clustering unit, used to perform clustering processing on 3D point clouds to obtain clustering results;

[0039] A point cloud cluster recognition unit is used to identify the 3D point cloud clusters of the occluded targets in the occluded area based on the clustering results;

[0040] The geometric information generation unit is used to infer the geometric contour of the occluded target based on the 3D point cloud cluster of the occluded target and obtain the point cloud geometric information.

[0041] Preferably, the information fusion module includes:

[0042] A fusion weight adjustment unit, configured to dynamically adjust the fusion weight of image segmentation information and point cloud geometric information according to environmental information based on a dynamic weight fusion network;

[0043] The fusion information generating unit is used to fuse the image segmentation information with the point cloud geometric information according to the fusion weight of the image segmentation information and the point cloud geometric information to obtain image fusion information.

[0044] Preferably, the image annotation module includes:

[0045] A speech data processing unit, used to extract speech text from speech data and convert the speech text into speech semantic vectors;

[0046] The semantic annotation unit is used to annotate visible objects and occluded objects in the image data with behavioral semantics based on image fusion information and speech semantic vectors.

[0047] This invention provides an intelligent image data annotation method based on multimodal semantic fusion. By fusing multimodal data such as images, audio, and point clouds, it can accurately semantically annotate occluded objects in complex scenes, making it suitable for autonomous driving model training. Furthermore, by incorporating environmental information and dynamically adjusting the fusion weights of image segmentation and point cloud geometry, this method improves the robustness and accuracy of image data annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of an intelligent image data annotation method based on multimodal semantic fusion provided by an embodiment of the present invention.

[0049] Figure 2 A schematic flow chart of step S300 in the intelligent image data annotation method based on multimodal semantic fusion provided in an embodiment of the present invention.

[0050] Figure 3 A schematic flow chart of step S400 in the intelligent image data annotation method based on multimodal semantic fusion provided in an embodiment of the present invention.

[0051] Figure 4 A schematic flow chart of step S500 in the intelligent image data annotation method based on multimodal semantic fusion provided in an embodiment of the present invention.

[0052] Figure 5 This is a flow chart of step S600 in the intelligent image data annotation method based on multimodal semantic fusion provided by an embodiment of the present invention.

[0053] Figure 6 A schematic diagram of the structure of an intelligent image data annotation system based on multimodal semantic fusion provided by an embodiment of the present invention.

[0054] Figure 7 A schematic diagram of the structure of a data alignment module provided by an embodiment of the present invention.

[0055] Figure 8 A schematic diagram of the structure of a point cloud processing module provided in an embodiment of the present invention.

[0056] Figure 9 This is a structural diagram of the information fusion module provided in an embodiment of the present invention.

[0057] Figure 10 This is a structural diagram of an image annotation module provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0059] like Figure 1 As shown, in one embodiment of the present invention, a method for intelligent image data annotation based on multimodal semantic fusion is provided, comprising the following steps:

[0060] S100, acquiring image data, point cloud data, voice data, and environmental information at the same time and space;

[0061] S200, based on the deep learning model, identifying visible targets in the image data and occlusion areas between multiple targets, performing semantic segmentation on the image data, and obtaining image segmentation information;

[0062] S300, aligning the point cloud data with the image data to generate a 3D point cloud aligned with the image pixels;

[0063] S400, inferring the geometric outline of the occluded target based on the 3D point cloud to obtain point cloud geometric information;

[0064] S500, obtaining combined environmental information, fusing image segmentation information with point cloud geometric information to obtain image fusion information;

[0065] S600 , combining the voice data and labeling the visible objects and the obscured objects in the image data with behavioral semantics according to the image fusion information.

[0066] Specifically, in step S100, the image data is data collected by a visible light or infrared camera; the point cloud data is data collected by a lidar; in actual applications, a hardware trigger signal (such as a GPS timestamp or synchronization pulse) can be used to ensure that the lidar and the camera collect data at the same time point to achieve time synchronization; in addition, the voice data includes vehicle-mounted voice command data and navigation voice data, etc.; the environmental information includes light intensity, atmospheric particulate matter concentration, precipitation, ground flatness, and ground reflectivity, etc.

[0067] In step S200 , visible targets refer to unobstructed targets in the image, including pedestrians, vehicles, obstacles, road signs, etc.; occluded areas between multiple targets refer to areas in the image that contain obscured targets.

[0068] In addition, the embodiment of the present invention adopts a deep learning model as an image segmentation model; specifically, the deep learning model can adopt the Mask R-CNN model, but is not limited thereto. Taking the Mask R-CNN model as an example, its construction and implementation method are as follows:

[0069] 1. Data preparation and annotation: Annotate the complete bounding box of each object (even if it is occluded), accurately annotate the visible part of the object (excluding occluded areas), and additionally annotate the bounding box and mask of the occluded objects (for example, annotate the relationship of "truck occluding pedestrian").

[0070] 2. Data augmentation: Randomly insert occlusions (such as rectangular blocks and other objects) into the training images (RGB images) to simulate real occlusion scenarios.

[0071] 3. Model training and optimization: Use the pre-trained Mask R-CNN (such as COCO pre-trained weights) as the base network; choose ResNet-50 / 101 or the more efficient Swin Transformer as the backbone network; use the attention mechanism and loss function for optimization.

[0072] 4. Occlusion bounding box generation and correction: The model outputs the bounding box and confidence of the target. If the target is occluded, the bounding box may cover the occluder (e.g., the truck frame includes the occluded pedestrian). The occluded area is determined by the missing part of the instance segmentation mask (e.g., the lower body of the pedestrian is not masked). If the occlusion branch is trained, the association between the occluder and the occluded target is directly output (e.g., "truck → pedestrian"). The bounding box is adjusted according to the visible part of the segmentation mask, and only the visible area is selected.

[0073] 5. Image Segmentation: The model generates a segmentation mask for the target, accurately labeling the visible portion; the segmentation mask for occluded areas is naturally missing (for example, the portion of a pedestrian obscured by a truck has no mask).

[0074] like Figure 2 As shown, in a preferred embodiment of the present invention, the step of aligning the point cloud data with the image data to generate a 3D point cloud aligned with the image pixels, i.e., step S300, specifically includes:

[0075] S310, calculating the extrinsic parameter matrix between the image data and the point cloud data by a feature point matching method;

[0076] S320, converting the point cloud data into the coordinate system of the image data through the extrinsic parameter matrix;

[0077] S330 , based on the coordinate system of the image data, projecting the point cloud data onto the image plane through the intrinsic parameter matrix of the image data to generate a 3D point cloud aligned with the image pixels.

[0078] Specifically, a calibration plate with a checkerboard or AprilTag is placed in the scene, and data (point cloud data and image data) are collected using a lidar and camera respectively. Then, the extrinsic parameter matrix (rotation matrix and translation vector) between the two is calculated through feature point matching (such as the corner points of the calibration plate). In addition, the common view features in natural scenes (such as building edges and vehicle contours) can be used to optimize the extrinsic parameter matrix through the ICP (iterative closest point) algorithm.

[0079] In addition, the calibration method of the intrinsic parameter matrix is ​​as follows: calibrate the intrinsic parameters (focal length, distortion coefficient, etc.) of the camera (visible light or infrared) and the intrinsic parameters (beam angle, distance error model, etc.) of the lidar respectively.

[0080] It should be noted that the point cloud data can be preprocessed before alignment; the preprocessing methods include denoising, downsampling, ground segmentation, etc.; specifically, the denoising method is: use statistical filtering to remove outliers (such as sensor noise); the downsampling method is: apply voxel grid filtering to downsample, balancing point cloud density and computational efficiency; the ground segmentation method is: use RANSAC algorithm or plane fitting (such as normal estimation + region growing) to separate the ground point cloud to avoid ground interference in subsequent target clustering processing.

[0081] like Figure 3 As shown, in a preferred embodiment of the present invention, the step of inferring the geometric outline of the occluded object based on the 3D point cloud and obtaining the point cloud geometric information, step S400 specifically includes:

[0082] S410, performing clustering processing on the 3D point cloud to obtain a clustering result;

[0083] S420, identifying the 3D point cloud cluster of the occluded target in the occluded area according to the clustering result;

[0084] S430 : Inferring the geometric outline of the occluded target based on the 3D point cloud cluster of the occluded target to obtain point cloud geometric information.

[0085] Specifically, in step S410, the DBSCAN algorithm can be used to perform density clustering on the non-ground point cloud by setting the neighborhood radius (eps) and the minimum number of points (min_samples), and output a preliminary clustering result containing 3D point cloud clusters, each of which represents a candidate target (such as a vehicle, pedestrian, etc.); then, based on the Euclidean clustering algorithm, the preliminary clustering result can be clustered again to merge spatially adjacent but low-density 3D point cloud clusters (such as scattered point clouds of occluded targets) to obtain the final clustering result, which is actually a point cloud depth map.

[0086] In step S420, based on the clustering results (i.e., the point cloud depth map), areas of sudden depth changes (e.g., a sudden gap in front of the vehicle) are detected and marked as potential occlusion boundaries. The normal direction differences between adjacent 3D point cloud clusters are calculated. If the differences are significant (e.g., greater than 30°), they are determined to be the boundaries of different targets; otherwise, they can be determined to be the boundaries of the same target. Furthermore, if the projections of multiple 3D point cloud clusters in three-dimensional space overlap and have continuous depth, they can be determined to be occluded portions of the same target. The bounding box size and shape (e.g., aspect ratio) of these 3D point cloud clusters are then calculated, and the scattered clusters that meet the characteristics of the target category (e.g., vehicle) are merged to form the 3D point cloud clusters representing the occluded target in the occlusion area.

[0087] In step S430, a model such as a Point Completion Network (PCN) or GRNet can be constructed, taking incomplete 3D point cloud clusters as input and outputting complete point clouds to infer the geometric outline of the occluded object and generate point cloud geometry information. For example, if a pedestrian in an image is partially occluded by a truck, the complete geometric outline can be predicted based on the continuity of the pedestrian's 3D point cloud clusters in the point cloud. This model can be trained adversarially using a large number of training datasets of complete and incomplete point cloud pairs (such as the ShapeNet and KITTI datasets) to generate reasonable geometric outlines.

[0088] like Figure 4 As shown, in a preferred embodiment of the present invention, the step of fusing the image segmentation information with the point cloud geometric information in combination with the environmental information to obtain the image fusion information, i.e., step S500, specifically includes:

[0089] S510, based on a dynamic weight fusion network, dynamically adjusting the fusion weight of the image segmentation information and the point cloud geometric information according to the environmental information;

[0090] S520 , fusing the image segmentation information with the point cloud geometric information according to a fusion weight of the image segmentation information and the point cloud geometric information to obtain image fusion information.

[0091] Specifically, the step of dynamically adjusting the fusion weights of image segmentation information and point cloud geometric information according to environmental information based on the dynamic weight fusion network, i.e., step S510, specifically includes:

[0092] First, the confidence of each pixel in the image segmentation information is calculated using the output of the image segmentation model (i.e., the aforementioned Mask R-CNN model) or uncertainty estimation (such as Monte Carlo Dropout) to obtain the image segmentation confidence. Simultaneously, based on point cloud density and depth consistency (such as the depth difference between adjacent points), the confidence of 3D point cloud clusters in the point cloud geometry information is calculated to obtain the point cloud geometry confidence. Next, based on the image segmentation confidence and point cloud geometry confidence, and in combination with environmental information, the fusion weights of the image segmentation information and point cloud geometry information are dynamically adjusted. Specifically, a lightweight neural network (such as an MLP or a small-scale Transformer) is designed, which inputs environmental information (such as light intensity, atmospheric particulate matter concentration, precipitation, ground flatness, and ground reflectivity), image segmentation confidence, and point cloud geometry confidence, and outputs real-time fusion weights.

[0093] In practical applications, the weight of point cloud geometry information can be dynamically increased for areas with low image segmentation confidence and point cloud geometry confidence (such as occluded areas and shadow areas). For environments with low light intensity and high atmospheric particulate matter concentration, the acquisition of image data will be affected, so the weight of image segmentation information can be dynamically reduced. For environments with high precipitation (including rainfall and snowfall, etc.), the acquisition of both image data and point cloud data will be greatly affected. In this case, the weight of image segmentation information and point cloud geometry information can be balanced, and the specific adjustment can be made using the above method. In environments with poor ground flatness and low ground reflectivity, the low ground reflectivity will cause laser energy to be absorbed or scattered, resulting in weak echo signals, sparse point clouds, or even undetectable points. Uneven ground may lead to uneven point cloud density, so the weight of point cloud geometry information can be dynamically reduced.

[0094] In addition, the step of fusing the image segmentation information with the point cloud geometric information according to the fusion weight of the image segmentation information and the point cloud geometric information to obtain image fusion information, i.e., step S520, specifically includes:

[0095] First, features from the image segmentation information and point cloud geometric information are extracted to obtain image segmentation features (segmentation mask and bounding box information, etc.) and point cloud geometric features (three-dimensional features such as point cloud density, depth, normal direction, and target bounding box). Then, the image segmentation information and point cloud geometric information are aligned, and the image segmentation features and point cloud geometric features are spliced. The fusion weights of the image segmentation information and point cloud geometric information are dynamically adjusted according to the above-mentioned environmental information for weighted fusion to obtain image fusion information.

[0096] like Figure 5 As shown, in a preferred embodiment of the present invention, the step of annotating visible objects and occluded objects in the image data with behavioral semantics based on the image fusion information in combination with the speech data, i.e., step S600, specifically includes:

[0097] S610, extracting speech text from the speech data, and converting the speech text into a speech semantic vector;

[0098] S620: Annotate visible objects and occluded objects in the image data with behavioral semantics based on the image fusion information and the speech semantic vector.

[0099] Specifically, pre-trained models (such as Whisper and Google Speech-to-Text) can be used to convert speech data into speech text. In addition, models such as Sentence-BERT and BERT can be used to convert speech text into speech semantic vectors to capture behavioral intentions (such as "pedestrians are waving" and "vehicles are about to turn left").

[0100] In practical applications, speech data, such as in-vehicle voice command data and navigation voice data, contains semantic information that helps analyze target behavior. For example, the speech text "There is a pedestrian crossing the road ahead" extracted from speech data indicates that there is a target (pedestrian) crossing the road ahead. If the target is obscured by a vehicle, image fusion information combined with speech semantic vectors can more accurately annotate the boundary of the obscured target (pedestrian), and can also annotate behavioral semantics such as "pedestrian-crossing the road." Another example is the speech text "Congestion ahead" extracted from speech data, which indicates that there are many vehicles ahead and the road is congested. In this case, it can be predicted that the target (vehicle) is slowing down or stopping. If the target is obscured by other vehicles, image fusion information combined with speech semantic vectors can more accurately annotate the boundary of the obscured target (vehicle), and can also annotate behavioral semantics such as "vehicle-decelerating" or "vehicle-stopping."

[0101] like Figure 6 As shown, in another embodiment of the present invention, an intelligent image data annotation system based on multimodal semantic fusion is provided, which is used to implement the above-mentioned intelligent image data annotation method, and includes:

[0102] The data acquisition module 10 is used to acquire image data, point cloud data, voice data and environmental information in the same time and space;

[0103] An image segmentation module 20 is configured to identify visible objects and occluded areas between multiple objects in the image data based on a deep learning model, perform semantic segmentation on the image data, and obtain image segmentation information;

[0104] A data alignment module 30 is used to align the point cloud data with the image data to generate a 3D point cloud aligned with the image pixels;

[0105] The point cloud processing module 40 is used to infer the geometric outline of the occluded object based on the 3D point cloud and obtain point cloud geometric information;

[0106] An information fusion module 50 is used to combine the environmental information and fuse the image segmentation information with the point cloud geometry information to obtain image fusion information;

[0107] The image annotation module 60 is used to combine the voice data and annotate the visible objects and occluded objects in the image data with behavioral semantics according to the image fusion information.

[0108] like Figure 7 As shown, in a preferred embodiment of the present invention, the data alignment module 30 includes:

[0109] an extrinsic parameter matrix determining unit 31, configured to calculate an extrinsic parameter matrix between image data and point cloud data by a feature point matching method;

[0110] A coordinate system conversion unit 32 is used to convert the point cloud data into the coordinate system of the image data through the extrinsic parameter matrix;

[0111] The point cloud projection unit 33 is used to project the point cloud data onto the image plane through the intrinsic parameter matrix of the image data based on the coordinate system of the image data, so as to generate a 3D point cloud aligned with the image pixels.

[0112] like Figure 8 As shown, in a preferred embodiment of the present invention, the point cloud processing module 40 includes:

[0113] The point cloud clustering unit 41 is used to perform clustering processing on the 3D point cloud to obtain a clustering result;

[0114] a point cloud cluster identification unit 42 for identifying the 3D point cloud cluster of the occluded target in the occluded area based on the clustering result;

[0115] The geometric information generating unit 43 is configured to infer the geometric outline of the occluded object based on the 3D point cloud cluster of the occluded object, and obtain point cloud geometric information.

[0116] like Figure 9 As shown, in a preferred embodiment of the present invention, the information fusion module 50 includes:

[0117] A fusion weight adjustment unit 51 is used to dynamically adjust the fusion weight of the image segmentation information and the point cloud geometric information according to the environmental information based on the dynamic weight fusion network;

[0118] The fusion information generating unit 52 is configured to fuse the image segmentation information with the point cloud geometric information according to a fusion weight of the image segmentation information and the point cloud geometric information to obtain image fusion information.

[0119] like Figure 10 As shown, in a preferred embodiment of the present invention, the image annotation module 60 includes:

[0120] The speech data processing unit 61 is used to extract speech text from the speech data and convert the speech text into speech semantic vectors;

[0121] The semantic tagging unit 62 is used to tag visible objects and occluded objects in the image data with behavioral semantics based on the image fusion information and the speech semantic vector.

[0122] It should be noted that the above modules and units can be implemented in the form of a computer program, which can be run on a computer device. The computer program constituting each module can be stored in the memory of the computer device so that the processor executes each step of the above method.

[0123] It should be understood that, although the various steps in the flow chart of each embodiment of the present invention are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0124] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory.

[0125] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. An intelligent image data annotation method based on multimodal semantic fusion, characterized in that: The following steps are involved: Acquire image data, point cloud data, voice data, and environmental information at the same time and space; Based on the deep learning model, the visible objects in the image data and the occluded areas between multiple objects are identified, and the image data is semantically segmented to obtain image segmentation information; Align the point cloud data with the image data to generate a 3D point cloud aligned with the image pixels; According to the 3D point cloud, the geometric outline of the occluded target is inferred to obtain the point cloud geometric information; Combined with environmental information, the image segmentation information is fused with the point cloud geometry information to obtain image fusion information; Combined with speech data, the visible and occluded objects in the image data are annotated with behavioral semantics based on image fusion information.

2. The intelligent image data annotation method based on multimodal semantic fusion according to claim 1 is characterized in that: The steps of aligning the point cloud data with the image data to generate a 3D point cloud aligned with the image pixels include: Calculate the extrinsic parameter matrix between image data and point cloud data through feature point matching method; Convert the point cloud data into the coordinate system of the image data through the external parameter matrix; Based on the coordinate system of the image data, the point cloud data is projected onto the image plane through the intrinsic parameter matrix of the image data to generate a 3D point cloud aligned with the image pixels.

3. The intelligent image data annotation method based on multimodal semantic fusion according to claim 1 is characterized in that: Based on the 3D point cloud, the geometric outline of the occluded object is inferred and the steps of obtaining the point cloud geometric information are as follows: Perform clustering processing on the 3D point cloud to obtain clustering results; Based on the clustering results, identify the 3D point cloud clusters of the occluded targets in the occluded area; According to the 3D point cloud cluster of the occluded target, the geometric outline of the occluded target is inferred and the point cloud geometric information is obtained.

4. The intelligent image data annotation method based on multimodal semantic fusion according to claim 1 is characterized in that: The steps of fusing image segmentation information with point cloud geometry information in combination with environmental information to obtain image fusion information include: Based on the dynamic weight fusion network, the fusion weight of image segmentation information and point cloud geometric information is dynamically adjusted according to the environmental information; Image segmentation information and point cloud geometric information are fused according to their fusion weights to obtain image fusion information.

5. The intelligent image data annotation method based on multimodal semantic fusion according to claim 1 is characterized in that: The steps of labeling visible objects and occluded objects in the image data with behavioral semantics based on the image fusion information in combination with the speech data include: Extract the speech text from the speech data and convert the speech text into speech semantic vectors; Based on the image fusion information and speech semantic vectors, visible objects and occluded objects in the image data are annotated with behavioral semantics.

6. An intelligent image data annotation system based on multimodal semantic fusion, used to implement the intelligent image data annotation method according to any one of claims 1 to 5, characterized in that: include: Data acquisition module, used to obtain image data, point cloud data, voice data and environmental information in the same time and space; The image segmentation module is used to identify visible objects in image data and occluded areas between multiple objects based on a deep learning model, perform semantic segmentation on the image data, and obtain image segmentation information; The data alignment module is used to align the point cloud data with the image data to generate a 3D point cloud aligned with the image pixels; The point cloud processing module is used to infer the geometric outline of the occluded target based on the 3D point cloud and obtain the point cloud geometric information; The information fusion module is used to combine the environmental information, fuse the image segmentation information with the point cloud geometry information, and obtain the image fusion information; The image annotation module is used to combine voice data and annotate visible and occluded objects in the image data with behavioral semantics based on image fusion information.

7. The intelligent image data annotation system based on multimodal semantic fusion according to claim 6, characterized in that: The data alignment module includes: an extrinsic parameter matrix determination unit, configured to calculate an extrinsic parameter matrix between image data and point cloud data by a feature point matching method; A coordinate system conversion unit, used to convert point cloud data into the coordinate system of image data through an external parameter matrix; The point cloud projection unit is used to project the point cloud data onto the image plane through the intrinsic parameter matrix of the image data based on the coordinate system of the image data, and generate a 3D point cloud aligned with the image pixels.

8. The intelligent image data annotation system based on multimodal semantic fusion according to claim 6, characterized in that: The point cloud processing module includes: Point cloud clustering unit, used to perform clustering processing on 3D point clouds to obtain clustering results; A point cloud cluster recognition unit is used to identify the 3D point cloud clusters of the occluded targets in the occluded area based on the clustering results; The geometric information generation unit is used to infer the geometric contour of the occluded target based on the 3D point cloud cluster of the occluded target and obtain the point cloud geometric information.

9. The intelligent image data annotation system based on multimodal semantic fusion according to claim 6, characterized in that: The information fusion module includes: A fusion weight adjustment unit, configured to dynamically adjust the fusion weight of image segmentation information and point cloud geometric information according to environmental information based on a dynamic weight fusion network; The fusion information generating unit is used to fuse the image segmentation information with the point cloud geometric information according to the fusion weight of the image segmentation information and the point cloud geometric information to obtain image fusion information.

10. The intelligent image data annotation system based on multimodal semantic fusion according to claim 6, characterized in that: The image annotation module includes: A speech data processing unit, used to extract speech text from speech data and convert the speech text into speech semantic vectors; The semantic annotation unit is used to annotate visible objects and occluded objects in the image data with behavioral semantics based on image fusion information and speech semantic vectors.

Images