Image data annotation system and method based on deep learning

Abstract

The invention discloses an image data annotation system based on deep learning, and relates to the technical field of image processing, and the system comprises a multi-modal acquisition unit, a preprocessing module, a feature fusion unit, a scene classification module and a visualization module. The method comprises the following steps: a multi-modal acquisition unit acquires scene information; a preprocessing module performs preliminary analysis on the scene information; a feature fusion unit performs fusion analysis on image features and voice features; a scene classification module performs hierarchical prediction classification on dynamic scenes; according to the method, through the steps of multi-modal fusion, geographical dynamic adaptation, asymmetric feature fusion, hierarchical classification and the like, the precision, robustness and automation level of image data annotation are remarkably improved; geographic priori knowledge and deep learning are deeply fused, an efficient and reliable marking tool is provided for the fields of smart cities, automatic driving and the like, and the method has a wide application prospect.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image data annotation system and method based on deep learning. Background Art

[0002] In the field of map navigation service applications, ground vehicle collection is one of the main methods. Vehicles are equipped with various sensors, such as lidar, cameras, etc., which can efficiently collect road data; UAV collection is also gradually being applied. It can quickly obtain large-area regional images, especially suitable for areas that are difficult to reach or have inconvenient transportation, such as remote mountainous areas, islands, etc., and can supplement the deficiencies of ground collection; Satellite remote sensing image collection provides macroscopic geographical information for maps, which can be used to draw basic map information such as large-scale topographies, urban outlines, etc. And with the development of satellite technology, the image resolution is continuously improved and the information is more detailed; Therefore, there are more extensive and refined requirements for image processing and annotation after image collection. However, traditional methods usually only rely on image information and ignore the synergistic effect of multi-modal data such as voice and geographical features. In the field of map navigation service applications, the limitation of single-image modality processing and annotation is that the scene understanding is one-sided, which in turn leads to low work efficiency of image collection, processing and annotation, and it is difficult to meet the current navigation needs of users. In view of the above technical defects, a solution is now proposed. Summary of the Invention

[0003] The purpose of the present invention is to solve the problem of the limitation of one-sided scene understanding in single-image modality processing and annotation, and the resulting defects of low work efficiency of image collection, processing and annotation, which are difficult to meet the current navigation needs of users.

[0004] To achieve the above purpose, the present invention adopts the following technical solutions: An image data annotation method based on deep learning, comprising the following steps: Step 1, a multi-modal acquisition unit obtains scene information: the scene information includes geographical image data and user voice data Vu; wherein, the geographical image data includes geographical image Ig and geographical feature Gk; Step 2, a preprocessing module preliminarily analyzes the scene information: a standardized geographical image Ie is obtained through the geographical image data; a standardized voice signal Vc is extracted through the user voice data; Step 3: The feature fusion unit performs fusion analysis on the image features and speech features. By constructing an image feature extraction network, the standardized geographical image Ie is input and combined with the geographical feature Gk to output the deep image feature Fi. By constructing a speech feature extraction network, the standardized speech signal Vc is input and combined with the geographical feature Gk to output the deep semantic feature Fs. Then, an asymmetric fusion model is constructed to obtain the fusion feature Ff of the scene f. Step 4: The scene classification module performs hierarchical prediction and classification on the dynamic scene. A hierarchical classifier is constructed to expand the classification task according to the hierarchical structure, so as to obtain the prediction probability P(Q|F) that the fusion feature F of any dynamic scene conforms to the scene category Q. Step 5: The visualization module integrates and annotates the fusion feature F of the dynamic scene and its prediction probability P(Q|F) that conforms to the scene category Q as the scene classification result and performs visual display.

[0005] Furthermore, the specific process of performing fusion analysis on the image features and speech features is as follows: By constructing an image feature extraction network, the standardized geographical image Ie is input and combined with the geographical feature to extract and output the deep image feature Fi. By constructing a speech feature extraction network, the standardized speech signal Vc is input and combined with the geographical feature to extract the geographical semantic feature Sg, and through further optimization and refinement, the deep semantic feature Fs is obtained and output. Then, an asymmetric fusion model is constructed. By inputting and splicing the deep image feature Fi, the deep semantic feature Fs, and the geographical feature Gk, the feature weight coefficient is obtained through linear transformation, and then the deep image feature Fi and the deep semantic feature Fs are weighted and fused to obtain the fusion feature Ff of the scene.

[0006] Furthermore, the specific process of obtaining the standardized geographical image Ie is as follows: The geographical image Ie includes the parameter values of the pixel points in the three RGB channels. A two-dimensional coordinate system is established through the geographical image Ie to mark the coordinates of the pixel points. The image width of the geographical image Ie includes W pixel points, and the image height includes H pixel points. Through the coordinates and RGB parameters of the geographical image Ie, a three-dimensional data matrix is constructed, so the geographical image Ie belongs to the real number set of the three-dimensional data matrix, that is ; Perform Gaussian filtering on the geographical image Ig to extract the low-frequency illumination information of the image and obtain the filtered image TFg. Substitute the geographical image Ig and the filtered image TFg into the logarithmic domain for calculation and then find the difference to obtain the removal of the low-frequency illumination component and the retention of the image reflection component. The geographical feature Gk includes altitude alt, longitude lon, and latitude lat; the location of the scene image is located through longitude and latitude, and the image is enhanced adaptively by altitude, a dynamic mapping relationship between altitude and illumination parameters is established, and a geographical parameter adjustment factor λ is obtained; The geographical parameter adjustment factor λ is combined with the filtered image TFg to obtain an image TZg with enhanced illumination components, and then it is fused with the image reflection component to obtain a normalized geographical image Ie.

[0007] Furthermore, the specific process of extracting the normalized voice signal Vc is as follows: The user voice data Vu includes the voice signal frequency and time nodes. The marked time nodes include T time dimensions, and the voice signal frequency includes F frequency dimensions; Through the time dimension and frequency dimension of the user voice data Vu, a two-dimensional data matrix is constructed, and the user voice data Vu belongs to the real number set of the two-dimensional data matrix, that is, ; A voice separation model is built. The time-domain voice signal Vu is transformed to the frequency domain through Fourier transform, and then a preset mask matrix M is used to mask the noise. It is substituted into the non-linear filter U for feature extraction, and then inverse Fourier transform is performed to separate the pure voice signal, which is marked as the normalized voice signal Vc.

[0008] Furthermore, the specific process of outputting the depth image feature Fi by constructing an image feature extraction network is as follows: The normalized geographical image Ie is subjected to feature extraction through a deep residual network model, and the feature vector of the geographical image Ie is obtained and marked as the image feature e; Through multi-layer perceptron transformation of the geographical feature Gk, and then splicing with the image feature e, the depth image feature Fi with dimension R d1 is output.

[0009] Furthermore, the specific process of outputting the depth semantic feature Fs by constructing a voice feature extraction network is as follows: Through ASR automatic speech recognition of the normalized voice signal Vc, the voice signal is converted into text mode; Then combined with the geographical feature Gk, it is substituted into the BERT-GEO model to output the geographical semantic feature Sg; The geographical semantic feature Sg is input into the Transformer architecture, and the depth semantic feature Fs with dimension R d2 is output, so as to extract the semantic information related to the geographical location in the voice signal text.

[0010] Furthermore, the specific process of constructing an asymmetric fusion model to obtain the fusion feature Ff of the scene is as follows: By concatenating the input depth image feature Fi, depth semantic feature Fs, and geographical feature Gk, a feature vector set is obtained; If the dimensions of the depth image feature Fi, depth semantic feature Fs, and geographical feature Gk are marked as d1, d2, and d3 respectively, then the dimension of the feature vector set is (d1 + d2 + d3); Set the dimension of the weight matrix Wa as m×(d1 + d2 + d3). By performing a linear transformation on the concatenated feature vector set and multiplying the weight matrix Wa with the feature vector set through matrix multiplication, different - dimensional features are weighted and combined to obtain a new vector with dimension m. Then, the new vector is input into the Softmax function to obtain the attention weights of the features, which are marked as feature weight coefficients; Then, the depth image feature Fi and depth semantic feature Fs are weighted and fused. The feature fusion calculation is performed in an element - by - element multiplication manner and multiplied by the corresponding feature weight coefficients to obtain the fused feature Ff of the scene.

[0011] Furthermore, the specific process of hierarchical prediction classification is as follows: Construct a hierarchical classifier. Mark any dynamic scene as the scene category Q, set the N - layer class labels of the scene category Q, and mark the n - th layer class label as Q n , and mark the class label of the layer above the n - th layer as Q n-1 ; Input the class label Q n-1 of the (n - 1) - th layer and the fused feature Ff into the hierarchical classifier. Through the Softmax activation function calculation, obtain the conditional probability P(Q n |Q n-1 , Ff) of the n - th layer. Then, through the consecutive multiplication calculation of the conditional probabilities of the N layers, obtain the prediction probability P(Q|F) that the fused feature F of any dynamic scene conforms to the scene category Q.

[0012] An image data annotation system based on deep learning includes a multi - modal acquisition unit, a pre - processing module, a feature fusion unit, a scene classification module, and a visualization module. This system applies the above - mentioned image data annotation method based on deep learning; The multi - modal acquisition unit is used to obtain scene information. Among them, the multi - modal acquisition unit includes an image acquisition module and a voice acquisition module; the geographical image data and user voice data are respectively collected through the image acquisition module and the voice acquisition module; The pre - processing module is used to perform a preliminary analysis on the scene information to obtain a standardized geographical image and a standardized voice signal; The feature fusion unit is used to perform fusion analysis on image features and speech features. Among them, the feature fusion unit includes an image feature extraction network, a speech feature extraction network, and an asymmetric fusion module. The image feature extraction network and the speech feature extraction network are respectively used to extract image features and speech features, and output deep image features and deep semantic features; the asymmetric fusion module combines the deep image features and the deep semantic features to obtain fusion features; The scene classification module is used to perform hierarchical prediction classification on dynamic scenes to obtain the prediction probability of the scene category; The visualization module is used to integrate and annotate the fusion features of any dynamic scene and their prediction probabilities that match the scene category as the scene classification result.

[0013] In summary, due to the adoption of the above technical solutions, the beneficial effects of the present invention are as follows: The present invention enhances the annotation accuracy through multi-modal data collaboration, complements geographical images with user speech, and performs adaptive enhancement processing on images in combination with geographical features to achieve dynamic illumination compensation of images; improves the information utilization rate through asymmetric feature fusion, fuses deep image features and deep semantic features, and realizes the efficient fusion of multi-modal feature data of scenes through cross-modal attention mechanism and differential feature space design; and then optimizes the dynamic scene processing through a hierarchical classifier to improve the fineness of scene classification prediction; Through steps such as multi-modal fusion, geographical dynamic adaptation, asymmetric feature fusion, and hierarchical classification, the present invention significantly improves the accuracy, robustness, and automation level of image data annotation, deeply integrates geographical prior knowledge with deep learning, provides an efficient and reliable annotation tool for fields such as smart cities and autonomous driving, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Shows a connection diagram of the system modules of the present invention; Figure 2 Shows a flowchart of the overall solution of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0015] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0016] Embodiment 1: As Figure 1 - Figure 2As shown in the figure, this technical solution constructs an image data annotation system based on deep learning through multi-modal fusion, dynamic adaptation of geographical parameters, asymmetric feature fusion, and hierarchical classifier design, including a multi-modal acquisition unit, a preprocessing module, a feature fusion unit, a scene classification module, and a visualization module; Among them, the multi-modal acquisition unit, the preprocessing module, the feature fusion unit, the scene classification module, and the visualization module are communicatively connected; the multi-modal acquisition unit includes an image acquisition module and a voice acquisition module; the feature fusion unit includes an image feature extraction network, a voice feature extraction network, and an asymmetric fusion module; The multi-modal acquisition unit is used to obtain scene information. Among them, the geographical image data and the user voice data are respectively collected through the image acquisition module and the voice acquisition module; The preprocessing module is used to perform a preliminary analysis on the scene information to obtain a standardized geographical image and a standardized voice signal; The feature fusion unit is used to perform fusion analysis on the image feature and the voice feature. Among them, the image feature and the voice feature are respectively extracted through the image feature extraction network and the voice feature extraction network, and the deep image feature and the deep semantic feature are output; the asymmetric fusion module combines the deep image feature and the deep semantic feature to obtain a fusion feature; The scene classification module is used to perform hierarchical prediction classification on the dynamic scene to obtain the prediction probability of the scene category; The visualization module is used to integrate and annotate the fusion feature of any dynamic scene and its prediction probability that conforms to the scene category as the scene classification result, and perform visual display.

[0017] The specific working steps are as follows: S1. The multi-modal acquisition unit obtains scene information: The scene information includes geographical image data and user voice data Vu; among them, the geographical image data includes geographical image Ig and geographical feature Gk; The geographical feature Gk includes altitude alt, longitude lon, and latitude lat.

[0018] S2. The preprocessing module performs a preliminary analysis on the scene information: A standardized geographical image Ie is obtained through the geographical image data; a standardized voice signal Vc is extracted through the user voice data; S2-1. The specific process of the preliminary analysis of the geographical image data is as follows: Combining the geographical feature with Gaussian filtering, perform denoising adjustment processing on the geographical image Ig to obtain a standardized geographical image Ie. The specific process is as follows: The geographical image Ie includes the parameter values of pixel points in the three RGB channels. A two-dimensional coordinate system is established through the geographical image Ie, and the pixel points are marked with coordinates. The pixel point coordinates include the abscissa x and the ordinate y of the pixel point. The image width of the geographical image Ie includes W pixel points, and the image height includes H pixel points. Based on the coordinates and RGB parameters of the geographical image Ie, a three-dimensional data matrix is constructed. Then the geographical image Ie belongs to the real number set of the three-dimensional data matrix, that is ; Perform Gaussian filtering on the geographical image Ig, perform smoothing processing on the image through convolution operation, extract the low-frequency illumination information of the image, and obtain the filtered image TFg. Among them, the parameters of Gaussian filtering are experimentally tested according to the image characteristics and the experience of those skilled in the art. Substitute the geographical image Ig and the filtered image TFg into the logarithmic domain calculation and then find the difference through the Retinex image enhancement algorithm, so as to obtain the removal of the low-frequency illumination component and retain the image reflection component. Locate the position of the scene image through longitude and latitude, and perform image enhancement through altitude adaptation, and establish a dynamic mapping relationship between altitude and illumination parameters. Specifically: by setting the altitude comparison value alt0, compare the altitude alt of the scene image with the altitude comparison value, obtain the difference between the two and substitute it into the Sigmoid function to obtain the geographical parameter adjustment factor λ. Combine the geographical parameter adjustment factor λ with the filtered image TFg to obtain the image TZg with enhanced illumination component, and then fuse it with the image reflection component to obtain the standardized geographical image Ie.

[0019] S2-2. The specific process of initially analyzing the user voice data is as follows: Perform denoising and separation processing on the user voice data Vu, and extract the standardized voice signal Vc. The specific process is as follows: The user voice data Vu includes the voice signal frequency and time nodes. The marked time nodes include T time dimensions, and the voice signal frequency includes F frequency dimensions. Based on the time dimension and frequency dimension of the user voice data Vu, a two-dimensional data matrix is constructed. Then the user voice data Vu belongs to the real number set of the two-dimensional data matrix, that is ; Build a voice separation model, convert the time-domain voice signal Vu to the frequency domain through Fourier transform, then preset the mask matrix M to shield the noise, substitute it into the non-linear filter U for feature extraction, and then perform inverse Fourier transform to separate the pure voice signal and mark it as the standardized voice signal Vc.

[0020] S3. The feature fusion unit performs fusion analysis on the image features and speech features: By constructing an image feature extraction network, the standardized geographical image Ie is input and combined with the geographical feature Gk, so as to output the deep image feature Fi; By constructing a speech feature extraction network, the standardized speech signal Vc is input and combined with the geographical feature Gk, so as to output the deep semantic feature Fs; Then construct an asymmetric fusion model to obtain the fusion feature Ff of the scene f. The specific process of the feature fusion module is as follows: S3-1. By constructing an image feature extraction network, the standardized geographical image Ie is input and combined with the geographical feature, so as to extract and output the deep image feature Fi. The specific process is as follows: Through the ResNet-50 deep residual network model, the standardized geographical image Ie is subjected to feature extraction to obtain the feature vector of the geographical image Ie, which is marked as the image feature e; Through the multi-layer perceptron MLP transformation of the geographical feature Gk, it is then concatenated with the image feature e, and after linear transformation and activation function processing, the deep image feature Fi with a dimension of R 2048 is output. Among them, the feature extraction, linear transformation and activation function of the ResNet-50 deep residual network model are existing technologies. Based on the existing technologies, the present invention introduces the geographical feature Gk for data splicing processing, which is used for multi-modal fine annotation of the data of geographical images.

[0021] S3-2. By constructing a speech feature extraction network, the standardized speech signal Vc is input and combined with the geographical feature, so as to extract the geographical semantic feature Sg, and through further optimization and refinement, the deep semantic feature Fs is obtained and output. The specific process is as follows: Through the automatic speech recognition ASR of the standardized speech signal Vc, the speech signal is converted into a text mode, and then combined with the geographical feature Gk and substituted into the BERT-GEO model to output the geographical semantic feature Sg, so as to extract the semantic information related to the geographical location in the speech signal text; It should be noted that: The geographical feature Gk includes a large number of geographical-related feature parameters, not limited to altitude and longitude and latitude. The present invention emphasizes altitude and longitude and latitude in order to evaluate the influence of altitude on light intensity, so as to solve the problem of unstable image quality. In the process of extracting geographical semantic features, a large database of geographical features is required for comparison and support. For example, natural geographical features and human geographical features. Natural geographical features include topographical features, climate features, hydrological features, vegetation and soil, and mineral resources, etc.; Human geographical features include population and settlements, economic geography, and culture and society, etc.; The geographical feature Gk specifically collects parameters in combination with actual applications, so as to update and supplement the database of geographical features; Input the geographical semantic feature Sg into the Transformer architecture. Process the input geographical semantic feature Sg through the self-attention mechanism of the Transformer. The self-attention mechanism can capture the correlation between input features and assign different weights to each feature. Then, through the multi-layer neural network structure in the Transformer architecture, set semantic samples such as crowd density and scene atmosphere, further refine and transform the semantic features, and output the deep semantic feature Fs with a dimension of R. 512 The deep semantic feature Fs with a dimension of R; Among them, the further refinement and transformation of the semantic features by the multi-layer neural network structure in the existing Transformer architecture include operations such as the multi-head self-attention mechanism, residual connection, and layer normalization. Those skilled in the art can implement it through existing technologies to enhance the expression ability of features; the output dimension is R. 512 The deep semantic feature with a dimension of R refers to the feature after being processed by the Transformer being mapped into a 512-dimensional vector space for comprehensively and finely representing geographical features; Extract the scene semantics through voice data. For example, when the user describes "the fog is thick on the mountain top", combined with the lighting and terrain features of the geographical image, solve the ambiguity problem that may be caused by a single modality. The multi-modal fusion improves the annotation accuracy of complex scenes.

[0022] S3-3. Then, construct an asymmetric fusion model. By inputting and splicing the depth image feature Fi, the deep semantic feature Fs, and the geographical feature Gk, obtain the feature weight coefficient through linear transformation, and then perform weighted fusion on the depth image feature Fi and the deep semantic feature Fs to obtain the fusion feature Ff of the scene. The specific process is as follows: By inputting and splicing the depth image feature Fi, the deep semantic feature Fs, and the geographical feature Gk, obtain the feature vector set; Mark the dimensions of the depth image feature Fi, the deep semantic feature Fs, and the geographical feature Gk as d1, d2, and d3 respectively. Then the dimension of the feature vector set is (d1 + d2 + d3); set the dimension of the weight matrix Wa as m×(d1 + d2 + d3). The weight matrix Wa is preset after calculating a large number of experimental data on the degree of correlation influence of the input feature data on scene classification, and is continuously adjusted according to the input data samples; Through linear transformation of the spliced feature vector set, perform matrix multiplication on the weight matrix Wa and the feature vector set, so as to perform weighted combination on features of different dimensions, obtain a new vector with a dimension of m, then input the new vector into the Softmax function to obtain the attention weight of the feature, and mark it as the feature weight coefficient. The feature weight coefficient reflects the importance degree of features of different dimensions for the scene classification task; For example, in a certain geographical scenario, if the model learns that the geographical feature Gk is more important for the classification task, then the attention weight of the part related to the geographical feature Gk calculated by the Softmax function will be relatively large, so that this part of the feature will be given a higher weight in the subsequent weighted fusion; Then, the depth image feature Fi and the depth semantic feature Fs are weighted and fused. The feature fusion calculation is performed by element-wise multiplication and multiplied by the corresponding feature weight coefficient to obtain the fused feature Ff of the scene.

[0023] S4. The scene classification module performs hierarchical prediction and classification on the dynamic scene: a hierarchical classifier is constructed to expand the classification task according to the hierarchical structure, so as to obtain the prediction probability P(Q|F) that the fused feature F of any dynamic scene conforms to the scene category Q; Construct a hierarchical classifier, and the hierarchical classifier expands the classification task according to the hierarchical structure. Specifically: by classifying the dynamic scene, for example, the first layer is to distinguish indoor and outdoor, and the second layer further subdivides specific scenes on the basis of the first layer, such as shopping malls, hospitals, etc., so as to build a hierarchical classifier based on the neural network; Mark any dynamic scene as the scene category Q, set the N-layer category labels of the scene category Q, and mark the n-layer category label as Q n , and mark the category label of the upper layer of the nth layer as Q n-1 ; Input the category label Q n-1 of the (n - 1)th layer and the fused feature Ff into the hierarchical classifier, and calculate the conditional probability P(Q n |Q n-1 , Ff) of the nth layer through the Softmax activation function. Furthermore, by calculating the product of the conditional probabilities of the N layers, obtain the prediction probability P(Q|F) that the fused feature F of any dynamic scene conforms to the scene category Q.

[0024] S5. The visualization module integrates and annotates the fused feature F of the dynamic scene and its prediction probability P(Q|F) that conforms to the scene category Q as the scene classification result and performs visual display; For image data, not only objects can be annotated, but also the entire image scene can be understood and annotated, such as scene category, atmosphere, etc. Matching predictions are made according to the scene category labels, so as to achieve scene classification and annotation.

[0025] In summary, the present invention enhances the annotation accuracy through multi-modal data collaboration, complements geographical images with user voices, adaptively enhances the images in combination with geographical features to achieve dynamic illumination compensation of the images; improves the information utilization rate through asymmetric feature fusion, fuses depth image features and depth semantic features, and realizes the efficient fusion of multi-modal feature data of the scene through a cross-modal attention mechanism and a differential feature space design; and then optimizes the dynamic scene processing through a hierarchical classifier to improve the fineness of scene classification prediction. Through steps such as multi-modal fusion, geographical dynamic adaptation, asymmetric feature fusion, and hierarchical classification, the present invention significantly improves the accuracy, robustness, and automation level of image data annotation, deeply integrates geographical prior knowledge with deep learning, provides an efficient and reliable annotation tool for fields such as smart cities and autonomous driving, and has broad application prospects.

[0026] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution.

[0027] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0028] The above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, makes equivalent substitutions or changes, and should be covered by the protection scope of the present invention.

Claims

1. An image data annotation method based on deep learning, characterized in that: It includes the following steps: Step 1, the multi-modal acquisition unit obtains scene information: the scene information includes geographical image data and user voice data Vu; among them, the geographical image data includes geographical image Ig and geographical feature Gk; Step 2, the preprocessing module conducts a preliminary analysis on the scene information: obtaining a standardized geographical image Ie through the geographical image data; extracting a standardized voice signal Vc through the user voice data; Step 3, the feature fusion unit conducts a fusion analysis on the image feature and the voice feature: by constructing an image feature extraction network, inputting the standardized geographical image Ie and combining it with the geographical feature Gk, so as to output a deep image feature Fi; by constructing a voice feature extraction network, inputting the standardized voice signal Vc and combining it with the geographical feature Gk, so as to output a deep semantic feature Fs; Then construct an asymmetric fusion model to obtain the fusion feature Ff of the scene f; Step 4, the scene classification module conducts a hierarchical prediction classification on the dynamic scene: constructing a hierarchical classifier to expand the classification task according to the hierarchical structure, so as to obtain the prediction probability P(Q|F) that the fusion feature F of any dynamic scene conforms to the scene category Q; Step 5, the visualization module integrates and annotates the fusion feature F of the dynamic scene and its prediction probability P(Q|F) that conforms to the scene category Q as the scene classification result, and conducts a visual display.

2. The method for annotating image data based on deep learning according to claim 1, wherein: The specific process of conducting a fusion analysis on the image feature and the voice feature is as follows: By constructing an image feature extraction network, inputting the standardized geographical image Ie and combining it with the geographical feature, so as to extract and output a deep image feature Fi; By constructing a voice feature extraction network, inputting the standardized voice signal Vc and combining it with the geographical feature, so as to extract the geographical semantic feature Sg, and through further optimization and refinement, obtain and output a deep semantic feature Fs; Then construct an asymmetric fusion model, splice and input the deep image feature Fi, the deep semantic feature Fs and the geographical feature Gk, obtain the feature weight coefficient through a linear transformation, and then conduct a weighted fusion on the deep image feature Fi and the deep semantic feature Fs to obtain the fusion feature Ff of the scene.

3. The method for image data annotation based on deep learning according to claim 2, wherein: The specific process of obtaining the standardized geographical image Ie is as follows: The geographical image Ie includes the parameter values of pixel points in the three RGB channels. A two-dimensional coordinate system is established through the geographical image Ie, and the pixel points are marked with coordinates. The image width of the geographical image Ie includes W pixel points, and the image height includes H pixel points; a three-dimensional data matrix is constructed through the coordinates and RGB parameters of the geographical image Ie, then the geographical image Ie belongs to the real number set of the three-dimensional data matrix, that is ; Perform Gaussian filtering on the geographical image Ig, extract the low-frequency illumination information of the image, and obtain the filtered image TFg; Substitute the geographical image Ig and the filtered image TFg into the logarithmic domain for calculation and then find the difference, so as to obtain the removal of the low-frequency illumination component and retain the image reflection component; The geographical feature Gk includes altitude alt, longitude lon and latitude lat; Locate the position where the scene image is located through the longitude and latitude, and perform image enhancement adaptively through the altitude, establish a dynamic mapping relationship between the altitude and the illumination parameters, and obtain the geographical parameter adjustment factor λ; Combine the geographical parameter adjustment factor λ with the filtered image TFg to obtain the image TZg with enhanced illumination component, and then fuse it with the image reflection component to obtain the standardized geographical image Ie.

4. A method for image data annotation based on deep learning according to claim 3, characterized in that: The specific process of extracting the standardized voice signal Vc is as follows: The user voice data Vu includes the voice signal frequency and the time node, the marked time node includes T time dimensions, and the voice signal frequency includes F frequency dimensions; Construct a two-dimensional data matrix through the time dimension and frequency dimension of the user voice data Vu. Then the user voice data Vu belongs to the real number set of the two-dimensional data matrix, that is ; Build a voice separation model. Convert the time-domain voice signal Vu to the frequency domain through Fourier transform, then preset the mask matrix M to mask the noise, substitute it into the non-linear filter U for feature extraction, and then perform inverse Fourier transform to separate the pure voice signal, which is marked as the standardized voice signal Vc.

5. A method for annotating image data based on deep learning according to claim 4, characterized in that: The specific process of outputting the depth image feature Fi by constructing an image feature extraction network is as follows: Extract features from the standardized geographical image Ie through the deep residual network model, obtain the feature vector of the geographical image Ie and mark it as the image feature e; By performing a multi-layer perceptron transformation on the geographical feature Gk and then concatenating it with the image feature e, a deep image feature Fi with a dimension of R is output. d1 ​ 6. The method for annotating image data based on deep learning according to claim 5, wherein: The specific process of outputting the depth semantic feature Fs by constructing a voice feature extraction network is as follows: Perform ASR automatic speech recognition on the standardized voice signal Vc to convert the voice signal into text mode; Then combine with the geographical feature Gk, substitute it into the BERT-GEO model, and output the geographical semantic feature Sg; Input the geographical semantic feature Sg into the Transformer architecture to output the deep semantic feature Fs with a dimension of R, so as to extract the semantic information related to the geographical location in the text of the speech signal. d2 ​ 7. A method for image data annotation based on deep learning according to claim 6, characterized in that: The specific process of constructing an asymmetric fusion model to obtain the fusion feature Ff of the scene is as follows: Concatenate the input depth image feature Fi, depth semantic feature Fs and geographical feature Gk to obtain a feature vector set; Mark the dimensions of the depth image feature Fi, depth semantic feature Fs and geographical feature Gk as d1, d2 and d3 respectively, then the dimension of the feature vector set is (d1 + d2 + d3); Set the dimension of the weight matrix Wa as m×(d1 + d2 + d3). Through linear transformation of the concatenated feature vector set, perform matrix multiplication of the weight matrix Wa and the feature vector set to perform weighted combination of features of different dimensions, obtain a new vector with dimension m, then input the new vector into the Softmax function to obtain the attention weight of the feature, and mark it as the feature weight coefficient; Then perform weighted fusion of the depth image feature Fi and depth semantic feature Fs, perform feature fusion calculation in an element-wise multiplication manner, and multiply by the corresponding feature weight coefficient to obtain the fusion feature Ff of the scene.

8. A method for annotating image data based on deep learning according to claim 7, characterized in that: The specific process of hierarchical prediction classification is as follows: Construct a hierarchical classifier to label any dynamic scene as scene category Q, set the N-level category labels for scene category Q, and label the n-level category label as Q n , and label the category label of the layer above the nth layer as Q n-1 ; Input the class label Q of the (n - 1)-th layer n-1 and the fusion feature Ff into the hierarchical classifier, and calculate the conditional probability P(Q n |Q n-1 , Ff) of the n-th layer through the Softmax activation function. Then, through the calculation of the product of the conditional probabilities of N layers, obtain the prediction probability P(Q|F) that the fusion feature F of any dynamic scene conforms to the scene class Q.

9. An image data annotation system based on deep learning, characterized in that: It includes a multi-modal acquisition unit, a preprocessing module, a feature fusion unit, a scene classification module and a visualization module. This system applies a deep learning-based image data annotation method described in any one of the above claims 1-8; The multi-modal acquisition unit is used to obtain scene information. Among them, the multi-modal acquisition unit includes an image acquisition module and a voice acquisition module; collect geographical image data and user voice data through the image acquisition module and the voice acquisition module respectively; The preprocessing module is used to perform preliminary analysis on the scene information to obtain a standardized geographical image and a standardized voice signal; The feature fusion unit is used to perform fusion analysis on the image feature and the voice feature. Among them, the feature fusion unit includes an image feature extraction network, a voice feature extraction network and an asymmetric fusion module. Extract the image feature and the voice feature through the image feature extraction network and the voice feature extraction network respectively, and output the depth image feature and the depth semantic feature; the asymmetric fusion module combines the depth image feature and the depth semantic feature to obtain the fusion feature; The scene classification module is used to perform hierarchical prediction classification on dynamic scenes and obtain the predicted probabilities of scene categories; The visualization module is used to integrate and annotate the fusion features of any dynamic scene and their predicted probabilities that match the scene categories as the scene classification results.

Images