Multi-modal cross-domain analysis method, equipment and medium

Abstract

The invention discloses a multi-modal cross-domain analysis method and device and a medium, and the method comprises the steps: carrying out the feature extraction of different modal data in a standardized data set of multi-modal data through a deep learning encoder, carrying out the cross-modal feature mapping of different modal features through a feature alignment technology, generating a cross-modal alignment feature vector, and carrying out the recognition of the cross-modal alignment feature vector; according to the cross-modal alignment feature vector, training the recurrent neural network model architecture through an adaptive learning rate algorithm, regularization and a Dropout strategy to obtain a recurrent neural network model; and finally, according to the recurrent neural network model, performing feature fusion on the cross-modal alignment feature vector to be analyzed to obtain a cross-modal fusion feature vector, and performing cross-domain analysis on the cross-modal fusion feature vector. Through the neurons circularly connected with the recurrent neural network model, the internal state can be maintained when the time sequence data is processed, so that the long-distance dependency relationship in the sequence can be captured, and the comprehensiveness of the analysis result is improved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a multimodal cross-domain analysis method, device, and medium. Background Art

[0002] With the rapid development of artificial intelligence (AI), multimodal data (such as images, text, audio, and video) has become an essential component of the information society. However, single-modal analysis methods have significant information limitations, making it difficult to fully capture the underlying connections within the data. Furthermore, their analytical results are partial and lack generalizability. Therefore, effectively integrating these data modalities from diverse sources and enabling cross-domain information understanding and analysis remains a major challenge.

[0003] Existing multimodal cross-domain fusion analysis technology mainly uses convolutional neural networks (CNN) for feature fusion. However, convolutional neural network models cannot effectively capture long-term dependencies when processing time series data such as video and voice, resulting in limited time series data processing capabilities and insufficient comprehensiveness of analysis results. Summary of the Invention

[0004] The embodiments of the present application provide a multimodal cross-domain analysis method, device, and medium for solving the problems of insufficient time series data processing capabilities and incomplete analysis results in existing multimodal cross-domain analysis.

[0005] The embodiments of this application adopt the following technical solutions:

[0006] On the one hand, an embodiment of the present application provides a multimodal cross-domain analysis method, which includes: using a deep learning encoder to extract features of different modal data in a standardized data set of multimodal data, and using feature alignment technology to perform cross-modal feature mapping on different modal features to generate a cross-modal aligned feature vector, and then based on the cross-modal aligned feature vector, the recurrent neural network model architecture is trained through an adaptive learning rate algorithm, regularization and Dropout strategy to obtain a recurrent neural network model; finally, based on the recurrent neural network model, the cross-modal aligned feature vector to be analyzed is feature fused to obtain a cross-modal fusion feature vector, so as to perform cross-domain analysis on the cross-modal fusion feature vector.

[0007] In one example, a recurrent neural network model architecture is trained using an adaptive learning rate algorithm, specifically including: iteratively calculating the first-order moment estimation and second-order moment estimation of the gradient based on the gradient change of the cross-modal aligned feature vector, iteratively updating the learning rate of the recurrent neural network model based on the first-order moment estimation and the second-order moment estimation, and stopping the iteration when the preset maximum number of iterations is reached to determine the learning rate of the recurrent neural network model.

[0008] In one example, the recurrent neural network model architecture is trained through regularization, specifically including: introducing a regularization term into the loss function of the recurrent neural network model, the regularization term including L1 and / or L2; according to a preset regularization coefficient, adding the regularization term to the loss function to determine the total loss function, and adjusting the complexity of the recurrent neural network model parameters according to the total loss function.

[0009] In one example, the recurrent neural network model architecture is trained using the Dropout strategy, specifically including: in the preliminary training stage, in the hidden layer of the recurrent neural network model, some neurons and connections between some neurons are randomly masked by a preset dropout rate; in the forward propagation stage of the model training, Bernoulli sampling is performed on the output vector of each hidden layer to generate a mask matrix; the mask matrix is multiplied by each output element of the hidden layer to obtain the masked output; in the model verification stage, the Dropout operation is canceled, and the outputs of all neurons are scaled by a preset ratio to keep the expected values consistent.

[0010] In one example, before training the recurrent neural network model architecture through an adaptive learning rate algorithm, regularization, and Dropout strategy based on the cross-modal alignment feature vector, the method also includes: in the recurrent neural network model, weighted summing the feature vectors of different modalities in the cross-modal alignment feature vector to obtain an early fusion feature vector, modeling and predicting the feature vectors of different modalities respectively, and fusing the prediction results of each modality to obtain a late fusion feature vector, merging the early fusion feature vector with the late fusion feature vector to train the model to obtain a cross-modal fusion feature vector.

[0011] In one example, before performing cross-domain analysis on the cross-modal fusion feature vector, the method further includes: obtaining the target domain to be analyzed of the cross-modal fusion feature vector, judging whether the target domain fits the source domain based on the source domain of the cross-modal fusion feature vector, and if not, determining the pre-trained parameters of the cross-domain analysis model corresponding to the source domain, migrating the pre-trained parameters of the source domain to the target domain, and adjusting them according to the preset task requirements.

[0012] In one example, before performing weighted summation on the different modal feature vectors in the cross-modal alignment feature vector to obtain the early fusion feature vector, the method further includes: expanding the different modal feature vectors through data enhancement technology to obtain diversified cross-modal alignment feature vectors, and the expansion includes image magnification, synonym replacement, and video frequency modulation.

[0013] In one example, after performing cross-domain analysis on the cross-modal fusion feature vector, the method further includes: denoising and smoothing the analysis results to obtain standard analysis results, obtaining analysis results of historical related types, and comparing the standard analysis results with the analysis results of historical related types according to a preset similarity. If the comparison similarity is greater than the preset similarity, the standard analysis result is marked as reasonable. When the standard analysis result is reasonable, the standard analysis result is stored in a database and sent to the client for display.

[0014] On the other hand, an embodiment of the present application provides a multimodal cross-domain analysis device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any one of the above-mentioned multimodal cross-domain analysis methods.

[0015] On the other hand, an embodiment of the present application provides a multimodal cross-domain analysis non-volatile computer storage medium storing computer executable instructions, which can execute any of the above-mentioned multimodal cross-domain analysis methods.

[0016] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0017] This application uses cyclically connected neurons in a recurrent neural network model to maintain internal states when processing time series data, thereby capturing long-distance dependencies in the sequence and improving the comprehensiveness of the analysis results. Furthermore, by training the recurrent neural network model through an adaptive learning algorithm, the learning rate can be automatically adjusted based on the first-order moment estimate and second-order moment estimate of the gradient, accelerating convergence and reducing the risk of overfitting. L1 / L2 regularization terms are added during model training to limit the model capacity by penalizing the complexity of the model parameters. The Dropout strategy reduces the co-adaptability between neurons by randomly discarding some neurons and their connections during training, helping to prevent overfitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solution of the present application, some embodiments of the present application will be described in detail below with reference to the accompanying drawings, in which:

[0019] Figure 1 A schematic diagram of a multimodal cross-domain analysis method provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of the structure of a multimodal cross-domain analysis device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0022] Some embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0023] Figure 1 This is a flow chart of a multimodal cross-domain analysis method provided in an embodiment of the present application. This method can be applied to different business areas. Certain input parameters or intermediate results in this process allow for manual intervention and adjustment to help improve accuracy.

[0024] The analysis method involved in the embodiments of the present application can be implemented by a terminal device or a server, and the present application does not impose any special restrictions on this. For ease of understanding and description, the following embodiments are described in detail using a controller as an example.

[0025] It is necessary to explain in advance that:

[0026] Regularization is an important technique in machine learning used to prevent model overfitting and improve generalization ability. The core idea is to add additional "penalty terms" to the loss function to constrain model complexity. Among them, the L1 regularization term is to add the sum of the absolute values of the weight parameters as a penalty term to the loss function, and the L2 regularization term is to add the sum of the squares of the weight parameters as a penalty term to the loss function.

[0027] Dropout is a regularization technique for neural networks that randomly "turns off" (i.e. temporarily removes) neurons and their connections with a certain probability during training, forcing the model to learn more robust features and reduce the dependence between neurons.

[0028] Feature alignment technology is a method used in machine learning and deep learning to address the differences between different feature spaces, different distributions or different modal data. Its core goal is to map features from different sources (such as different data sets, different modalities, and different fields) to a unified shared space, making them comparable and consistent in this space, thereby improving the performance of the model in cross-domain and cross-modal tasks.

[0029] Based on this, Figure 1 The process in may include the following steps:

[0030] S101: Using a deep learning encoder, extract features from different modal data in a standardized dataset of multimodal data, and perform cross-modal feature mapping on different modal features through feature alignment technology to generate a cross-modal aligned feature vector.

[0031] It should be noted that in some embodiments of the present application, before performing feature extraction through the deep learning encoder, it is necessary to first collect data sets from different modalities (such as images, text, audio, etc.), and these data sets should contain sufficient samples to support model training and verification;

[0032] The collected multimodal data is then cleaned, including removing noise, outliers, and duplicates, to ensure data quality and consistency. For supervised learning tasks, the data needs to be labeled, meaning each sample is assigned one or more labels. This labeling process can be assisted by automated tools. Finally, the data from different modalities is converted into a format suitable for model processing. For example, image data can be converted into a pixel matrix, and text data can be converted into word or sentence vectors. Furthermore, the dataset can be categorized as needed, including training, validation, and test sets, which can be divided according to user needs.

[0033] Furthermore, the encoder structure in deep learning technology is used to extract the key features of each modal data. For image feature extraction, a deep convolutional neural network (CNN) is used as an image feature extractor, such as ResNet, VGG, etc.; for text feature extraction, a recurrent neural network (RNN) or its variants (LSTM, GRU) and a Transformer model are used for text feature extraction; audio data can be extracted using a deep neural network (DNN) combined with convolutional layers and recurrent layers. For feature extraction of other modalities, the user can choose the appropriate feature extraction method based on the specific modality, which is decided by the user and will not be described in detail here.

[0034] Furthermore, after feature extraction, cross-modal feature mapping is achieved through feature alignment technology to generate cross-modal aligned feature vectors, so that data from different modalities can be compared and analyzed on the same dimension.

[0035] S102: According to the cross-modal alignment feature vector, the recurrent neural network model architecture is trained through an adaptive learning rate algorithm, regularization and Dropout strategy to obtain a recurrent neural network model.

[0036] It should be noted that in some embodiments of this application, before the training model obtains the cross-modal fusion feature vector, the different modal feature vectors are first expanded through data augmentation technology to obtain diversified cross-modal alignment feature vectors. This not only improves the robustness of the model but also increases the diversity of the training data. For image data, operations such as rotation, scaling, cropping, and flipping can be performed; for text data, operations such as synonym replacement and sentence reorganization can be performed.

[0037] Furthermore, in the recurrent neural network model, the feature vectors of different modalities in the cross-modal alignment feature vector are weighted and summed to obtain an early fusion feature vector; then the feature vectors of different modalities are modeled and predicted separately, and the prediction results of each modality are fused to obtain a late fusion feature vector; the early fusion feature vector is merged with the late fusion feature vector to train the model to obtain a cross-modal fusion feature vector.

[0038] Furthermore, in the process of training the model, an adaptive learning rate optimization algorithm (such as Adam, RMSprop) is used to iteratively calculate the first-order moment estimate and the second-order moment estimate of the gradient according to the gradient change of the cross-modal alignment feature vector; then, the learning rate of the recurrent neural network model is iteratively updated according to the first-order moment estimate and the second-order moment estimate; the number of learning rate iterations of the model is controlled by a preset maximum number of iterations, and when the preset maximum number of iterations is reached, the iteration is stopped to determine the learning rate of the recurrent neural network model.

[0039] At the same time, during the model training process, L1 regularization terms and / or L2 regularization terms are introduced into the loss function of the recurrent neural network model to limit the model complexity by penalizing the absolute value or square value of the model parameters. Then, the weight of the regularization term is dynamically adjusted according to the preset regularization coefficient λ, where the value range of λ is [0.001, 0.1], and the value of λ is optimized through cross-validation. During the model training process, the regularization term is added to the original loss function to form the total loss function, which is formulated as follows:

[0040] L total =L 原始 +λ( i ∑|w i ∣+ j ∑w j2 );

[0041] Among them, wi is the model weight parameter, and the L1 regularization term and the L2 regularization term can be used alone or in combination.

[0042] Furthermore, during the training of the model, Dropout reduces the co-adaptability between neurons by randomly discarding some neurons and their connections during the training process. In the hidden layer of the recurrent neural network model, some neurons and their connections are randomly masked according to a preset dropout rate p, where the value of p ranges from [0.2 to 0.5]. Then, in the forward propagation phase of the model training, Bernoulli sampling is performed on the output vector of each hidden layer to generate a mask matrix, which is formulated as follows:

[0043] Mask~Bernoulli(1-p)

[0044] Finally, the mask matrix is multiplied element-by-element by the hidden layer output to obtain the masked output. At the same time, during the model verification and testing phases, the Dropout operation is canceled and the outputs of all neurons are scaled by a ratio of 1-p to keep the expected value the same.

[0045] S103: According to the recurrent neural network model, feature fusion is performed on the cross-modal aligned feature vector to be analyzed to obtain a cross-modal fusion feature vector, so as to perform cross-domain analysis on the cross-modal fusion feature vector.

[0046] It should be noted that in some embodiments of this application, a cross-domain analysis model is designed and implemented based on the fused multimodal feature vectors. This model can comprehensively analyze and predict multimodal data across different domains or datasets. For example, in sentiment analysis tasks, image and text data can be fused to determine emotional tendencies; in recommendation systems, users' multimodal behavioral data can be used to make personalized recommendations.

[0047] It should be noted that before cross-domain analysis of the cross-modal fusion feature vector, it will be optimized through transfer learning. First, the target domain to be analyzed of the cross-modal fusion feature vector is obtained, and then the source domain of the cross-modal fusion feature vector is judged whether the target domain and the source domain fit. If not, the pre-trained parameters of the cross-domain analysis model corresponding to the source domain are determined; the pre-trained parameters of the source domain are migrated to the target domain and adjusted according to the preset task requirements.

[0048] To further improve the generalization capabilities of cross-domain analysis models, a multi-task learning framework can be employed. During training, the objective functions of multiple related tasks are optimized, enabling the model to learn more general and robust feature representations. These tasks can include conversion between different modalities, classification, regression, and more.

[0049] Finally, based on specific task requirements, the fused results are optimized and processed to output data analysis results with practical application value. The specific method includes denoising and smoothing the analysis results to obtain standard analysis results, then obtaining historical analysis results of related types. Based on a preset similarity (90%), the standard analysis results are compared with the historical analysis results of related types. If the comparison similarity is greater than the preset similarity, the standard analysis results are marked as reasonable. When the standard analysis results are reasonable, they are stored in the database and sent to the client for display.

[0050] It should be noted that although the embodiments of this application are based on Figure 1 Steps S101 to S103 are described in sequence, but this does not mean that steps S101 to S103 must be executed in a strict order. Figure 1 The order shown in FIG1 is to introduce and explain step S101 to step S103 in order to facilitate those skilled in the art to understand the technical solution of the embodiment of the present application. In other words, in the embodiment of the present application, the order between step S101 to step S103 can be appropriately adjusted according to actual needs.

[0051] pass Figure 1 The present application uses a recurrent neural network model to connect neurons in a loop, which can maintain internal states when processing time series data, thereby capturing long-distance dependencies in the sequence and improving the comprehensiveness of the analysis results. In addition, the recurrent neural network model is trained by an adaptive learning algorithm, which can automatically adjust the learning rate based on the first-order moment estimate and second-order moment estimate of the gradient, accelerate convergence and reduce the risk of overfitting; L1 / L2 regularization terms are added during the model training process to limit the capacity of the model by penalizing the complexity of the model parameters, and the Dropout strategy reduces the co-adaptability between neurons by randomly discarding some neurons and their connections during training, which helps to prevent overfitting.

[0052] Figure 2 A schematic diagram of the structure of a multimodal cross-domain analysis device provided in an embodiment of the present application includes:

[0053] at least one processor; and,

[0054] a memory communicatively connected to at least one processor; wherein,

[0055] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can perform any one of the multimodal cross-domain analysis methods mentioned above.

[0056] Some embodiments of the present application provide a multimodal cross-domain analysis non-volatile computer storage medium storing computer-executable instructions capable of executing any one of the above-mentioned multimodal cross-domain analysis methods.

[0057] The various embodiments in this application are described in a progressive manner. Similar portions between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the device and medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant portions, refer to the descriptions of the method embodiments.

[0058] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects to their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0059] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0060] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0061] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0062] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0063] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0064] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM), and non-volatile memory such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0065] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0066] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0067] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the technical principles of the present application should fall within the scope of protection of the present application.

Claims

1. A multimodal cross-domain analysis method, characterized in that: The method comprises: Through deep learning encoder, feature extraction is performed on the data of different modalities in the standardized dataset of multimodal data, and cross-modal feature mapping of different modal features is performed through feature alignment technology to generate cross-modal aligned feature vectors; According to the cross-modal aligned feature vector, a recurrent neural network model architecture is trained using an adaptive learning rate algorithm, regularization, and a dropout strategy to obtain a recurrent neural network model; According to the recurrent neural network model, feature fusion is performed on the cross-modal aligned feature vector to be analyzed to obtain a cross-modal fusion feature vector, so as to perform cross-domain analysis on the cross-modal fusion feature vector.

2. The method according to claim 1, characterized in that The recurrent neural network model architecture is trained using an adaptive learning rate algorithm, specifically: According to the gradient change of the cross-modal aligned feature vector, the first-order moment estimate and the second-order moment estimate of the gradient are iteratively calculated; Iteratively updating the learning rate of the recurrent neural network model based on the first-order moment estimate and the second-order moment estimate; When the preset maximum number of iterations is reached, the iteration is stopped and the learning rate of the recurrent neural network model is determined.

3. The method according to claim 1, characterized in that Regularization is used to train the recurrent neural network model architecture, specifically: Introducing a regularization term into the loss function of the recurrent neural network model; the regularization term includes L1 and / or L2; According to a preset regularization coefficient, the regularization term is added to the loss function to determine a total loss function; According to the total loss function, the complexity of the recurrent neural network model parameters is adjusted.

4. The method according to claim 1, wherein The recurrent neural network model architecture is trained using the Dropout strategy, which includes: In the initial training phase, in the hidden layer of the recurrent neural network model, some neurons and connections between neurons are randomly blocked by a preset dropout rate; During the forward propagation phase of model training, Bernoulli sampling is performed on the output vector of each hidden layer to generate a mask matrix; Multiplying the mask matrix by each output element of the hidden layer to obtain a masked output; During the model validation phase, the Dropout operation is canceled and the outputs of all neurons are scaled according to a preset ratio to keep the expected values consistent.

5. The method according to claim 1, wherein Before training the recurrent neural network model architecture using an adaptive learning rate algorithm, regularization, and a dropout strategy based on the cross-modal aligned feature vector, the method further includes: In the recurrent neural network model, performing weighted summation on the feature vectors of different modalities in the cross-modal alignment feature vector to obtain an early fusion feature vector; Model and predict the eigenvectors of different modalities separately, and fuse the prediction results of each modality to obtain the late fusion eigenvector; The early fusion feature vector is merged with the late fusion feature vector to train a model to obtain a cross-modal fusion feature vector.

6. The method according to claim 1, characterized in that Before performing cross-domain analysis on the cross-modal fusion feature vector, the method further includes: Obtaining a target domain to be analyzed of the cross-modal fusion feature vector; Determining whether the target domain fits the source domain according to the source domain of the cross-modal fusion feature vector; If not, determining pre-training parameters of the cross-domain analysis model corresponding to the source domain; The pre-trained parameters of the source domain are transferred to the target domain and adjusted according to preset task requirements.

7. The method according to claim 5, characterized in that Before performing weighted summation of different modality feature vectors in the cross-modal alignment feature vector to obtain an early fusion feature vector, the method further includes: The different modal feature vectors are expanded by data enhancement technology to obtain diversified cross-modal aligned feature vectors; the expansion includes image magnification, synonym replacement, and video frequency modulation.

8. The method according to claim 1, characterized in that After performing cross-domain analysis on the cross-modal fusion feature vector, the method further includes: De-noise and smooth the analysis results to obtain standard analysis results; Obtaining historical analysis results of related types, and comparing the standard analysis results with the historical analysis results of related types according to a preset similarity; If the comparison similarity is greater than the preset similarity, the standard analysis result is marked as reasonable; When the standard analysis result is reasonable, the standard analysis result is stored in a database, and the standard analysis result is sent to a client for display.

9. A multimodal cross-domain analysis device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimodal cross-domain analysis method described in any one of claims 1 to 8.

10. A multimodal cross-domain analysis storage medium storing computer-executable instructions, characterized in that: The computer-executable instructions are capable of executing a multimodal cross-domain analysis method as described in any one of claims 1-8 above.

Images