Multi-modal cross-domain conversion method, system and device and storage medium
The cross-domain modal conversion is processed through the autoencoder network and multiple alignment technologies, which solves the problems of training data acquisition difficulties and pattern crashes in the prior art, and achieves cross-domain conversion with high accuracy and generation quality.
Patent Information
- Application Number
- CN202510016996.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-06
- Publication Date
- 2025-05-02
AI Technical Summary
Existing cross-domain modal conversion technology is costly and impractical when acquiring large amounts of paired training data, and the domain alignment method cannot effectively handle mixed or mismatched clusters, resulting in pattern crashes and confusion, affecting the accuracy and generation quality of the conversion.
The source domain and target domain are trained using an autoencoder network to obtain the characteristic specification data of the hidden space, and aligned the data space through center of gravity translation, semi-discrete optimal transmission algorithm, constraint harmony mapping technology and graph constraint geometric registration method to establish a cross-domain conversion relationship.
It achieves the accuracy and generation quality of cross-domain transformation, avoids pattern crashes and mixing, and provides one-to-one and continuous mappings, suitable for conversion between multimodals.
Smart Images
Figure CN119919280A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of modal conversion and relates to a multi-modal cross-domain conversion method, system, device and storage medium. Background Art
[0002] Cross-domain generative models based on encoder and decoder AI architectures have attracted much attention in modality transfer, and domain alignment between their latent spaces is crucial for generation accuracy. Cross-domain modality transfer technology has shown broad application prospects in multiple fields, including image conversion, natural language processing, and signal conversion. However, supervised training of translation models is not very practical due to the difficulty and high cost of obtaining large amounts of paired training data in many conversion tasks. In addition, domain alignment methods usually deal with the initial distribution directly, while mismatched or mixed clusters can lead to mode collapse and confusion problems in the decoder, resulting in low accuracy and generation quality of cross-domain conversion between modalities. Summary of the invention
[0003] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a multi-modal cross-domain conversion method, system, device and storage medium, which can improve the accuracy and generation quality of modal cross-domain conversion.
[0004] In order to achieve the above object, the present invention adopts the following technical solutions: A multimodal cross-domain conversion method includes the following processes: S1: The original modality and target modality with labels are used as the source domain and target domain respectively. The labels of the source domain and the target domain correspond one to one. The source domain and the target domain are trained using the autoencoder network to obtain the feature norm representation data of their respective latent spaces. S2: According to the difference in labels in the feature specification representation data, the center of gravity is shifted to align the data space positions in the source domain and the target domain; S3: A semi-discrete optimal transfer algorithm is used to eliminate the spatial gap between labels in the source and target domains; S4: Constrained reconciliation mapping technology is used to map the latent spaces of the source domain and the target domain to their respective regularization parameter domains; S5: Use the graph-constrained geometric registration method to align the regularization parameter domains of the source domain and the target domain to obtain the cross-domain transformation relationship between the source domain and the target domain; S6: Apply the cross-domain conversion relationship to perform cross-domain conversion on the modality to be converted to obtain the target modality.
[0005] Preferably, in S1, the latent spaces of the source domain and the target domain are each reduced to two dimensions by using the distributed random neighbor embedding algorithm t-SNE to obtain feature normative representation data.
[0006] Preferably, the specific process of S2 includes: S2.1: Remove abnormal data points caused by the training error of the autoencoder network; S2.2: Calculate the centroid coordinates of the source domain and the target domain under different labels in their respective latent spaces; S2.3: Perform center of gravity shift.
[0007] Furthermore, the center of gravity is shifted in any of the following ways: one is that the center of gravity of the target domain data remains unchanged, and the center of gravity of the source domain data is shifted; the other is that the centers of gravity of both the source domain and the target domain data are shifted.
[0008] Preferably, the specific process of S3 includes: S3.1: Collect data points of uniform size in the source and target domains; S3.2: Use a semi-discrete optimal transmission algorithm to perform area-preserving mapping between data points and latent space data points so that the measurements of the two remain consistent.
[0009] Preferably, the specific process of S4 includes: S4.1: generate triangulated data points; S4.2: Determine the segmentation curve and segmentation nodes between the segmentation labels according to the structural position information of the labels; S4.3: Through the constraints of boundary nodes and segmentation nodes, the segmentation curve is straightened using the harmonic mapping method.
[0010] Preferably, the specific process of S5 includes: according to the correspondence relationship between the segmentation lines in the source domain and the target domain, using a graph-constrained geometric registration method to distort the triangular mesh of the source domain to be similar to the shape of the target domain.
[0011] A multimodal cross-domain conversion system, comprising: Training module: used to take the original modality and target modality with labels as the source domain and target domain respectively, with the labels of the source domain and the target domain corresponding one to one, and use the autoencoder network to train the source domain and the target domain to obtain the feature norm representation data of their respective latent spaces; Center of gravity translation module: It is used to translate the center of gravity according to the difference in labels in the feature specification representation data, so as to align the spatial positions of the data in the source domain and the target domain; Spatial gap elimination module: used to eliminate the spatial gap between labels in the source domain and the target domain using a semi-discrete optimal transfer algorithm; Regularization parameter domain mapping module: used to map the latent spaces of the source domain and target domain to their respective regularization parameter domains using constraint reconciliation mapping technology; Regularization parameter domain alignment module: used to align the regularization parameter domains of the source domain and the target domain using the geometric registration method of graph constraints to obtain the cross-domain transformation relationship between the source domain and the target domain; Cross-domain conversion module: used to apply the cross-domain conversion relationship, perform cross-domain conversion on the modality to be converted, and obtain the target modality.
[0012] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the deep learning model malicious sample detection method when executing the computer program.
[0013] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the deep learning model malicious sample detection method.
[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention constructs a generative model by accurately and seamlessly aligning the latent space of the source environment space and the target environment space, transforms the latent space, and strictly satisfies the corresponding constraints between semantic clusters. In essence, it is the differential isomorphism between the transformed source domain latent space and the target domain latent space, and it is expected to be a one-to-one and continuous mapping to avoid mode collapse or mixing in modal tasks. The traditional alignment of the source domain latent space and the target domain latent space can achieve point-to-point generation of different categories, but cannot solve mode collapse and mixing and curve-to-curve generation. The present invention overcomes the inherent limitations of direct mapping of discrete sample points, liberates the generation process with the help of geometric mapping methods, avoids mode collapse and mixing, and realizes continuous generation of curves to curves. Compared with existing mapping methods, it has better or equivalent accuracy and generation quality. In addition, this method has broad application prospects in application fields such as modal processing, cross-domain recommendation systems, and unsupervised domain adaptation. Its distribution is transformed into the target domain to achieve effective representation and cross-domain conversion of data. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 It is a logic flow chart of the multi-modal cross-domain conversion method of the present invention; Figure 2 is a schematic diagram of the multimodal cross-domain conversion method of the present invention; Figure 3 Examples of data sets used in embodiments of the present invention; Figure 4 This is the result of using t-SNE to reduce the dimension of the data set after encoding it in the present invention; Figure 5 It is a schematic diagram of the result of the center of gravity translation of the present invention; Figure 6 This is a schematic diagram of the result of merging data spaces using the semi-discrete optimal transmission method of the present invention; Figure 7It is a schematic diagram of the result of mapping the latent spaces of the source domain and the target domain to their respective regularization parameter domains using the constraint reconciliation mapping technology of the present invention; Figure 8 A schematic diagram of the result of cross-domain conversion from a source domain to a target domain obtained by using geometric mapping with graph constraints of the present invention; Fig. 9 It is a schematic diagram of the conversion application of the present invention. DETAILED DESCRIPTION
[0016] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.
[0017] Generation from one modality to another requires a cross-domain generative model, which is usually implemented by a deep network. An effective strategy is to perform cross-domain latent code conversion based on an encoder-decoder architecture such as an autoencoder. For generality, this paper studies the problem of multi-class cross-domain generation. Given a dataset with multiple classes, the environment space can be embedded into the corresponding latent space using encoding and dimensionality reduction techniques (such as UMAP or distributed random neighbor embedding algorithm t-SNE), which usually project high-dimensional data into two-dimensional or three-dimensional space. For real-world datasets, the latent space always contains outliers and class mixtures. This is mainly due to the confusion and mixing of data samples or the insufficient distinguishing ability of data embedding techniques. In cross-domain generation, the source latent space and the target latent space usually have different cluster distribution structures, that is, different geometric shapes and topological structures. Therefore, accurate multi-class cross-domain generation refers to establishing a transformation between cross-domain latent spaces with hard constraints from cluster to cluster.
[0018] like Figure 1 and Figure 2 As shown, the multimodal cross-domain conversion method integrating optimal transmission theory and geometric mapping technology described in the present invention builds a generation model by accurately and seamlessly aligning the latent space of the source environment space and the target environment space, converts the latent space, and strictly satisfies the corresponding constraints between semantic clusters. It includes the following steps: S1: The labeled original modality is used as the source domain, and the labeled target modality is used as the target domain. The labels of the source domain and the target domain correspond one to one. The source domain and the target domain are trained using an autoencoder network to obtain the feature specification representation data of the latent space.
[0019] Although the autoencoders corresponding to different modalities are different, the feature specification representation data is consistent. Therefore, the invention is applicable to the conversion between multiple modalities. Common modalities include vision (image, video), hearing (audio, speech), language (text, natural language), touch, etc.
[0020] In this embodiment, the labels of the original image are different uppercase numbers, and the labels of the target image are lowercase numbers. The uppercase number labels and lowercase number labels with the same meaning correspond one to one, that is, the uppercase number "one" corresponds to the lowercase number "1", and so on.
[0021] S1.1: Perform image preprocessing on the source domain and the target domain, including noise removal, geometric transformation, and image enhancement.
[0022] S1.2: The source domain and target domain are trained and tested for autoencoders respectively, and the latent space of each model is reduced to two dimensions through the distributed random neighbor embedding algorithm t-SNE to obtain feature normative representation data.
[0023] S2: Based on the above feature specification representation data, the center of gravity is shifted according to the different labels so that the spatial positions of the data in the source domain and the target domain are aligned.
[0024] S2.1: Remove abnormal data points caused by the training and testing errors of the autoencoder model.
[0025] S2.2: Calculate the centroid coordinates of the source domain and the target domain under different labels of their respective latent spaces. In this embodiment, different labels are different numbers.
[0026] S2.3: Perform center of gravity translation, which can be divided into two ways: one is to keep the center of gravity of the target domain data unchanged and translate the center of gravity of the source domain data, and the other is to translate the center of gravity of both the source domain and the target domain data.
[0027] S3: To eliminate the spatial gaps between tags in the data space, semi-discrete optimal transmission is used for merging.
[0028] S3.1: Collect data points of uniform size in the source and target domains.
[0029] S3.2: Use the semi-discrete optimal transmission algorithm to perform area-preserving mapping between the data points in S3.1 and the data points in the latent space so that their measurements are consistent.
[0030] S4: In order to reduce the difficulty of subsequent cross-domain feature space registration, the constrained harmonic mapping technique is used to map the latent spaces of the source domain and the target domain to their respective regularization parameter domains.
[0031] S4.1: Triangulate the sampled data points in S3.
[0032] S4.2: Determine the segmentation curve and segmentation nodes between the divided labels according to the structural position information of the label data.
[0033] S4.3: Through the constraints of boundary nodes and segmentation nodes, the segmentation curve is straightened using the harmonic mapping method.
[0034] S5: Use the graph-constrained geometric registration method to align the regularization parameter domains of the source domain and the target domain to obtain the cross-domain transformation relationship between the source domain and the target domain; the specific approach is to use the graph-constrained geometric registration method to distort the triangular mesh of the source domain to be similar to the shape of the target domain based on the correspondence between the segmentation lines in the source domain and the target domain.
[0035] S6: Using the cross-domain conversion relationship, the modality to be converted is converted across domains to obtain the target modality.
[0036] The following is a detailed introduction to the multi-modal cross-domain conversion method that integrates the optimal transmission theory and geometric mapping technology using an actual image example: Step 1: Download the handwritten digit and Chinese character image set from the website and perform image preprocessing operations. Figure 3 shown.
[0037] Step 2: Use the autoencoder to perform latent space feature extraction on the image dataset Extraction, where the encoder is , the decoder is , the results after dimensionality reduction using t-SNE are as follows Figure 4 shown.
[0038] Step 3: Shift the center of gravity of the two datasets and Operation to align it with the spatial position of the label, such as Figure 5 shown.
[0039] Step 4: In order to eliminate the spatial gap between the tag pairs in the two data spaces, a semi-discrete optimal transmission method is used and To merge, such as Figure 6 shown.
[0040] Assume that the region defined in Euclidean space , on which there is a measure is defined as uniformly distributed and continuous, and a measure is defined as is the feature embedding distribution extracted by the encoder. is a discrete Dirac measure consisting of several data points:
[0041]
[0042] and The following are guaranteed:
[0043] This work constructs a transmission map T , so that Can be mapped to , the transmission mapping that minimizes the total cost:
[0044] The above will be a plane Decompose into m Regions: , Make each area . The transmission loss is:
[0045] The optimal transport map is the gradient of the Brenier potential. At regular points, the Brenier potential is continuous and differentiable, and its gradient is also continuous. At singular points, the Brenier potential is continuous but not differentiable, so the optimal transport map at that point is discontinuous. The Brenier potential is defined as:
[0046] Divide it into the closest Then by adjusting the height , so that The projected area value and Solve for the height , the convex energy function was defined by previous studies as:
[0047] This function has a unique extreme point. Taking partial derivatives, we get:
[0048] By using the gradient descent method, we find the point where the derivative is 0 and get the optimal height of the problem. .in addition, The Monte Carlo method is used to sample. Specifically, it is uniformly obtained from Medium Sampling N points, and then through Get the area value.
[0049] Step 5: In order to reduce the difficulty of subsequent cross-domain feature space registration, the latent space of the source domain and the target domain is respectively used by the constrained harmonic mapping technology. and Mapped to their respective regularization parameter domains, such as Figure 7 shown.
[0050] Construct triangulated network from sampling points , where the cluster boundary is constructed as a curve graph . To this end, this work transforms the curve graph into a straight line graph through intrinsic harmonic mapping. The intrinsic harmonic mapping is as follows:
[0051] in is the edge weight. In this work, the vertex The weights are also different. (1) If Not in picture G On, use The mean coordinates between the neighboring points replace the weights; (2) If For the picture G Inside the curve edge, adaptive blending weights are used: ,
[0052] (3) If For the picture G At the upper node, the ring mean theorem is used to calculate the adaptive harmonic weights.
[0053] Step 6. Geometric mapping by using graph constraints , obtain the cross-domain transformation between the source domain and the target domain, such as Figure 8 shown.
[0054] Based on the graph obtained in step 6 G The structure is explained with the help of a graph-constrained geometric registration process: by determining the locations of source and target graph nodes, using interpolation to assign source locations to target locations, and setting the combinations of vertices on the graph edges using only adjacent edges on the graph, the resulting mapping is called a differential isomorphism.
[0055] Through the mapping relationship of all transformations (cross-domain transformation relationship), the final alignment from source space to target space is established .
[0056] Step 7: Apply the transformed mapping relationship to perform cross-domain transformation on the modality to be transformed to obtain the target modality, such as Figure 8 As shown (left picture is input, right picture is output).
[0057] Step 8: Use translation accuracy and FID (Fréchet Inception Distance) to evaluate the quality of the translation results, using the current representative translation model as a comparison benchmark. The present invention has the highest accuracy of 94.5% while achieving comparable FID, 18.36% higher than the second place.
[0058] Experimental results on grayscale and color images show that the proposed method performs well in terms of efficiency, effectiveness, and applicability, and exhibits higher accuracy than existing models.
[0059] In addition to the conversion of upper and lower case numbers in the above examples, the method of the present invention can also be applied to the conversion of upper and lower case letters, as well as more complex images, such as medical image conversion (MRI image to CT image), image background conversion (such as changing summer scenery to winter snow scene), image translation conversion (such as Chinese characters and numbers to Arabic numbers), object category conversion (such as cat to leopard) and augmented reality applications (such as virtual makeup), etc.
[0060] The present invention not only supports image conversion, but also covers multiple fields such as natural language processing and signal conversion. For example, in the field of natural language processing, this study can realize machine translation, such as translating Chinese vocabulary text into vocabulary text in other languages. In addition, in the field of signal conversion, this study also supports speech translation, such as converting Chinese speech into dialect speech. At the same time, this study can also realize image-to-text conversion, such as converting an image into text describing its content; text-to-speech conversion, such as a text-to-speech synthesis system that converts text into a speech signal; and speech-to-text conversion, such as an automatic speech recognition system that converts a speech signal into text. The following is an apparatus embodiment of the present invention, which can be used to execute a method embodiment of the present invention. For details not omitted in the apparatus embodiment, please refer to the method embodiment of the present invention.
[0061] In another embodiment of the present invention, a deep learning model malicious sample detection system is provided, which can be used to implement the above-mentioned deep learning model malicious sample detection method. Specifically, the deep learning model malicious sample detection system includes a training module, a center of gravity translation module, a spatial gap elimination module, a regular parameter domain mapping module, a regular parameter domain alignment module and a cross-domain conversion module.
[0062] Among them, the training module is used to take the labeled original modality and target modality as the source domain and target domain respectively, and the labels of the source domain and the target domain correspond one to one. The source domain and the target domain are trained using the autoencoder network to obtain the feature specification representation data of their respective latent spaces.
[0063] The center of gravity translation module is used to perform center of gravity translation according to the difference in labels in the feature specification representation data, so as to align the spatial positions of the data in the source domain and the target domain.
[0064] The spatial gap elimination module is used to eliminate the spatial gap between labels in the source and target domains using a semi-discrete optimal transfer algorithm.
[0065] The regularization parameter domain mapping module is used to map the latent spaces of the source domain and the target domain to their respective regularization parameter domains using the constraint reconciliation mapping technique.
[0066] The regular parameter domain alignment module is used to align the regular parameter domains of the source domain and the target domain using a graph-constrained geometric registration method to obtain a cross-domain transformation relationship from the source domain to the target domain.
[0067] The cross-domain conversion module is used to apply the cross-domain conversion relationship, perform cross-domain conversion on the modality to be converted, and obtain the target modality.
[0068] In another embodiment of the present invention, a terminal device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the deep learning model malicious sample detection method, including: S1: the original modality and the target modality with labels are used as the source domain and the target domain respectively, and the labels of the source domain and the target domain correspond one to one, and the source domain and the target domain are trained using the autoencoder network to obtain The feature norms of their respective latent spaces represent the data; S2: According to the difference in labels in the feature norms, the center of gravity is shifted to align the data space positions in the source domain and the target domain; S3: A semi-discrete optimal transfer algorithm is used to eliminate the spatial gap between the labels in the source domain and the target domain; S4: Constrained reconciliation mapping technology is used to map the latent spaces of the source domain and the target domain to their respective regular parameter domains; S5: The regular parameter domains of the source domain and the target domain are aligned using a graph-constrained geometric registration method to obtain a cross-domain transformation relationship from the source domain to the target domain; S6: The cross-domain transformation relationship is applied to perform a cross-domain transformation on the modality to be transformed to obtain the target modality.
[0069] In another embodiment, the present invention further provides a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory.
[0070] The processor may load and execute one or more instructions stored in a computer-readable storage medium to implement the corresponding steps of the method for detecting malicious samples of a deep learning model in the above-mentioned embodiment; one or more instructions in the computer-readable storage medium are loaded by the processor and the following steps are executed: S1: the original modality and the target modality with labels are used as the source domain and the target domain respectively, and the labels of the source domain and the target domain correspond one to one, and the source domain and the target domain are trained using an autoencoder network to obtain feature specification representation data of their respective latent spaces; S2: according to the feature specification representation Due to the difference in labels in the data, the center of gravity is shifted to align the data space positions in the source domain and the target domain; S3: A semi-discrete optimal transfer algorithm is used to eliminate the spatial gap between the labels in the source domain and the target domain; S4: Constrained reconciliation mapping technology is used to map the latent spaces of the source domain and the target domain to their respective regular parameter domains; S5: The regular parameter domains of the source domain and the target domain are aligned using a graph-constrained geometric registration method to obtain a cross-domain conversion relationship from the source domain to the target domain; S6: The cross-domain conversion relationship is applied to perform a cross-domain conversion on the modality to be converted to obtain the target modality.
[0071] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0072] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0073] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0074] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0075] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0076] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0077] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
[0078] It should be understood that the above description is for illustration and not for limitation. Many embodiments and many applications beyond the examples provided will be apparent to those skilled in the art upon reading the above description. Therefore, the scope of this patent should not be determined with reference to the above description, but rather with reference to the preceding claims and the full scope of equivalents to which such claims are entitled. For the purpose of comprehensiveness, all articles and references, including disclosures of patent applications and publications, are incorporated herein by reference. The omission of any aspect of the subject matter disclosed herein in the preceding claims is not intended to be a waiver of such subject matter, nor should it be considered that the applicant has not considered such subject matter to be part of the disclosed inventive subject matter.
Claims
1. A multimodal cross-domain conversion method, characterized in that: The process includes: S1: The original modality and target modality with labels are used as the source domain and target domain respectively. The labels of the source domain and the target domain correspond one to one. The source domain and the target domain are trained using the autoencoder network to obtain the feature norm representation data of their respective latent spaces. S2: According to the difference in labels in the feature specification representation data, the center of gravity is shifted to align the data space positions in the source domain and the target domain; S3: A semi-discrete optimal transfer algorithm is used to eliminate the spatial gap between labels in the source and target domains; S4: Constrained reconciliation mapping technology is used to map the latent spaces of the source domain and the target domain to their respective regularization parameter domains; S5: Use the graph-constrained geometric registration method to align the regularization parameter domains of the source domain and the target domain to obtain the cross-domain transformation relationship between the source domain and the target domain; S6: Apply the cross-domain conversion relationship to perform cross-domain conversion on the modality to be converted to obtain the target modality.
2. The multimodal cross-domain conversion method according to claim 1, characterized in that: In S1, the latent spaces of the source domain and the target domain are reduced to two dimensions through the distributed random neighbor embedding algorithm t-SNE to obtain feature normative representation data.
3. The multimodal cross-domain conversion method according to claim 1, characterized in that: The specific process of S2 includes: S2.1: Remove abnormal data points caused by the training error of the autoencoder network; S2.2: Calculate the centroid coordinates of the source domain and the target domain under different labels in their respective latent spaces; S2.3: Perform center of gravity shift.
4. The multimodal cross-domain conversion method according to claim 3, characterized in that: The center of gravity is shifted in any of the following ways: one is that the center of gravity of the target domain data remains unchanged, and the center of gravity of the source domain data is shifted; the other is that the centers of gravity of both the source domain and the target domain data are shifted.
5. The multimodal cross-domain conversion method according to claim 1, characterized in that: The specific process of S3 includes: S3.1: Collect data points of uniform size in the source and target domains; S3.2: Use a semi-discrete optimal transmission algorithm to perform area-preserving mapping between data points and latent space data points so that the measurements of the two remain consistent.
6. The multimodal cross-domain conversion method according to claim 1, characterized in that: The specific process of S4 includes: S4.1: generate triangulated data points; S4.2: Determine the segmentation curve and segmentation nodes between the segmentation labels according to the structural position information of the labels; S4.3: Through the constraints of boundary nodes and segmentation nodes, the segmentation curve is straightened using the harmonic mapping method.
7. The multimodal cross-domain conversion method according to claim 1, characterized in that: The specific process of S5 includes: according to the correspondence between the segmentation lines in the source domain and the target domain, the triangular mesh of the source domain is distorted to be similar to the shape of the target domain using a graph-constrained geometric registration method.
8. A multimodal cross-domain conversion system, characterized in that: include: Training module: used to take the original modality and target modality with labels as the source domain and target domain respectively, with the labels of the source domain and the target domain corresponding one to one, and use the autoencoder network to train the source domain and the target domain to obtain the feature norm representation data of their respective latent spaces; Center of gravity translation module: It is used to translate the center of gravity according to the difference in labels in the feature specification representation data, so as to align the spatial positions of the data in the source domain and the target domain; Spatial gap elimination module: used to eliminate the spatial gap between labels in the source domain and the target domain using a semi-discrete optimal transfer algorithm; Regularization parameter domain mapping module: used to map the latent spaces of the source domain and target domain to their respective regularization parameter domains using constraint reconciliation mapping technology; Regularization parameter domain alignment module: used to align the regularization parameter domains of the source domain and the target domain using the geometric registration method of graph constraints to obtain the cross-domain transformation relationship between the source domain and the target domain; Cross-domain conversion module: used to apply the cross-domain conversion relationship, perform cross-domain conversion on the modality to be converted, and obtain the target modality.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the deep learning model malicious sample detection method as described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for detecting malicious samples using a deep learning model as described in any one of claims 1 to 7 are implemented.