Multi-modal data generation method and device and storage medium

Abstract

The invention discloses a multi-modal data generation method and device and a storage medium. The method and device are used for improving the overall training effect of a multi-modal model. Obtaining a multi-modal data set of the pre-trained multi-modal model; obtaining label information of the text data and the audio data according to the label information of the multi-modal data set; extracting a plurality of image data materials from the image data set according to the label information; inputting the image data material into a convolutional neural network model, and determining a plurality of feature regions on the image data material through a tag feature enhancement module according to the tag information; performing feature enhancement processing on each feature region according to hierarchical parameters among the tag information to generate tag features; generating a plurality of groups of normal distribution parameters for the label features through normal distribution sampling; and reconstructing the plurality of groups of normal distribution parameters and the corresponding label information through an encoder to generate multi-modal reconstructed image data.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of multimodal data, and more particularly to a method, device and storage medium for generating multimodal data. Background Art

[0002] Multimodal data refers to information that combines different types of data, such as images, text, and voice, to provide a richer and more comprehensive form of information expression. Different forms of existence or sources of information can be called a modality. Data composed of two or more modalities is called multimodal data.

[0003] This application mainly targets biological multimodal models. The training and optimization of large biological multimodal models usually require massive amounts of data in different modalities, such as text data (biological types, genera, families, etc.), image data (photos of organisms in different categories), video data (of organisms in different categories), etc.

[0004] This poses some difficulties for the training of multimodal models. Researchers often need to study specific levels and specific features, and in this case, they need to use data of specific levels and specific features to train multimodal models. Although there is already a large amount of biological sample data of different levels and features on the Internet, the amount of image data for object features is small. The existing multimodal data generation technology can only reconstruct the features of the entire image as a whole, and cannot reconstruct the image for the target features required by researchers, making the target features of the generated image data unclear, resulting in a reduction in the overall training effect of the multimodal model. Summary of the invention

[0005] The present application discloses a method, device and storage medium for generating multimodal data, which are used to improve the overall training effect of the multimodal model.

[0006] The first aspect of the present application discloses a method for generating multimodal data, comprising: Obtain a multimodal data set of a pre-trained multimodal model, where the multimodal data set includes text data and audio data that have been annotated. Obtain label information of text data and audio data according to the annotation information of the multimodal data set; Extracting a number of image data materials from the image data set according to the label information, each image data material being associated with at least one label information; The image data material is input into the convolutional neural network model, and a number of feature regions are determined on the image data material according to the label information through the label feature enhancement module, the convolutional neural network model including the label feature enhancement module, an encoder and a decoder; Perform feature enhancement processing on each feature region according to the hierarchical parameters between the label information to generate label features. The hierarchical parameters represent the feature weight parameters between the label information. Generate several sets of normal distribution parameters for label features through normal distribution sampling; Several groups of normal distribution parameters and corresponding label information are reconstructed through the encoder to generate multimodal reconstructed image data.

[0007] Optionally, the convolutional neural network model also includes a noise network; After performing feature enhancement processing on each feature region according to the hierarchical parameters between the label information to generate the label features, and before generating a plurality of groups of normal distribution parameters for the label features by normal distribution sampling, the generation method further includes: The label features are input into the noise network, preset noise is added to the label features through the noise network, and then denoising is performed through stochastic differential equation prior iterations.

[0008] Optionally, the convolutional neural network model also includes an edge enhancement network; After inputting the label feature into the noise network, adding preset noise to the label feature through the noise network, and then denoising through the stochastic differential equation prior iteration, before generating several groups of normal distribution parameters for the label feature through normal distribution sampling, the generation method also includes: Generate a number of label information features for the image data material according to the label information and the image database; Input the label features, label information features and layer parameters into the enhanced network; Selective enhancement processing is performed according to label features, label information features and priority parameters to generate edge-enhanced label features.

[0009] Optionally, the convolutional neural network model also includes a compensation network; After performing selective enhancement processing according to the label features, label information features and priority parameters to generate edge-enhanced label features, and before generating several groups of normal distribution parameters for the label features by normal distribution sampling, the generation method further includes: Generate a level compensation value according to the feature area and level parameters of different feature areas in the current label feature; The grayscale compensation is performed on the feature area corresponding to the label feature according to the level compensation value.

[0010] Optionally, feature enhancement processing is performed on each feature region according to the hierarchical parameters between the label information to generate label features, including: Perform attention calculation on the pixel points of each feature area in the image data material to generate attention value data; Perform hierarchical analysis on label information to generate hierarchical parameters; Perform attention adjustment processing on the attention value data according to the feature area and level parameters; The attention value data and several feature areas of the image data material are multiplied by channel correspondence to generate label features.

[0011] Optionally, after performing feature enhancement processing on each feature region according to the hierarchical parameters between the label information to generate the label features, and before generating a plurality of groups of normal distribution parameters for the label features by normal distribution sampling, the generation method further includes: Perform feature encoding and dimensionality reduction on label features.

[0012] Optionally, after reconstructing a plurality of groups of normal distribution parameters and corresponding label information through an encoder to generate multimodal reconstructed image data, the generating method further includes: The image data is annotated and saved in a multimodal dataset; The pre-trained multimodal model is trained using the multimodal dataset.

[0013] The second aspect of the present application discloses a device for generating multimodal data, comprising: A first acquisition unit is used to acquire a multimodal data set of a pre-trained multimodal model, where the multimodal data set includes text data and audio data that have been annotated; A second acquisition unit, used to acquire label information of text data and audio data according to the annotation information of the multimodal data set; An extraction unit, used to extract a plurality of image data materials from the image data set according to the label information, each image data material being associated with at least one label information; A determination unit, used for inputting the image data material into the convolutional neural network model, and determining a plurality of feature regions on the image data material according to the label information through the label feature enhancement module, the convolutional neural network model including the label feature enhancement module, an encoder and a decoder; A first generating unit is used to perform feature enhancement processing on each feature area according to a hierarchical parameter between the label information to generate a label feature, wherein the hierarchical parameter represents a feature weight parameter between the label information, and the hierarchical parameter represents a feature weight parameter between the label information; A second generating unit is used to generate a plurality of groups of normal distribution parameters for the label features by normal distribution sampling; The third generating unit is used to reconstruct a plurality of groups of normal distribution parameters and corresponding label information through an encoder to generate multi-modal reconstructed image data.

[0014] Optionally, the convolutional neural network model also includes a noise network; After the first generating unit and before the second generating unit, the generating device further includes: The noise processing unit is used to input the label features into the noise network, add preset noise to the label features through the noise network, and then denoise through the stochastic differential equation prior iteration.

[0015] Optionally, the convolutional neural network model also includes an edge enhancement network; After the noise processing unit and before the second generating unit, the generating device further includes: A fourth generating unit, used to generate a plurality of label information features for the image data material according to the label information and the image database; An input unit, used to input label features, label information features and layer parameters into the enhanced network; The fifth generating unit is used to perform selective enhancement processing according to the label feature, the label information feature and the priority parameter to generate an edge-enhanced label feature.

[0016] Optionally, the convolutional neural network model also includes a compensation network; After the fifth generating unit and before the second generating unit, the generating device further includes: A sixth generating unit, used for generating a level compensation value according to the feature area and level parameters of different feature areas in the current tag feature; The compensation unit is used to perform grayscale compensation on the feature area corresponding to the label feature according to the level compensation value.

[0017] Optionally, the first generation unit includes: Perform attention calculation on the pixel points of each feature area in the image data material to generate attention value data; Perform hierarchical analysis on label information to generate hierarchical parameters; Perform attention adjustment processing on the attention value data according to the feature area and level parameters; The attention value data and several feature areas of the image data material are multiplied by channel correspondence to generate label features.

[0018] Optionally, after the first generating unit and before the second generating unit, the generating device further includes: The dimension reduction unit is used to perform feature encoding and dimension reduction processing on label features.

[0019] Optionally, after the third generating unit, the generating device further includes: A saving unit, used to save the image data into a multimodal data set after data annotation; A training unit is used to train a pre-trained multimodal model using a multimodal dataset.

[0020] A third aspect of the present application provides a device for generating multimodal data, comprising: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the first aspect and any optional generation method of the first aspect.

[0021] A fourth aspect of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the program executes the first aspect and any optional generation method of the first aspect.

[0022] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages: In the present application, a multimodal data set of a pre-trained multimodal model is first obtained, and the multimodal data set includes text data and audio data that have been annotated. The label information of the text data and the audio data is obtained according to the annotation information of the multimodal data set. A number of image data materials are extracted from the image data set according to the label information, and each image data material is associated with at least one label information. The image data material is input into a convolutional neural network model, and a number of feature areas are determined on the image data material through a label feature enhancement module and according to the label information. The convolutional neural network model includes a label feature enhancement module, an encoder and a decoder. Feature enhancement processing is performed on each feature area according to the hierarchical parameters between the label information to generate label features, and the hierarchical parameters represent the feature weight parameters between the label information. Several groups of normal distribution parameters are generated for the label features by normal distribution sampling. Several groups of normal distribution parameters and corresponding label information are reconstructed by an encoder to generate multimodal reconstructed image data.

[0023] Several image data materials are extracted from the image data set through label information. This type of image data materials extracted from the image data set are consistent with the audio data and text data in type labels, and the number of materials is also greater. At the same time, the encoder and decoder are used for reconstruction, and the feature weight parameters of different label features are determined according to the label information of the image data material. Then, different feature areas are enhanced to different degrees through the feature weight parameters. The multimodal reconstructed image data is finally generated. The image data is adapted to the text data and audio data, and there are more material samples and more simulated image data. Secondly, through the label features generated by the hierarchical parameters, the features to be studied are more prominent and more in line with the training expectations of the researchers. The multimodal model can better train the required features, which improves the overall training effect of the multimodal model. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0025] Figure 1 A schematic diagram of an embodiment of a method for generating multimodal data of the present application; Figure 2 A schematic diagram of an embodiment of a method for processing label features using a noise network in the present application; Figure 3 A schematic diagram of an embodiment of a method for enhancing label features of the present application; Figure 4 A schematic diagram of an embodiment of a method for compensating a characteristic area of ​​the present application; Figure 5 A schematic diagram of an embodiment of a method for generating hierarchical parameters for the present application; Figure 6 A schematic diagram of an embodiment of a method for processing tag features of the present application; Figure 7 A schematic diagram of an embodiment of a method for training a pre-trained multimodal model for the present application; Figure 8 A schematic diagram of an embodiment of a device for generating multimodal data of the present application; Fig. 9 A schematic diagram of another embodiment of a device for generating multimodal data of the present application. DETAILED DESCRIPTION

[0026] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0027] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0028] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0029] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0030] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0032] In the prior art, this application mainly targets biological multimodal models. The training and optimization of large biological multimodal models usually require massive amounts of data in different modalities, such as text data (biological types, biological genera, biological families, etc.), image data (photos of organisms in different categories), video data (of organisms in different categories), etc.

[0033] This poses some difficulties for the training of multimodal models. Researchers often need to study specific levels and specific features, and in this case, they need to use data of specific levels and specific features to train multimodal models. Although there is already a large amount of biological sample data of different levels and features on the Internet, the amount of image data for object features is small. The existing multimodal data generation technology can only reconstruct the features of the entire image as a whole, and cannot reconstruct the image for the target features required by researchers, making the target features of the generated image data unclear, resulting in a reduction in the overall training effect of the multimodal model.

[0034] Based on this, the present application discloses a method, device and storage medium for generating multimodal data, which are used to improve the overall training effect of the multimodal model.

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] The method of the present application can be applied to a server, a device, a terminal or other devices with logic processing capabilities, and the present application does not limit this. For the convenience of description, the following description is made by taking the execution subject as an example of a terminal.

[0037] See also Figure 1 The present application provides an embodiment of a method for generating multimodal data, comprising: 101. Obtain a multimodal data set of a pre-trained multimodal model, where the multimodal data set includes text data and audio data that have been annotated; In this embodiment, in order to build and train an efficient biological multimodal model, the terminal needs to obtain a multimodal data set in advance. This data set includes at least text data that has been strictly annotated and high-quality audio data. Because the simulation of these two types of data is relatively simple, researchers need to generate fine-grained aligned image data or even video data based on text data and audio data. Specifically, when determining a multimodal model, it is necessary to predefine biological objects and their characteristics, such as biological name text, hierarchical text (kingdom, phylum, class, order, family, genus, species), biological feature text, etc., and convert its call or text to audio as audio data. The audio data can be parsed and then processed for audio expansion to generate a large amount of audio data.

[0038] The text data and audio data obtained at this time have been data-marked, and then a specific image in the image data block needs to be selected based on the text data and audio data.

[0039] 102. Obtain label information of text data and audio data according to the annotation information of the multimodal data set; In multimodal data sets, different biological data have different annotation information, and the annotation information can be used to classify the data well. Annotation information refers to the process of manually or artificially defining the category of each data when collecting audio data, text data, and image data. For example, if multiple audios are collected, which are the roars of brown bears at different growth stages, each audio data can be annotated manually or artificially, and each audio data can be annotated with "brown bear" and the corresponding "growth stage" as the annotation information.

[0040] Like the audio data mentioned above, text data and image data also have annotation information and are generated in a similar manner.

[0041] This application generates label information through annotated information. For example, a segment of audio can generate label information such as "brown bear" (common name), "Mammalia", "Carnivora", "Ursidae", etc. by annotating information. More label information can be obtained by expanding the annotation information so that more comprehensive label information can be obtained. Text data is mainly generated through semantic models, which are not limited here. By integrating the label information of text data and audio data, a label information set that can extract several image data materials from the image data set is generated, that is, the label information is regularly bound through logical sum, logical AND, and logical OR. Researchers usually bind according to their own research needs.

[0042] Specifically, according to the research expectations preset by the researchers, the label information is associated, and the association (integration) of the label information can extract specific images that meet the fine-grained text data and audio data from the image database. The image database in this embodiment refers to a biological image database (image data set), which contains image information of different organisms, and also generates annotations or labels when collecting, that is, each image in the image database has corresponding annotations and labels, that is, the biological name, biological level, growth stage, etc. are annotated to facilitate retrieval. Through the label information of audio data and text data, images with similar labels can be collected from the image database. For example, the label information generated according to the label information of audio data and text data is "brown bear" (common name), "Mammalia", "Carnivora", and "Ursidae". These label information are used as search terms for association and integration, and multiple search formulas are generated to find matching images from the image database. Through such a search, fine-grained and more matching image data can be obtained.

[0043] However, the number of images of different organisms or different biological features in the image database may not meet the requirements of multimodal model training, so it is necessary to expand based on the limited images to generate enough training images.

[0044] 103. Extracting a plurality of image data materials from the image data set according to the label information, each image data material being associated with at least one label information; In this embodiment, the terminal integrates the tag information of the text data and the audio data to generate a tag information set that can extract a number of image data materials from the image data set (image database). That is, information such as a search formula is generated, and then a number of image data materials are extracted from the image data set through the tag information set.

[0045] The image data selected in this way can select images with specific biological features, for example, the complete image data of a bear can also select images of bear claws and paws with specific features (degree of wear, different age groups), as well as images of bear teeth with characteristics of different age groups. This type of extraction of several image data materials, audio data, and text data from the image data set is consistent in terms of fine-grainedness. Compared with the conventional selection method, the fine-grainedness of the selected image data materials is completely consistent, and the number of materials is also greater. For example: multiple paragraphs of text data describe research content related to brown bear claws, and audio data include the sound of brown bears using their claws and other audio related to bear claws. The manually annotated information (or the annotation information generated by artificial intelligence) is expanded and converted into label information to generate "bear claw", "brown bear" (common name), "Mammalia", "Carnivora", "Ursidae" and "juvenile", "adult", etc. These label information are integrated to generate multiple search formulas. The search formulas are used to search for images with corresponding labels in the image database, and finally images of brown bear claws of different ages are obtained. Such images can include images of the bear claws of the entire bear or images of only the bear claws, which are not limited here.

[0046] 104. Input the image data material into the convolutional neural network model, and determine a plurality of feature regions on the image data material according to the label information through the label feature enhancement module, wherein the convolutional neural network model includes the label feature enhancement module, an encoder, and a decoder; Next, the terminal inputs the image data material into the convolutional neural network model, and determines several feature areas on the image data material through the label feature enhancement module and based on the label information. The terminal determines several feature areas on the image data material based on the label information. Different label information corresponds to different feature areas. The feature areas of the lower-level labels and the feature areas of the upper-level labels will be segmented so that the label information corresponds to an exclusive feature area. Specifically, not all label features have feature areas. For example, the label information "Mammalia" does not refer to a specific visual feature, so there is no feature area corresponding to the label information "Mammalia", while visual features such as "bear claws" and "bear teeth" will have feature areas.

[0047] The convolutional neural network model includes a label feature enhancement module, an encoder and a decoder.

[0048] 105. Perform feature enhancement processing on each feature region according to the hierarchical parameters between the label information to generate label features, where the hierarchical parameters represent feature weight parameters between the label information; In this embodiment, the terminal performs feature enhancement processing on each feature region according to the hierarchical parameters between the tag information to generate tag features. Specifically, the terminal performs hierarchical analysis on the tag information to generate hierarchical parameters.

[0049] Specifically, the terminal needs to determine different label information in an image, determine the importance between the label information, and use this to generate the weight of each label information, and then determine the feature area corresponding to the label information.

[0050] 106. Generate several groups of normal distribution parameters for label features through normal distribution sampling; 107. Reconstruct several groups of normal distribution parameters and corresponding label information through an encoder to generate multimodal reconstructed image data.

[0051] After the label features are generated, the terminal generates different normal distribution parameters for the label features through normal distribution sampling. The terminal then inputs several sets of normal distribution parameters and corresponding label information into the decoder for reconstruction to generate multimodal reconstructed image data. The multimodal reconstructed image data generated in this way is not only fine-grained and consistent with the text data and audio data, but also has more material samples and more simulated image data. Secondly, the label features generated by the hierarchical parameters are more in line with the researchers' training expectations, the generated image data is more consistent with the text data and audio data, the fine-grained alignment of the entire multimodal data is higher, and the reconstructed image data is more prominent in specific features.

[0052] In this embodiment, a multimodal data set of a pre-trained multimodal model is first obtained, and the multimodal data set includes text data and audio data that have been annotated. The label information of the text data and the audio data is obtained according to the annotation information of the multimodal data set. A number of image data materials are extracted from the image data set according to the label information, and each image data material is associated with at least one label information. The image data material is input into the convolutional neural network model, and a number of feature areas are determined on the image data material according to the label information through the label feature enhancement module. The convolutional neural network model includes a label feature enhancement module, an encoder and a decoder. Feature enhancement processing is performed on each feature area according to the hierarchical parameters between the label information to generate label features, and the hierarchical parameters represent the feature weight parameters between the label information. Several groups of normal distribution parameters are generated for the label features by normal distribution sampling. Several groups of normal distribution parameters and corresponding label information are reconstructed by the encoder to generate multimodal reconstructed image data.

[0053] Several image data materials are extracted from the image data set through label information. This type of image data materials extracted from the image data set are consistent with the audio data and text data in type labels, and the number of materials is also greater. At the same time, the encoder and decoder are used for reconstruction, and the feature weight parameters of different label features are determined according to the label information of the image data material. Then, different feature areas are enhanced to different degrees through the feature weight parameters. The multimodal reconstructed image data is finally generated. The image data is adapted to the text data and audio data, and there are more material samples and more simulated image data. Secondly, through the label features generated by the hierarchical parameters, the features to be studied are more prominent and more in line with the training expectations of the researchers. The multimodal model can better train the required features, which improves the overall training effect of the multimodal model.

[0054] See also Figure 2 The present application provides an embodiment of a method for processing label features using a noise network, including: 201. Input the label features into the noise network, add preset noise to the label features through the noise network, and then denoise through stochastic differential equation prior iteration.

[0055] In this embodiment, the terminal inputs the label features into a preset noise network, adds noise to the label features, and then performs denoising through an adjusted stochastic differential equation prior iteration.

[0056] Its form is:

[0057] Where S(t) represents the random variable of label feature, U is the level parameter, μ is the drift rate, σ is the volatility, and W(t) is the standard Brownian motion.

[0058] Based on the diffusion model to generate priors, realistic images are synthesized through iterative denoising of stochastic differential equations. The two data, the preset label feature, a random variable and the level parameter, are added. Combined with the original stochastic differential equations, the image realism is increased. Given an input image with user guidance (in the form of manipulating RGB pixels), SDEdit first adds noise to the input and then iteratively denoises the generated image through SDE priors to increase its realism. SDEdit does not require task-specific training or inversion, and can naturally achieve a balance between realism and fidelity.

[0059] See also Figure 3 The present application provides an embodiment of a method for enhancing label features, comprising: 301. Generate a plurality of label information features for the image data material according to the label information and the image database; 302. Input the label feature, label information feature and layer parameter into the enhancement network; 303. Selectively enhance the label features according to the label features, label information features and priority parameters to generate edge-enhanced label features.

[0060] After the terminal performs denoising and noise addition on the label features, in order to prevent inaccurate feature area selection, which results in the feature area being smaller than the actual feature area, it is necessary to enhance the features of the outer area of ​​the feature area. First, the terminal needs to select another reference image from the image database based on the label information, and use the feature extraction module of the convolutional neural network model to collect features of the feature area of ​​the reference image to generate label information features. If the reference image has multiple label feature labels, it will have multiple label information features.

[0061] Next, the terminal inputs the label features, label information features and hierarchical parameters into the enhancement network for selective enhancement processing to generate enhanced label features.

[0062] GT=G0+(U1-1)*(U2-1)*...*(UN-1)*(G1*...GN) Among them, GT is the enhanced outer ring area feature, G0 is the original outer ring area feature, U1 to UN are N level parameters corresponding to the outer ring area feature, and G1 to GN are N label information features of the outer ring area. See also Figure 4 The present application provides an embodiment of a method for compensating a characteristic area, comprising: 401. Generate a level compensation value according to the feature area and level parameters of different feature areas in the current tag feature; 402. Perform grayscale compensation on the feature region corresponding to the label feature according to the hierarchical compensation value.

[0063] In this embodiment, after enhancing the outer region of the label feature, since there is actually label overlap between each feature region, that is, the feature region corresponding to an upper-level label information originally contains the feature region corresponding to a lower-level label information. In order to enhance the specificity of each label information in this embodiment, the feature region of the lower-level label information is divided, and the remaining part belongs to the upper-level label information. During the division process, the edge processing of the feature region is not delicate enough, which makes some feature regions have a small area and receive less attention. At this time, it is necessary to generate a hierarchical compensation value according to the feature region area and hierarchical parameter of different feature regions in the current label feature. Then perform grayscale compensation on the feature region corresponding to the label feature according to the hierarchical compensation value.

[0064] The specific method is as follows: First, determine the edge region according to the positions of two adjacent upper and lower feature regions. The edge region includes the small feature region plus half of the large feature region. The area of the upper feature region is S1, and the area of the lower feature region is S2. Assume S1 < S2. Then the hierarchical compensation value I is: I = μ1(U1 * S1) / (S1 + S2) Where μ1 is the grayscale mean value of the upper feature region, and U1 is the hierarchical parameter of the upper feature region. Finally, the terminal compensates the hierarchical compensation value I to the edge region to complete the compensation.

[0065] Please refer to Figure 5 , an embodiment of a method for generating a hierarchical parameter provided by this application includes: 501. Calculate the attention of each pixel point in the feature region of the image data material to generate attention value data; 502. Perform hierarchical analysis on the label information to generate a hierarchical parameter; 503. Perform attention adjustment processing on the attention value data according to the feature region and the hierarchical parameter; 504. Multiply the attention value data and several feature regions of the image data material channel by channel to generate a label feature.

[0066] In this embodiment, the terminal first calculates the attention of each pixel point in the feature region of the image data material to generate attention value data. Here, it can be self-attention calculation for the entire image or attention value calculation for multiple feature regions to generate an attention matrix.

[0067] Then, hierarchical analysis is performed for different label information. Specifically, it needs to be determined based on the label information in the image and the features to be studied by the researchers. For example, researchers need to analyze the claw features of bears. After searching for image materials, there may be images of whole brown bears, whole pandas, and red panda paws. The label features of these images are different. The features of the bear paw also include the features of the red panda's fur and the palm of the red panda. The image of the whole brown bear also includes features such as the bear's body. At this time, the bear paw (target feature) and other features need to be hierarchically analyzed.

[0068] The terminal will first count the number of label information in the image, take the label information of the target feature as the highest level, set a highest level parameter first, and then set level parameters for different label information. Specifically, the level parameters are calculated based on the feature correlation relationship between the label information of the target feature and other features. For example, the correlation between bear paws and bear claws is greater than that between bear ears, so the level parameter of bear paws is higher than that of bear ears.

[0069] In this embodiment, the terminal can first generate several fine-grained analog tags for text data and audio data through the semantic model and the audio analysis model. This is an analog tag automatically generated by the model method, which can complete the missing tags that are considered to be annotated. Then the terminal obtains several fine-grained annotation tags for text data and audio data, and determines the fine-grained analog tags and fine-grained annotation tags as tag information. Finally, the fine-grained analog tags and fine-grained annotation tags are analyzed at the upper and lower levels to generate hierarchical parameters.

[0070] The terminal performs hierarchical analysis on the tag information and generates hierarchical parameters, and then performs attention adjustment processing on the attention value data according to the feature area and hierarchical parameters.

[0071] The terminal performs attention adjustment processing on the attention value data according to the feature area and the hierarchical parameters. Specifically, the terminal uses the hierarchical parameters to adjust the attention value on each feature area. The adjustment formula is as follows:

[0072] in, is the adjusted attention value data, is the attention value data before adjustment, and U is the level parameter corresponding to the feature area corresponding to the pixel.

[0073] Next, the terminal multiplies the attention value data and several feature areas of the image data material by channel correspondence to generate label features. The label features can pay more attention to specific features, and according to the adjustments made according to different hierarchical parameters, they can better enhance the expected features at multiple levels, making the label features more in line with research expectations (training expectations).

[0074] See also Figure 6 The present application provides an embodiment of a method for processing tag features, including: 601. Perform feature encoding and dimensionality reduction processing on the label features.

[0075] Feature encoding is a key link in the field of machine learning and data mining. It involves preprocessing, converting, extracting and creating new features of raw data to improve the performance of the model. The goal of feature encoding is to convert raw data into a format that can be understood and processed by the machine learning model. This includes converting non-numeric data (such as categorical and textual data) into numerical data, and creating new features to capture the relationships and patterns between data. In this embodiment, in order to reduce the complexity of the model and improve performance, the terminal needs to perform feature encoding and dimensionality reduction processing on the features to reduce the amount of data calculation.

[0076] See also Figure 7 The present application provides an embodiment of a method for training a pre-trained multimodal model, comprising: 701. Annotate the image data and save them in a multimodal dataset; 702. Train the pre-trained multimodal model using the multimodal dataset.

[0077] In this embodiment, after the terminal generates the image data, the image data is annotated and saved in a multimodal data set, and finally the pre-trained multimodal model is trained using the multimodal data set.

[0078] See also Figure 8 The present application provides an embodiment of a device for generating multimodal data, comprising: A first acquisition unit 801 is used to acquire a multimodal data set of a pre-trained multimodal model, where the multimodal data set includes text data and audio data that have been annotated; A second acquisition unit 802 is used to acquire label information of text data and audio data according to the annotation information of the multimodal data set; An extraction unit 803 is used to extract a plurality of image data materials from the image data set according to the label information, each image data material being associated with at least one label information; A determination unit 804 is used to input the image data material into the convolutional neural network model, and determine a plurality of feature regions on the image data material according to the label information through the label feature enhancement module, the convolutional neural network model including the label feature enhancement module, an encoder and a decoder; The first generating unit 805 is used to perform feature enhancement processing on each feature region according to the hierarchical parameters between the tag information to generate tag features, where the hierarchical parameters represent feature weight parameters between the tag information; Optionally, the first generating unit 805 includes: Perform attention calculation on the pixel points of each feature area in the image data material to generate attention value data; Perform hierarchical analysis on label information to generate hierarchical parameters; Perform attention adjustment processing on the attention value data according to the feature area and level parameters; The attention value data and several feature areas of the image data material are multiplied by channel correspondence to generate label features.

[0079] A noise processing unit 806 is used to input the label feature into the noise network, add preset noise to the label feature through the noise network, and then perform denoising through stochastic differential equation prior iteration; The fourth generating unit 807 is used to generate a plurality of label information features for the image data material according to the label information and the image database; An input unit 808, used to input the label features, label information features and level parameters into the enhancement network; A fifth generating unit 809 is used to perform selective enhancement processing according to the label feature, the label information feature and the priority parameter to generate an edge-enhanced label feature; A sixth generating unit 810, configured to generate a level compensation value according to the feature area and level parameters of different feature areas in the current tag feature; A compensation unit 811 is used to perform grayscale compensation on a feature area corresponding to a tag feature according to a level compensation value; A dimension reduction unit 812 is used to perform feature encoding and dimension reduction processing on the label features; The second generating unit 813 is used to generate a plurality of groups of normal distribution parameters for the label features by normal distribution sampling; The third generating unit 814 is used to reconstruct a plurality of groups of normal distribution parameters and corresponding label information through an encoder to generate multi-modal reconstructed image data; A storage unit 815 is used to store the image data in a multimodal data set after data annotation; The training unit 816 is used to train the pre-trained multimodal model using the multimodal dataset.

[0080] See also Fig. 9 , the present application provides a device for generating multimodal data, comprising: Processor 901 , memory 902 , input-output unit 903 , and bus 904 .

[0081] The processor 901 is connected to the memory 902 , the input and output unit 903 , and the bus 904 .

[0082] The memory 902 stores a program, and the processor 901 calls the program to execute the following steps: Figure 1 , Figure 2 and Figure 3 , Figure 4 , Figure 5 , Figure 6 and Figure 7 The generation method in .

[0083] The present application provides a computer-readable storage medium, on which a program is stored, and when the program is executed on a computer, the program performs the following steps: Figure 1 , Figure 2 and Figure 3 , Figure 4 , Figure 5 , Figure 6 and Figure 7 The generation method in .

[0084] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0085] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0086] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0087] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0088] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A method for generating multimodal data, characterized in that: include: Acquire a multimodal data set of a pre-trained multimodal model, wherein the multimodal data set includes text data and audio data that have been annotated; Acquire label information of the text data and the audio data according to the annotation information of the multimodal data set; Extracting a plurality of image data materials from the image data set according to the label information, each image data material being associated with at least one label information; Inputting the image data material into a convolutional neural network model, and determining a plurality of feature regions on the image data material according to the label information through a label feature enhancement module, wherein the convolutional neural network model includes a label feature enhancement module, an encoder, and a decoder; Performing feature enhancement processing on each feature region according to the hierarchical parameters between the label information to generate label features, wherein the hierarchical parameters represent feature weight parameters between the label information; Generating several groups of normal distribution parameters for the label features by normal distribution sampling; A plurality of groups of the normal distribution parameters and corresponding label information are reconstructed through an encoder to generate multi-modal reconstructed image data.

2. The generation method according to claim 1, characterized in that: The convolutional neural network model also includes a noise network; After performing feature enhancement processing on each feature region according to the hierarchical parameters between the label information to generate label features, and before generating a plurality of groups of normal distribution parameters for the label features by normal distribution sampling, the generation method further includes: The label features are input into a noise network, preset noise is added to the label features through the noise network, and then denoising is performed through stochastic differential equation prior iteration.

3. The generation method according to claim 2, characterized in that: The convolutional neural network model also includes an edge enhancement network; After inputting the label feature into a noise network, adding preset noise to the label feature through the noise network, and then denoising through a priori iteration of a stochastic differential equation, and before generating several groups of normal distribution parameters for the label feature through normal distribution sampling, the generation method further includes: generating a plurality of label information features for the image data material according to the label information and the image database; Inputting the label feature, the label information feature and the level parameter into an enhancement network; Selective enhancement processing is performed according to the label feature, the label information feature and the priority parameter to generate edge-enhanced label features.

4. The generation method according to claim 3, characterized in that: The convolutional neural network model also includes a compensation network; After performing selection and enhancement processing according to the label feature, the label information feature and the priority parameter to generate edge-enhanced label features, and before generating a plurality of groups of normal distribution parameters for the label feature by normal distribution sampling, the generation method further includes: Generate a level compensation value according to the feature area and level parameters of different feature areas in the current label feature; Grayscale compensation is performed on the feature area corresponding to the label feature according to the level compensation value.

5. The generation method according to claim 1, characterized in that: Performing feature enhancement processing on each feature region according to the hierarchical parameters between the label information to generate label features, including: Performing attention calculation on the pixel points of each feature area in the image data material to generate attention value data; Performing hierarchical analysis on the tag information to generate hierarchical parameters; Performing attention adjustment processing on the attention value data according to the feature area and the level parameter; The attention value data and several feature areas of the image data material are multiplied correspondingly by channels to generate label features.

6. The generation method according to any one of claims 1 to 5, characterized in that: After performing feature enhancement processing on each feature region according to the hierarchical parameters between the label information to generate label features, and before generating a plurality of groups of normal distribution parameters for the label features by normal distribution sampling, the generation method further includes: The label features are subjected to feature encoding and dimensionality reduction processing.

7. The generation method according to any one of claims 1 to 5, characterized in that: After reconstructing several groups of the normal distribution parameters and the corresponding label information through an encoder to generate multimodal reconstructed image data, the generation method further includes: Annotating the image data and saving them in a multimodal dataset; The pre-trained multimodal model is trained using the multimodal dataset.

8. A device for generating multimodal data, characterized in that: include: A first acquisition unit is used to acquire a multimodal data set of a pre-trained multimodal model, wherein the multimodal data set includes text data and audio data that have been annotated; A second acquisition unit, configured to acquire label information of the text data and the audio data according to the annotation information of the multimodal data set; An extraction unit, configured to extract a plurality of image data materials from the image data set according to the label information, each image data material being associated with at least one label information; A determination unit, configured to input the image data material into a convolutional neural network model, and determine a plurality of feature regions on the image data material according to the label information through a label feature enhancement module, wherein the convolutional neural network model includes a label feature enhancement module, an encoder, and a decoder; A first generating unit is used to perform feature enhancement processing on each feature area according to the hierarchical parameters between the label information to generate label features, wherein the hierarchical parameters represent feature weight parameters between the label information; A second generating unit, used for generating a plurality of groups of normal distribution parameters for the label features by normal distribution sampling; The third generating unit is used to reconstruct the plurality of groups of normal distribution parameters and corresponding label information through an encoder to generate multi-modal reconstructed image data.

9. A device for generating multimodal data, characterized in that: include: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a program stored thereon, wherein the program, when executed on a computer, performs the generation method according to any one of claims 1 to 7.

Images