Automatic test evaluation construction method of multi-modal large model and related device

Abstract

The invention discloses an automatic test evaluation construction method of a multi-modal large model and a related device, relates to the technical field of multi-modal large models, and is used for detecting the performance of the multi-modal large model. The automatic test evaluation construction method for the multi-modal large model comprises the steps of obtaining sample data; constructing an adversarial network architecture; inputting the sample data into an adversarial network architecture for training to obtain target test data; judging whether the target test data is repeated with data in a preset block chain database or not; if not, storing the target test data to a block chain database; positioning the starting node and a task node corresponding to the target application scene in a pre-created knowledge graph; obtaining a target test index according to the starting node and the task node, and allocating a weight to the target test index according to the target application scene; inputting the target test data into the multi-modal large model for testing to obtain a test result; and generating an evaluation report according to the test result and the target test index.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of multimodal large models, and in particular to a method for constructing an automatic test and evaluation system for a multimodal large model and related devices. Background Art

[0002] With the rapid development of artificial intelligence technology, multimodal big models have become a hot area of ​​research and application. Multimodal big models integrate multiple data modalities such as text, images, audio, and video, aiming to achieve more powerful and intelligent information processing capabilities, and thus are widely used in many industries such as medical care, transportation, entertainment, and education. In the development process of multimodal big models, reliable and accurate testing and evaluation are crucial. However, the current existing testing and evaluation methods have many significant defects.

[0003] Among them, due to the lack of an effective data management mechanism, the test data used for test evaluation is very easy to be mixed into the model's training data, causing serious data "pollution" problems. Once the test data contains training samples that the model has already learned, the model can easily give seemingly excellent results when facing these "familiar" data. Such test results obviously cannot truly reflect the actual performance of the model when facing new and unfamiliar data. Therefore, this data "pollution" phenomenon will not only lead to serious distortion of the evaluation results, but also mislead the direction of model improvement, making it impossible for model developers to accurately judge the pros and cons of the model, thereby hindering the iteration and advancement of technology. Summary of the invention

[0004] The present application discloses a method and related device for constructing an automatic test evaluation of a multimodal large model, which are used to detect the performance of the multimodal large model.

[0005] The first aspect of the present application discloses a method for constructing an automatic test evaluation of a multimodal large model, comprising: Acquire sample data, where the sample data is sample data screened from a public modality data resource library according to a data generation algorithm; Constructing an adversarial network architecture according to multiple modal combinations of a multimodal large model, wherein the multiple modal combinations are multiple combinations of at least two modal combinations; Inputting the sample data into the adversarial network architecture for training to obtain target test data; Determine whether the target test data is duplicated with data in a preset blockchain database; If not, storing the target test data in the blockchain database; According to the target application scenario of the multimodal large model and the target test data, locate the starting node and the task node corresponding to the target application scenario in the pre-created knowledge graph, wherein the target application scenario is an application scenario corresponding to the target test data; Acquire a target test indicator according to the starting node and the task node, and assign a weight to the target test indicator according to the target application scenario; Inputting the target test data into the multimodal large model for testing to obtain a test result; An evaluation report is generated based on the test results and the target test indicators.

[0006] Optionally, the constructing of the adversarial network architecture according to the combination of multiple modes of the multimodal large model includes: According to the free combination of the image modality and the text modality of the multimodal large model, a first generator combining a convolutional neural network and a recurrent neural network is set, wherein the first generator is used to process the data of the image modality and the text modality; According to the free combination of the video modality and the audio modality of the multimodal large model, a second generator combining a 3D convolutional neural network and an audio generative adversarial network is set, wherein the second generator is used to process the data of the video modality and the audio modality; According to the first generator, a first discriminator of a corresponding multi-layer perceptron architecture is set, wherein the first discriminator is used to discriminate the authenticity of the data of the image modality and the text modality; According to the second generator, a second discriminator of a corresponding audio and video discrimination network architecture is set, wherein the second discriminator is used to discriminate the authenticity of the data of the video modality and the audio modality; Adding a first optimization function in the first generator and the second generator respectively; Adding a second optimization function to the first discriminator and the second discriminator respectively; An adversarial network architecture is constructed according to the first generator, the second generator, the first discriminator and the second discriminator.

[0007] Optionally, the first optimization function is: min =log(1-D i (G j (z))), where P min The probability that the data generated by the first generator or the second generator is judged as false by the discriminator, D i is the discriminator, G j is the generator, z is the input random noise or conditional vector, i,j={1,2}.

[0008] Optionally, the second optimization function is: max =log(D(x))+log(1-D i (G j (z))), where Pmax is the probability that the first discriminator or the second discriminator correctly discriminates the authenticity, D i is the discriminator, G j is the generator, z is the input random noise or conditional vector, x is the input real data, i,j={1,2}.

[0009] Optionally, the determining whether the target test data is repeated with data in a preset blockchain database includes: The target test data is stored in a local temporary buffer, and a hash value of the target test data is calculated and corresponding meta information is generated; According to the hash value, it is determined whether the target test data is repeated with the data in the preset blockchain database.

[0010] Optionally, storing the target test data in the blockchain database includes: The hash value and the meta-information are sent to a designated node of the blockchain database, so that after the designated node verifies the hash value and the meta-information, the hash value and the meta-information are packaged and sent to all nodes of the blockchain database.

[0011] Optionally, locating a starting node and a task node corresponding to the target application scenario in a pre-created knowledge graph according to the target application scenario of the multimodal large model and the target test data includes: According to the target test data, locate the starting node in a pre-created knowledge graph, wherein the pre-created knowledge graph is a knowledge graph constructed based on knowledge information in multiple actual application cases, and the knowledge graph includes scenario information, capability information, task information, indicator information, and association relationship information applied to a multimodal large model; Identify capability information and task information of the target application scenario; The corresponding task node is located according to the capability information and the task information.

[0012] The second aspect of the present application discloses an automatic test evaluation construction system for a multimodal large model, comprising: A sample data unit, used to obtain sample data, wherein the sample data is sample data screened from a public modality data resource library according to a data generation algorithm; An adversarial network architecture unit, used to construct an adversarial network architecture according to multiple modal combinations of the multimodal large model, wherein the multiple modal combinations are multiple combinations of at least two modal combinations; A training unit, used for inputting the sample data into the adversarial network architecture for training to obtain target test data; A judging unit, used to judge whether the target test data is repeated with data in a preset blockchain database; An execution unit, configured to store the target test data in the blockchain database if there is no duplication; A positioning unit, used to locate a starting node and a task node corresponding to the target application scenario in a pre-created knowledge graph according to the target application scenario of the multimodal large model and the target test data, wherein the target application scenario is an application scenario corresponding to the target test data; A target test indicator unit, used to obtain a target test indicator according to the start node and the task node, and assign a weight to the target test indicator according to the target application scenario; A testing unit, used for inputting the target test data into the multimodal large model for testing to obtain a test result; An evaluation unit is used to generate an evaluation report based on the test results and the target test indicators.

[0013] A third aspect of the present application provides an electronic device, including: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the automatic test evaluation construction method of the first aspect and any optional multi-modal large model of the first aspect.

[0014] The fourth aspect of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, it executes the automatic testing and evaluation construction method of the first aspect and any optional multimodal large model of the first aspect.

[0015] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages: The automatic test evaluation construction method of the multimodal large model of the present application first obtains sample data, which is sample data screened from the public modal data resource library according to the data generation algorithm; constructs an adversarial network architecture according to the multiple modal combinations of the multimodal large model, and the multiple modal combinations are multiple combinations of at least two modal combinations; inputs the sample data into the adversarial network architecture for training to obtain target test data; determines whether the target test data is repeated with the data in the pre-set blockchain database; if not, stores the target test data in the blockchain database; locates the task node corresponding to the starting node and the target application scenario in the pre-created knowledge graph according to the target application scenario and the target test data of the multimodal large model, and the target application scenario is the application scenario corresponding to the test target test data; obtains the target test index according to the starting node and the task node, and assigns weights to the target test index according to the target application scenario; inputs the target test data into the multimodal large model for testing to obtain the test results; generates an evaluation report according to the test results and the target test index. This method screens sample data from the public modal data resource library through a specific data generation algorithm, can preliminarily control the data at the source, and selects data that meets the basic test requirements of the multimodal large model. Then, the blockchain database is used to determine whether the target test data is repeated with the existing data in the database after obtaining it, and only the non-duplicate data is stored in the blockchain database. With the help of the blockchain's tamper-proof and traceable characteristics, the situation where training data is mixed into the target test data is effectively avoided, solving the common data "contamination" problem in traditional testing, ensuring the purity and independence of the target test data, so that the subsequent test evaluation results based on these target test data can truly reflect the actual performance of the multimodal large model on unseen data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0017] Figure 1 A schematic diagram of an embodiment of a method for constructing automatic testing and evaluation of a multimodal large model of the present application; Figure 2 A schematic diagram of an embodiment of a method for constructing an adversarial network architecture for the present application; Figure 3 A schematic diagram of an embodiment of a method for determining the repeatability of target test data in the present application; Figure 4 A schematic diagram of an embodiment of a method for locating a start node and a task node for the present application; Figure 5 A schematic diagram of an embodiment of a system for automatically testing and evaluating a multimodal large model of the present application; Figure 6 This is a schematic diagram of another embodiment of the electronic device of the present application. DETAILED DESCRIPTION

[0018] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0019] It should be understood that when used in the present application specification, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0020] It should also be understood that the term “and / or” used in the specification of the present application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0021] As used in the present specification, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if [the described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [the described condition or event] is detected" or "in response to detecting [the described condition or event]" depending on the context.

[0022] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0023] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0024] In the current field of multimodal large model testing, the acquisition and management of test data are in deep trouble and present an extremely chaotic situation, which has become one of the key bottlenecks restricting the progress of the industry.

[0025] Traditional test data acquisition methods are mainly concentrated in two aspects, but both have drawbacks that are difficult to overcome. First, they rely heavily on limited public data sets. Although public data sets have certain conveniences and seem to provide a ready-made data resource pool, they are actually full of problems. On the one hand, the coverage of these data sets is relatively narrow, often focusing on certain specific fields or common scenarios, and it is difficult to fully cover the diverse situations that multimodal large models may encounter in complex real-world applications. For example, when dealing with cross-cultural, niche industries or emerging business scenarios, there is an extreme lack of matching data in public data sets, and the adaptability of the model cannot be effectively tested. On the other hand, since many research teams can access and use these public resources, the data has been repeatedly mined and used, overexposed, and its uniqueness and freshness have been greatly reduced. For multimodal large model tests that pursue innovative evaluation, its value is declining.

[0026] Second, the method of collecting data by yourself is far from ideal. Self-collection means piecing together data from a large amount of online information, databases of different institutions, and field collection, which directly leads to extremely scattered data sources. Data from different sources are independent of each other in terms of collection standards, format specifications, quality control, etc., and there is no unified rule to follow. Multimodal information captured from the Internet may be messy due to differences in web page layout and arbitrary data labeling; data obtained from cooperation with different institutions often have many problems such as inconsistent resolution, confusing timestamps, and lack of correlation between modalities due to differences in the internal data management systems of each institution, resulting in uneven overall data quality and difficulty in meeting the stringent requirements of rigorous testing.

[0027] What is even more fatal is that, whether it is a public data set or self-collected data, it is very easy to mix into the model's training data under the existing process, thus causing a serious data "pollution" disaster. Take the life-threatening scenario of multimodal testing of medical images as an example. Medical imaging data is already scarce and precious. In order to pursue the adequacy of model training, the research team may inadvertently include the trained imaging data in the test scope. In this way, when the model faces these "familiar" test data again, it can quickly give seemingly excellent results based on its existing memory without deep analysis and complex reasoning.

[0028] Based on this, the present application discloses a method and related device for automatically testing and evaluating a large multimodal model, which are used to detect the performance of the large multimodal model.

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] The method of the present application can be applied to a server, a device, a terminal or other devices with logic processing capabilities, and the present application does not limit this. For the convenience of description, the following description is made by taking the execution subject as an example of a terminal.

[0031] See also Figure 1 The present application provides an embodiment of a method for constructing an automatic test evaluation of a multimodal large model, comprising: 101, obtaining sample data, where the sample data is sample data screened from a public modality data resource library according to a data generation algorithm; In this embodiment, the public modality data resource library may include data sets published by academic institutions and some open source multimodal data sets, etc., and any database with data copyright and use license can be used, and there is no specific limitation here. The data generation algorithm includes pre-set data screening rules and preprocessing algorithms, wherein the data screening rules are data screening rules formulated according to the application scenarios and test requirements of the multimodal large model. For example, if the multimodal large model is used for medical imaging diagnosis and report generation, the screened data should mainly be medical images (such as X-rays, CT, MRI images) and the corresponding medical diagnosis report texts. Among them, for image modalities, the screening rules may include image categories (such as human body parts, disease types), image quality indicators (such as clarity, contrast), etc. For text modalities, the rules may include the length range of the text, vocabulary professionalism (such as medical terms), the format of the text (such as whether it is a structured diagnostic report), etc. Correspondingly, the preprocessing algorithm for image modality data may include image scaling algorithm, normalization algorithm, color conversion algorithm, etc.; the preprocessing algorithm for text modality data may include text denoising algorithm, text segmentation algorithm and vocabulary building algorithm, etc.

[0032] According to the above data screening rules, the data files in the public modality data repository are traversed to read the metadata and data content itself. For data that meets the data screening rules, it is extracted and preprocessed, that is, sample data is obtained, and then the sample data is sorted and stored to ensure that data of different modalities can be correctly associated. For example, when the sample data meets the requirements of medical imaging and diagnostic reports, the images are scaled, normalized, etc., and the diagnostic reports are cleaned, segmented, etc., and then the processed sample data are combined and stored in a temporary local database or data structure.

[0033] 102. Constructing an adversarial network architecture according to multiple modal combinations of a multimodal large model, wherein the multiple modal combinations are multiple combinations of at least two modal combinations; In this embodiment, the types of modalities contained in the multimodal large model are first clarified, such as text, image, audio, video, etc., and then the modal combinations used to build the adversarial network architecture are determined. These modal combinations can be any combination of two or more modalities, such as text-image combinations, audio-video combinations, etc. Building an adversarial network architecture includes setting up a generator and a discriminator, and defining a loss function (or optimization function) in the generator and the discriminator. The generator should be able to receive input of a specified modal combination and generate realistic multimodal data, and the discriminator should be able to distinguish between real multimodal data and data generated by the generator.

[0034] 103, inputting the sample data into the adversarial network architecture for training to obtain target test data; In this embodiment, the sample data obtained in step 101 is input into the adversarial network framework, and the input order and format of the data should be ensured to match the adversarial network framework. During the training of the sample data by the adversarial network framework, the generator and the discriminator are respectively trained in an adversarial manner, and the parameters are continuously updated iteratively. During the training process, the loss values ​​of the generator and the discriminator, as well as the quality of the output target test data, should also be detected in real time, and the parameters such as the learning rate and the batch size of the sample data for one training should be adjusted according to actual needs.

[0035] 104, determine whether the target test data is repeated with the data in the preset blockchain database; if not, execute step 105; if so, discard the target test data.

[0036] In this embodiment, the pre-set blockchain database includes test data generated earlier and may also include open source multimodal data sets. The purpose of determining the repeatability of the target test data in the blockchain database is to ensure that the performance of the multimodal large model is not affected by duplicate data and to avoid "false" test results. A unique identifier can be set on the target test data, and the data in the blockchain database also have a unique identifier in the same format. By comparing the unique identifiers, it is determined whether there are duplications in multiple data.

[0037] 105, storing the target test data in the blockchain database; In this embodiment, if the judgment result of step 104 is that the target test data and the data in the blockchain database are not repeated, it means that the target test data is available and will not have a negative impact on the test of the multimodal large model. Therefore, it is confirmed that the target test data can be used in subsequent tests, and the target test data should be stored in the blockchain database to avoid reusing the target test data in the next test.

[0038] 106. According to the target application scenario and target test data of the multimodal large model, locate the starting node and the task node corresponding to the target application scenario in the pre-created knowledge graph, where the target application scenario is the application scenario corresponding to the test target test data; In this embodiment, the pre-created knowledge graph is a graph structure composed of nodes and edges, wherein the nodes of the knowledge graph represent different entities, such as application scenarios, model capabilities, tasks, and indicators, etc.; the edges of the knowledge graph represent the relationships between entities. For example, in the multimodal large model knowledge graph, "medical imaging diagnosis" is an application scenario node, and there is a "need" relationship edge between it and the "image recognition capability" node. According to the keyword tags extracted from the target test data, a keyword matching query is performed in the knowledge graph to locate the starting node; then the corresponding task node is searched and located according to the target application scenario, and the task node contains the target test task and the target test indicator. The starting node and multiple task nodes are connected to form a task chain, which can guide the multimodal large model to be tested.

[0039] 107, obtaining a target test indicator according to the starting node and the task node, and assigning a weight to the target test indicator according to the target application scenario; In this embodiment, the query function of the knowledge graph is used, starting from the starting node, passing through each corresponding task node, and obtaining the target test index through the association relationship between the starting node and the task node. In the knowledge graph, each task node is usually connected to a set of test index nodes, and these connection relationships represent the evaluation indicators corresponding to the task. For example, in the knowledge graph storage system, the Cypher query language is used to find the test index nodes related to the image lesion recognition task in the medical image diagnosis scenario. Analyze the characteristics and requirements of the target application scenario and determine the factors affecting the importance of each test index. Then assign weights to the target test index according to the importance factors. Taking medical image diagnosis as an example, the accuracy of diagnosis is crucial, because an incorrect diagnosis may lead to serious medical consequences, so the weight of accuracy-related indicators should be higher; and although the model processing time is also important, it has a slightly lower priority relative to accuracy.

[0040] 108, inputting the target test data into the multimodal large model for testing to obtain a test result; In this embodiment, for each batch of input target test data, the multimodal large model performs reasoning calculations according to its internal network architecture and algorithm. During the reasoning calculation process, the multimodal large model will perform corresponding processing according to the modal type of the target test data, for example: feature extraction of target test data in image modality, semantic understanding of test data in text modality, and so on. And multimodal fusion and task processing are performed according to the trained parameters. For example: in a visual question-and-answer task, the model will generate an answer based on the input image and related text questions. This reasoning calculation process is implemented by the forward propagation of the multimodal large model. In the deep learning framework, the prediction or reasoning function of the multimodal large model is called to execute, and finally the test results are output.

[0041] 109, generating an evaluation report based on the test results and target test indicators.

[0042] In this embodiment, all the results obtained during the test are collated, including but not limited to performance indicators, error logs, user feedback, etc. Ensure the completeness and accuracy of the results to avoid omissions or errors. Compare the collated test results with the target test indicators one by one. Evaluate whether the test results meet or exceed the target test indicators, identify which indicators meet the standards and which indicators do not meet the standards. Then, according to the purpose and scope of the evaluation, determine the structure and content of the evaluation report, and finally generate the evaluation report.

[0043] This method uses a specific data generation algorithm to filter sample data from the public modal data resource library, which can initially control the data at the source and select data that meets the basic testing requirements of the multimodal large model. Then, using the blockchain database, after obtaining the target test data, it is determined whether it is repeated with the existing data in the library, and only the non-duplicate data is stored in the blockchain database. With the help of the blockchain's non-tamperable and traceable characteristics, it effectively avoids the situation where the target test data is mixed with the training data, solves the common data "contamination" problem in traditional testing, ensures the purity and independence of the target test data, and enables the subsequent test evaluation results based on these target test data to truly reflect the actual performance of the multimodal large model on unseen data.

[0044] See also Figure 2 , with respect to step 102, the present application provides an embodiment of a method for constructing an adversarial network architecture, including: 201, according to the free combination of the image modality and the text modality of the multimodal large model, setting a first generator combining a convolutional neural network and a recurrent neural network, the first generator being used to process data of the image modality and the text modality; In this embodiment, for the image-text modality combination in the multimodal large model, a part of the first generator adopts an encoder-decoder structure. The encoder is composed of a multi-layer convolutional neural network, which is used to extract the feature representation of the input random noise vector and gradually map the low-dimensional noise to the high-dimensional image feature space. The decoder uses a transposed convolutional layer to gradually restore the image details. At the same time, another part of the first generator is equipped with a recurrent neural network branch, such as a long short-term memory network, which is used to generate a text description matching the image features. For example, when generating e-commerce product pictures and descriptions, the multi-layer convolutional neural network part generates product appearance images, and the recurrent neural network generates text such as product names, features, and usage scenarios based on image features.

[0045] Among them, the multi-layer convolutional neural network uses multiple convolutional layers, pooling layers, and fully connected layers. For example, the initial convolutional layer can use a 3x3 convolution kernel with a step size of 1 and a padding of 1, and then extract the preliminary features of the image. As the number of network layers deepens, the convolution kernel size is gradually increased or the step size is reduced to obtain more abstract image features. The pooling layer can use maximum pooling or average pooling to reduce the data dimension. The image features extracted by the multi-layer convolutional neural network will be flattened and input into the recurrent neural network part. The number of hidden units of the recurrent neural network can be set according to the specific text length and complexity. For example, if the average length of the generated text description is 50 words, the hidden unit can be set to 128 or 256. In the output layer of the recurrent neural network, the softmax function can be used to convert the output into a probability distribution of each word in the vocabulary to generate a text description.

[0046] 202, according to the free combination of the video modality and the audio modality of the multimodal large model, setting a second generator combining a 3D convolutional neural network and an audio generative adversarial network, the second generator being used to process data of the video modality and the audio modality; In this embodiment, for the video-audio modality combination in the multimodal large model, the second generator is designed as a combination of a 3D convolutional neural network and an audio generative adversarial network. The 3D convolutional neural network part is responsible for generating video frame sequences and learning the dynamic characteristics of the video in the time and space dimensions, while the audio generative adversarial network part generates the corresponding audio waveform to ensure that the audio and video are coordinated in terms of rhythm, emotional atmosphere, etc. For example, when generating a dance teaching video, the second generator can synchronously produce music that accurately matches the rhythm of the dance movements.

[0047] Among them, the 3D convolutional neural network is used to generate video frame sequences. Its convolution kernel operates simultaneously in the spatial (x, y axis) and temporal (t axis) dimensions. For example, a 3x3x3 convolution kernel with a step size of 1 and a padding of 1 is used to learn the spatiotemporal dynamic features of the video. The generator can gradually map low-dimensional features back to the high-dimensional video frame space through the deconvolution layer (transposed convolution layer). The audio generative adversarial network can be constructed based on the spectral features of the generated audio. According to the amplitude spectrum and phase spectrum of the generated audio, it is converted into an audio waveform using an inverse Fourier transform. When combining video and audio generation, the emotion, scene and other features of the video are correlated with the rhythm, timbre and other features of the audio through a shared feature fusion layer to generate high-quality video-audio modality data.

[0048] 203, according to the first generator, setting a first discriminator of a corresponding multi-layer perceptron architecture, the first discriminator being used to discriminate the authenticity of data in image modality and text modality; In this embodiment, the first discriminator constructs a classification network, and inputs a real or generated image-text pair into the first discriminator, and judges the authenticity of the input data through a series of convolutional layers and fully connected layers. In the early stage of training, the first discriminator can distinguish between real and generated data relatively easily, but as the training progresses, the generator is continuously optimized, making it more difficult to distinguish.

[0049] 204, according to the second generator, setting a second discriminator of the corresponding audio and video discrimination network architecture, the second discriminator is used to discriminate the authenticity of the data of the video modality and the audio modality; In this embodiment, the second discriminator is composed of a 3D convolutional neural discriminant network and an audio discriminant network, which respectively discriminate the authenticity of the video and audio. Only when the video-audio pair as a whole conforms to the distribution characteristics of real data is it judged to be true, prompting the generator to generate high-quality, highly modally fused data.

[0050] 205, adding a first optimization function in the first generator and the second generator respectively; In this embodiment, a probability value is outputted by the first optimization function, which indicates the probability that the data outputted by the first generator or the second generator is judged as false by the discriminator. The smaller the probability value, the closer the data outputted by the first generator or the second generator is to the real data, that is, the higher the quality of the outputted data.

[0051] Among them, the first optimization function is: P min =log(1-D i (G j (z))), where P min The probability that the data generated by the first generator or the second generator is judged as false by the discriminator, D i is the discriminator, G j is the generator, z is the input random noise or conditional vector, i,j={1,2}.

[0052] 206, adding a second optimization function to the first discriminator and the second discriminator respectively; In this embodiment, similarly, a probability value is outputted through the second optimization function, and the probability value represents the probability that the first discriminator or the second discriminator correctly discriminates the real data and the generated data, that is, discriminates which data is the real data directly obtained from the database, and which data is the generated data outputted by the first generator or the second generator. The larger the probability value, the better the discrimination performance of the first discriminator or the second discriminator, and the more accurate the control of the quality of the generated data.

[0053] Among them, the second optimization function is: P max =log(D(x))+log(1-D i (G j(z))), where P max is the probability that the first discriminator or the second discriminator correctly discriminates the authenticity, D i is the discriminator, G j is the generator, z is the input random noise or conditional vector, x is the input real data, i,j={1,2}.

[0054] 207 , construct an adversarial network architecture according to the first generator, the second generator, the first discriminator, and the second discriminator.

[0055] In this embodiment, the first generator, the second generator, the first discriminator and the second discriminator are combined together, and the generator and the discriminator are trained alternately to form an adversarial network architecture, thereby increasing the diversity and richness of the generated data and improving the accuracy and robustness of the discrimination.

[0056] Reference Figure 3 , with respect to step 104 to step 105, the present application provides an embodiment of a method for determining the repeatability of target test data, including: 301, storing the target test data in a local temporary buffer, calculating a hash value of the target test data and generating corresponding meta information; In this embodiment, the target test data is first received and stored in a local temporary cache area. The local temporary cache area is usually established to improve processing speed and efficiency, and it allows data to be quickly accessed during the processing process. The target test data is then hashed using a hash function (such as SHA-256, MD5, etc.). The hash function maps data of any size to a fixed-size hash value, which will change significantly when the data changes slightly, so it can be used to detect the integrity or uniqueness of the data. Then, based on the type, size, source, creation time and other information of the target test data, meta information is generated, that is, the meta information is a collection of the above information.

[0057] 302, judging whether the target test data is repeated with the data in the preset blockchain database according to the hash value; if so, discarding the target test data; if not, executing step 303; In this embodiment, according to the hash value calculated in step 301, a search is performed in the blockchain database to determine whether there is an identical hash value. If a record matching the calculated hash value is found in the blockchain database, it indicates that the target test data is duplicated with the existing data in the blockchain database, and thus the target test data can be discarded to avoid duplicate storage and processing; if no matching hash value is found, it indicates that the target test data is unique, and the next step of processing can be continued.

[0058] 303, sending the hash value and the meta information to the designated node of the blockchain database, so that the designated node, after verifying the hash value and the meta information, packages the hash value and the meta information and sends them to all nodes of the blockchain database.

[0059] In this embodiment, the hash value and meta information are sent together to a designated node in the blockchain database. This designated node is usually responsible for receiving new data requests and performing preliminary verification. After receiving the hash value and meta information, the designated node will verify it. The verification process may include checking whether the format of the hash value is correct, whether the meta information is complete, etc. In some cases, further verification of the target test data may be required to ensure its authenticity and reliability. If the hash value and meta information pass the verification, the designated node will package it into a new block and send it to all other nodes of the blockchain database. This process is achieved through the consensus mechanism in the blockchain network, ensuring that all nodes can receive and confirm the new block. Once the new block is confirmed by all nodes and added to the blockchain, the hash value and meta information of the target test data are permanently stored in the blockchain database and can be accessed and verified by anyone who needs it.

[0060] Reference Figure 4 For step 106, the present application provides an embodiment of a method for locating a start node and a task node, including: 401, according to the target test data, locate the starting node in the pre-created knowledge graph, the pre-created knowledge graph is a knowledge graph constructed based on knowledge information in multiple actual application cases, and the knowledge graph includes scenario information, capability information, task information, indicator information and association relationship information applied to the multimodal large model; In this embodiment, the target test data is first defined, and the target test data includes information such as data type and specific data content. The pre-created knowledge graph is constructed based on knowledge information in multiple actual application cases, and it contains rich scenario information, capability information, task information, indicator information, and association relationship information. This information is organized into nodes and edges to represent the associations and attributes between different entities. In the knowledge graph, according to the characteristics and requirements of the target application scenario, the starting node related to the scenario is located.

[0061] 402, identifying capability information and task information of the target application scenario; In this embodiment, in the knowledge graph, the capability information associated with the task node is identified, and this capability information describes the various capabilities that the multimodal large model needs to have when processing the target application scenario, such as natural language understanding ability, image recognition ability, logical reasoning ability, etc. Similarly, the task information associated with the task node should also be identified in the knowledge graph. This task information describes the specific tasks that the multimodal large model needs to complete when processing the target application scenario, such as dialogue understanding tasks, object detection tasks, product recommendation tasks, etc. The identified capability information and task information are then integrated to form a comprehensive understanding of the target application scenario. This helps to locate the corresponding task nodes based on this information in subsequent steps.

[0062] 403. Locate the corresponding task node according to the capability information and the task information.

[0063] In this embodiment, in the knowledge graph, according to the identified capability information and task information, the matching task nodes are searched. These task nodes may represent the specific capabilities or skills required by the multimodal large model when processing the target application scenario, as well as the specific tasks or subtasks that need to be completed. Together, they constitute all the tasks that the multimodal large model needs to complete when processing the target application scenario. After locating the task node, the path from the starting node to the task node can be further planned in the knowledge graph. This path represents a series of steps or stages that the multimodal large model needs to go through when processing the target application scenario.

[0064] The above embodiment describes the method for automatically testing and evaluating a multimodal large model provided by the present application. The following describes the automatic testing and evaluating system, electronic device, and storage medium for automatically testing and evaluating a multimodal large model provided by the present application: See also Figure 5 The present application provides an embodiment of an automatic test evaluation construction system for a multi-modal large model, including: The sample data unit 501 is used to obtain sample data, where the sample data is sample data screened from a public modality data resource library according to a data generation algorithm; The adversarial network architecture unit 502 is used to construct an adversarial network architecture according to multiple modal combinations of the multimodal large model, where the multiple modal combinations are multiple combinations of at least two modal combinations; A training unit 503 is used to input sample data into the adversarial network architecture for training to obtain target test data; A determination unit 504 is used to determine whether the target test data is repeated with the data in the preset blockchain database; An execution unit 505 is used to store the target test data in a blockchain database if there is no duplication; A positioning unit 506 is used to locate the starting node and the task node corresponding to the target application scenario in the pre-created knowledge graph according to the target application scenario and the target test data of the multimodal large model, where the target application scenario is the application scenario corresponding to the test target test data; A target test indicator unit 507 is used to obtain a target test indicator according to a start node and a task node, and to assign a weight to the target test indicator according to a target application scenario; The testing unit 508 is used to input the target test data into the multimodal large model for testing and obtain the test results; The evaluation unit 509 is used to generate an evaluation report according to the test results and target test indicators.

[0065] Optionally, the adversarial network architecture unit 502 is specifically used for: According to the free combination of the image modality and the text modality of the multimodal large model, a first generator combining a convolutional neural network and a recurrent neural network is set, and the first generator is used to process the data of the image modality and the text modality; According to the free combination of the video modality and the audio modality of the multimodal large model, a second generator combining a 3D convolutional neural network and an audio generative adversarial network is set, and the second generator is used to process the data of the video modality and the audio modality; According to the first generator, a first discriminator of a corresponding multi-layer perceptron architecture is set, and the first discriminator is used to discriminate the authenticity of data in image mode and text mode; According to the second generator, a second discriminator of the corresponding audio and video discrimination network architecture is set, and the second discriminator is used to discriminate the authenticity of the data of the video modality and the audio modality; Adding a first optimization function in the first generator and the second generator respectively; Adding a second optimization function to the first discriminator and the second discriminator respectively; An adversarial network architecture is constructed according to the first generator, the second generator, the first discriminator and the second discriminator.

[0066] Optionally, the first optimization function is: P min =log(1-D i (G j (z))), where P min The probability that the data generated by the first generator or the second generator is judged as false by the discriminator, D i is the discriminator, G j is the generator, z is the input random noise or conditional vector, i,j={1,2}.

[0067] Optionally, the second optimization function is: P max =log(D(x))+log(1-Di (G j (z))), where P max is the probability that the first discriminator or the second discriminator correctly discriminates the authenticity, D i is the discriminator, G j is the generator, z is the input random noise or conditional vector, x is the input real data, i,j={1,2}.

[0068] Optionally, the determining unit 504 is specifically configured to: Store the target test data in a local temporary cache, calculate the hash value of the target test data, and generate corresponding meta information; According to the hash value, determine whether the target test data is repeated with the data in the pre-set blockchain database.

[0069] Optionally, the execution unit 505 is specifically configured to: The hash value and the meta information are sent to a designated node of the blockchain database, so that after the designated node verifies the hash value and the meta information, the hash value and the meta information are packaged and sent to all nodes of the blockchain database.

[0070] Optionally, the positioning unit 506 is specifically configured to: According to the target test data, locate the starting node in the pre-created knowledge graph, which is a knowledge graph constructed based on knowledge information in multiple actual application cases. The knowledge graph includes scenario information, capability information, task information, indicator information, and association relationship information applied to the multimodal large model; Identify capability information and task information of target application scenarios; Locate the corresponding task node according to the capability information and task information.

[0071] See also Figure 6 , the present application provides an electronic device, including: Processor 601 , memory 602 , input / output unit 603 , and bus 604 .

[0072] The processor 601 is connected to the memory 602 , the input and output unit 603 , and the bus 604 .

[0073] The memory 602 stores a program, and the processor 601 calls the program to execute the following steps: Figure 1 The automatic test evaluation construction method of the multimodal large model shown in the embodiment, Figure 2 The method for constructing an adversarial network architecture shown in the embodiment, Figure 3 The method for determining the repeatability of target test data shown in the embodiment or Figure 4 The method for locating the starting node and the task node shown in the embodiment.

[0074] The present application provides a computer-readable storage medium, on which a program is stored, and when the program is executed on a computer, the program performs the following steps: Figure 1 The automatic test evaluation construction method of the multimodal large model shown in the embodiment, Figure 2 The method for constructing an adversarial network architecture shown in the embodiment, Figure 3 The method for determining the repeatability of target test data shown in the embodiment or Figure 4 The method for locating the starting node and the task node shown in the embodiment.

[0075] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0076] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0077] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0078] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0079] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A method for constructing automatic test and evaluation of a multimodal large model, characterized in that: include: Acquire sample data, where the sample data is sample data screened from a public modality data resource library according to a data generation algorithm; Constructing an adversarial network architecture according to multiple modal combinations of a multimodal large model, wherein the multiple modal combinations are multiple combinations of at least two modal combinations; Inputting the sample data into the adversarial network architecture for training to obtain target test data; Determine whether the target test data is duplicated with data in a preset blockchain database; If not, storing the target test data in the blockchain database; According to the target application scenario of the multimodal large model and the target test data, locate the starting node and the task node corresponding to the target application scenario in the pre-created knowledge graph, wherein the target application scenario is an application scenario corresponding to the target test data; Acquire a target test indicator according to the starting node and the task node, and assign a weight to the target test indicator according to the target application scenario; Inputting the target test data into the multimodal large model for testing to obtain a test result; An evaluation report is generated based on the test results and the target test indicators.

2. The method for constructing an automatic test and evaluation system for a multimodal large model according to claim 1, characterized in that: The adversarial network architecture is constructed based on the combination of multiple modes of the multimodal large model, including: According to the free combination of the image modality and the text modality of the multimodal large model, a first generator combining a convolutional neural network and a recurrent neural network is set, wherein the first generator is used to process the data of the image modality and the text modality; According to the free combination of the video modality and the audio modality of the multimodal large model, a second generator combining a 3D convolutional neural network and an audio generative adversarial network is set, wherein the second generator is used to process the data of the video modality and the audio modality; According to the first generator, a first discriminator of a corresponding multi-layer perceptron architecture is set, wherein the first discriminator is used to discriminate the authenticity of the data of the image modality and the text modality; According to the second generator, a second discriminator of a corresponding audio and video discrimination network architecture is set, wherein the second discriminator is used to discriminate the authenticity of the data of the video modality and the audio modality; Adding a first optimization function in the first generator and the second generator respectively; Adding a second optimization function to the first discriminator and the second discriminator respectively; An adversarial network architecture is constructed according to the first generator, the second generator, the first discriminator and the second discriminator.

3. The method for constructing an automatic test and evaluation system for a multimodal large model according to claim 2, characterized in that: The first optimization function is: min =log(1-D i (G j (z))), where P min The probability that the data generated by the first generator or the second generator is judged as false by the discriminator, D i is the discriminator, G j is the generator, z is the input random noise or conditional vector, i,j={1,2}.

4. The method for constructing an automatic test and evaluation system for a multimodal large model according to claim 2, characterized in that: The second optimization function is: max =log(D(x))+log(1-D i (G j (z))), where P max is the probability that the first discriminator or the second discriminator correctly discriminates the authenticity, D i is the discriminator, G j is the generator, z is the input random noise or conditional vector, x is the input real data, i,j={1,2}.

5. The method for constructing an automatic test and evaluation system for a multimodal large model according to claim 1, characterized in that: The determining whether the target test data is repeated with the data in the preset blockchain database includes: The target test data is stored in a local temporary buffer, and a hash value of the target test data is calculated and corresponding meta information is generated; According to the hash value, it is determined whether the target test data is repeated with the data in the preset blockchain database.

6. The method for constructing an automatic test and evaluation system for a multimodal large model according to claim 5, characterized in that: The storing the target test data in the blockchain database comprises: The hash value and the meta-information are sent to a designated node of the blockchain database, so that after the designated node verifies the hash value and the meta-information, the hash value and the meta-information are packaged and sent to all nodes of the blockchain database.

7. The method for constructing an automatic test and evaluation system for a multimodal large model according to claim 1, characterized in that: The step of locating a starting node and a task node corresponding to the target application scenario in a pre-created knowledge graph according to the target application scenario of the multimodal large model and the target test data includes: According to the target test data, locate the starting node in a pre-created knowledge graph, wherein the pre-created knowledge graph is a knowledge graph constructed based on knowledge information in multiple actual application cases, and the knowledge graph includes scenario information, capability information, task information, indicator information, and association relationship information applied to a multimodal large model; Identify capability information and task information of the target application scenario; The corresponding task node is located according to the capability information and the task information.

8. An automatic test evaluation construction system for a multimodal large model, characterized in that: include: A sample data unit, used to obtain sample data, wherein the sample data is sample data screened from a public modality data resource library according to a data generation algorithm; An adversarial network architecture unit, used to construct an adversarial network architecture according to multiple modal combinations of the multimodal large model, wherein the multiple modal combinations are multiple combinations of at least two modal combinations; A training unit, used for inputting the sample data into the adversarial network architecture for training to obtain target test data; A judging unit, used to judge whether the target test data is repeated with data in a preset blockchain database; An execution unit, configured to store the target test data in the blockchain database if there is no duplication; A positioning unit, used to locate a starting node and a task node corresponding to the target application scenario in a pre-created knowledge graph according to the target application scenario of the multimodal large model and the target test data, wherein the target application scenario is an application scenario corresponding to the target test data; A target test indicator unit, used to obtain a target test indicator according to the start node and the task node, and assign a weight to the target test indicator according to the target application scenario; A testing unit, used for inputting the target test data into the multimodal large model for testing to obtain a test result; An evaluation unit is used to generate an evaluation report based on the test results and the target test indicators.

9. An electronic device, characterized in that: include: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the automatic test and evaluation construction method for a multimodal large model according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, and when the program is executed on a computer, the program executes the automatic test and evaluation construction method for a multimodal large model as claimed in any one of claims 1 to 7.

Images