Multi-modal model performance evaluation method and device, equipment and storage medium

Abstract

The invention discloses a multi-modal model performance evaluation method, device and equipment and a storage medium, and the method comprises the steps: obtaining camera picture data which comprises text data, image data and audio data; obtaining a real result text according to the picture instruction, and inputting the camera picture data and the test instruction into a to-be-tested multi-modal model to obtain a reasoning result text; respectively evaluating the text data, the image data and the audio data according to the reasoning result text and the real result text, and respectively obtaining a text fusion result, an image evaluation result and an audio evaluation result; and performing fusion by adopting an MLP algorithm, and outputting a final evaluation result. According to the method, the text data, the image data and the audio data of the multi-modal model are evaluated to obtain different evaluation results, the different evaluation results are fused, the final evaluation result is output, and automatic evaluation of the multi-modal model is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal model automated evaluation, and in particular to a multimodal model performance evaluation method, device, equipment and storage medium. Background Art

[0002] With the continuous advancement of computer technology and the rapid update of equipment, multimodal models have ushered in a period of vigorous development, and a large number of multimodal models have emerged. In view of this trend, the automated evaluation of the performance of multimodal models has become particularly important.

[0003] In the prior art, the evaluation method of multimodal models mainly relies on manual identification to judge the multimodal output results, that is, professionals need to check the pictures, audios and multimodal reasoning results one by one and compare them with the real results. This method is not only time-consuming and labor-intensive, but also prone to human errors. Therefore, how to automatically evaluate the performance of multimodal models is a problem that technicians in this field need to solve. Summary of the invention

[0004] The embodiments of the present invention provide a multimodal model performance evaluation method, apparatus, device and storage medium, aiming to solve the problem that the evaluation method of multimodal models in the prior art is time-consuming, labor-intensive and error-prone.

[0005] In a first aspect, an embodiment of the present invention provides a multimodal model performance evaluation method, comprising:

[0006] Acquire surveillance video and continuously extract frames to extract camera image data, wherein the camera image data includes text data, image data and audio data;

[0007] Acquire the real result text according to the image instruction, and input the camera image data and the test instruction into the multimodal model to be tested to obtain the inference result text, and save the inference result text and the real result text;

[0008] According to the inference result text and the true result text, the text data, the image data and the audio data are evaluated respectively, and a text fusion result, an image evaluation result and an audio evaluation result are obtained respectively;

[0009] The text fusion result, the image evaluation result and the audio evaluation result are fused by using the MLP algorithm to output the final evaluation result.

[0010] In a second aspect, an embodiment of the present invention provides a multimodal model performance evaluation device, comprising:

[0011] An acquisition unit, used to acquire surveillance video and continuously extract frames to extract camera picture data, wherein the camera picture data includes text data, image data and audio data;

[0012] An input unit, used to obtain a real result text according to the image instruction, and input the camera image data and the test instruction into the multimodal model to be tested to obtain an inference result text, and save the inference result text and the real result text;

[0013] An evaluation unit, used to evaluate the text data, image data and audio data respectively according to the inference result text and the true result text, and obtain a text fusion result, an image evaluation result and an audio evaluation result respectively;

[0014] The fusion unit is used to fuse the text fusion result, the image evaluation result and the audio evaluation result by using the MLP algorithm, and output a final evaluation result.

[0015] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multimodal model performance evaluation method as described in the first aspect is implemented.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor causes the processor to execute the multimodal model performance evaluation method as described in the first aspect.

[0017] The embodiment of the present invention provides a multimodal model performance evaluation method, device, equipment and storage medium, the method comprising: obtaining monitoring video and continuously extracting frames to extract camera image data, wherein the camera image data includes text data, image data and audio data; obtaining the real result text according to the image instruction, and inputting the camera image data and the test instruction into the multimodal model to be tested to obtain the inference result text, and saving the inference result text and the real result text; evaluating the text data, image data and audio data respectively according to the inference result text and the real result text, and obtaining the text fusion result, image evaluation result and audio evaluation result respectively; using the MLP algorithm to fuse the text fusion result, image evaluation result and audio evaluation result, and outputting the final evaluation result. The present invention evaluates the multimodal model text data, image data and audio data to obtain different evaluation results, and then uses the MLP algorithm to fuse the different evaluation results, outputs the final evaluation result, and realizes the automatic evaluation of the multimodal model, which not only reduces the need for manual intervention, reduces the possibility of human error, but also greatly improves the evaluation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0019] Figure 1 A schematic diagram of a flow chart of a multimodal model performance evaluation method provided by an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of a sub-process of a multimodal model performance evaluation method provided by an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of another sub-process of a multimodal model performance evaluation method provided by an embodiment of the present invention;

[0022] Figure 4 A schematic diagram of another sub-process of a multimodal model performance evaluation method provided by an embodiment of the present invention;

[0023] Figure 5 A schematic diagram of another sub-process of a multimodal model performance evaluation method provided by an embodiment of the present invention;

[0024] Figure 6 A schematic block diagram of an algorithm for multimodal model performance evaluation provided by an embodiment of the present invention;

[0025] Figure 7 A schematic block diagram of a multimodal model performance evaluation device provided by an embodiment of the present invention;

[0026] Figure 8 A schematic block diagram of another subunit of a multimodal model performance evaluation device provided by an embodiment of the present invention;

[0027] Fig. 9 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0030] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0031] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0032] See also Figure 1 , Figure 1 A schematic flow chart of a multimodal model performance evaluation method provided by an embodiment of the present invention, the method comprising steps S101 to S104:

[0033] S101, acquiring a surveillance video and continuously extracting frames to extract camera image data, wherein the camera image data includes text data, image data, and audio data;

[0034] In this embodiment, before evaluating the multimodal model, data preparation is required, mainly collecting camera image data. The camera image data can be collected by a camera installed at an appropriate position, or by other image acquisition devices.

[0035] Specifically, video sources can be obtained from a webcam, a local video file, or a streaming service. Programming languages ​​and libraries can be used to capture or read video data, such as OpenCV or ffmpeg libraries in Python. By reading each frame in the video data and continuously extracting frames as needed (such as at fixed intervals or fixed number of frames), the camera image data can be extracted. For text data, because the text data in the video source is usually not directly embedded in the video frame, it is necessary to extract it from the video data through OCR (optical character recognition) technology, and use OCR libraries (such as Python's Tesseract or pytesseract combined with OpenCV) to perform text recognition on each frame to obtain text data. For audio data, since it is impossible to directly extract complete audio data from a single frame of video (audio changes continuously over time, while a single frame of video is static), it is necessary to extract image data from the video frame and process the audio stream in the video to obtain audio features or text (such as through speech recognition). ffmpeg or Python's moviepy library can be used to extract audio to obtain audio data.

[0036] S102, obtaining a real result text according to the image instruction, and inputting the camera image data and the test instruction into the multimodal model to be tested to obtain an inference result text, and saving the inference result text and the real result text;

[0037] In this embodiment, first, the test instruction can clearly describe the task or object to be identified in the camera image data, and the test instruction needs to match the camera image data. For each test instruction and the corresponding camera image data, there needs to be a real, predefined result text for subsequent comparison with the inference result text.

[0038] Next, make sure the multimodal model to be tested has been trained and is ready to accept input data. Input the test instructions and the corresponding camera image data into the multimodal model. Let the multimodal model perform inference based on the input camera image data and test instructions to generate inference result text. Finally, save the inference result text generated by the multimodal model so that it can be compared with the real result text later to evaluate the performance of the multimodal model.

[0039] S103, evaluating the text data, image data, and audio data respectively according to the inference result text and the true result text, and obtaining a text fusion result, an image evaluation result, and an audio evaluation result respectively;

[0040] In this embodiment, the model evaluation of the multimodal model is divided into three parts: text, image and audio. The first part is text evaluation, which consists of a word frequency analysis module, a vector encoding module and a large model evaluation module; the second part is image evaluation, and the third part is audio evaluation. The text data, image data and audio data are evaluated respectively according to the inference result text and the true result text, and the text fusion result, image evaluation result and audio evaluation result are obtained respectively.

[0041] In one embodiment, if Figure 2 As shown, the step S103 includes steps S201 to S204:

[0042] S201, preprocessing the text data, separating the text data according to punctuation marks, and dividing the text data into words;

[0043] In this embodiment, preprocessing text data, especially separating text data into words (or word segmentation) according to punctuation marks, is a basic step in natural language processing (NLP). This process helps subsequent tasks such as text analysis, information extraction, and machine learning model training.

[0044] Specifically, before word segmentation, it may be necessary to remove useless characters in the text data, such as extra spaces, tabs, line breaks, etc., as well as possible HTML tags or special symbols (if they do not affect the word segmentation step by punctuation, this step can be omitted). Convert the text data to a unified format, such as converting all characters to lowercase (or uppercase), which facilitates subsequent processing and comparison. Finally, separate the text data into words or phrases according to punctuation.

[0045] The inference result text and the real result text are simultaneously input into the word frequency analysis module, the vector encoding module and the large model evaluation module to obtain the results M1, M2 and M3 respectively, and then M1, M2 and M3 are input into the text fusion module to obtain the text fusion result T1. The target detection box information in the inference result text is extracted using the regular expression method, and then input into the IOU calculation module to obtain the image evaluation result T2.

[0046] S202, calculating the frequency of each word in the inference result text appearing in the real result text according to the following formula:

[0047]

[0048] In this embodiment, the word frequency analysis module adopts the TF-IDF method, which is divided into two parts. First, the word frequency (TF) is calculated, that is, the frequency of each word in the inference result text appearing in the real result text.

[0049] TF-IDF is a statistical method used to evaluate the importance of a word to a document set or a document in a corpus. It mainly consists of two parts: TF (Term Frequency) and IDF (Inverse Document Frequency).

[0050] S203, calculating the inverse document frequency of the word in the text data according to the following formula:

[0051]

[0052] In this embodiment, another part of the word frequency analysis module is to calculate the inverse document frequency (IDF), which calculates the rarity of a word in the entire text data. If a word appears frequently in the inference result text and rarely appears in the real result text, it is considered that the word has good category distinction ability and is suitable for classification.

[0053] S204 , multiplying the frequency by the inverse document frequency according to the following formula to obtain a word frequency analysis module result: TF-IDF(t,d)=TF(t,d)×IDF(t).

[0054] In this embodiment, the TF-IDF value is the product of TF and IDF, that is, this result is the similarity between the inference result text and the true result text. The larger the TF-IDF value, the higher the similarity between the inference result text and the true result text. The result is recorded as M1.

[0055] Further, such as Figure 3 As shown, the step S103 also includes steps S301 to S303:

[0056] S301, input the words into the Token Embeddings model, the Segment Embeddings model and the Position Embeddings model respectively to obtain word vectors, sentence vectors and word position vectors;

[0057] In this embodiment, Token Embeddings is a pre-built vocabulary, Segment Embeddings is a model used to distinguish the inference result text from the real result text, all word vectors in the inference result text are set to 0, and all vectors of the real result text are set to 1, and Position Embeddings is to positionally encode the position of the word vector, where the position encoding implementation formula is as follows:

[0058]

[0059] Among them, pos represents the position of the word, i represents the dimension of embedding, and dmodel represents the dimension of the word vector.

[0060] S302, inputting the word vector, sentence vector and word position vector into the Bert model to obtain the inference text vector C1 and the true result text vector C2;

[0061] In this embodiment, the word vector, sentence vector and word position vector are merged, [CLS] is added at the first position, and [SEP] is added at the space and end of the text data. Then, it is input into the Bert network model for vectorization to obtain the inference text vector C1 and the real result text vector C2. Then, the similarity between the two is calculated according to the following formula. Specifically, the Bert network structure is as follows:

[0062]

[0063] Among them, A and B represent the inference text vector C1 and the true result text vector C2 respectively, n represents the dimension of the vector, and the similarity value ranges from [-1,1], where -1 represents complete dissimilarity and 1 represents complete similarity.

[0064] S303, calculating the similarity between the inference result text vector C1 and the real result text vector C2, and obtaining a vector encoding module result.

[0065] In this embodiment, the similarity between the inference result text vector C1 and the true result text vector C2 is calculated according to step S302 to obtain the vector encoding module result, which is recorded as M2.

[0066] Further, such as Figure 4 As shown, the step S103 also includes steps S401 to S403:

[0067] S401, converting the inference result text and the real result text into an input format that meets the requirements of the multimodal model to be tested;

[0068] S402, call the API big model and enter prompt;

[0069] S403. Obtain the similarity between the inference result text and the real result text according to the API big model to obtain a big model calculation evaluation module.

[0070] In this embodiment, the multimodal model is called by calling the API method. Before the call, first, the inference result text and the real result text are processed and adjusted to the input format that meets the requirements of the multimodal model to be tested. On this basis, a prompt for asking about the text similarity is added. For example, the prompt can be "Judge the similarity between these two texts, and output a score between 0 and 1", etc. The similarity between the real result text and the inference result text is obtained by calling the API method, and the result is recorded as M3.

[0071] Furthermore, the step S103 further includes step S501:

[0072] S501, fusing the word frequency analysis module result, the vector encoding module result and the large model calculation evaluation module to obtain the text fusion result.

[0073] In this embodiment, the PCA principal component analysis method is used to fuse M1, M2 and M3. First, the variance of M1, M2 and M3 is calculated, and the formula is as follows:

[0074]

[0075] Among them, m represents the number of M, M represents M1, M2 and M3, and μ represents the average value of M. In order to facilitate subsequent calculations, the results of M1, M2 and M3 are decentralized, and the formula is as follows:

[0076]

[0077] Where X = [M1, M2, M3]. Next, find a direction in the text data space that maximizes the variance of the projection of each data. Let this direction be w, ||w||2 = 1, and then the coordinate value of each data in this direction is equal to w T M i , the variance formula under the decentralized and transformed coordinate axes is as follows:

[0078]

[0079] Wherein, T represents text data;

[0080] Will Recorded as C, the optimization objective is transformed into the following form:

[0081]

[0082] To solve this problem, use the Lagrange constant method to construct the function:

[0083] L(w,λ)=w T Cw+λ(1-wT w), where λ represents the Lagrange multiplier, which is a parameter introduced to serve as a medium or bridge when calculating conditional extreme values.

[0084] Then take the derivative of each component:

[0085]

[0086] The solution is:

[0087]

[0088] Substitute into the objective function:

[0089] max D(M)=max{w T Cw}=max{w T λw}=maxλ

[0090] It can be seen from the formula that the maximum variance is the maximum eigenvalue, so the eigenvector corresponding to the maximum eigenvalue is the direction of the new coordinate axis that needs to be found. Then, the text data is projected onto this coordinate axis to obtain the text fusion result after dimensionality reduction, which is recorded as T1.

[0091] Further, such as Figure 5 As shown, the step S103 also includes steps S601 to S603:

[0092] S601, extracting target detection information from the image data using a regular expression matching method according to the inference result text;

[0093] S602, inputting the target detection information and the real result text into an IOU calculation module, and calculating the number of labels in the inference result text and the real result text respectively;

[0094] S603: Calculate the difference between the number of labels in the inference result text and the number of labels in the true result text to obtain the image evaluation result.

[0095] In this embodiment, the IOU calculation module is divided into two parts: a regular expression matching module and an IOU calculation module. The regular expression matching module is implemented using the KMP algorithm to match " <ref> people< / ref> <box> (543,432),(723,729)< / box> The "field describes the three stages of the algorithm implementation, which uses regular expression matching to extract object detection information from image data:

[0096] (1) Preprocessing: First, build a partial match table. For each character in the pattern string, find its longest common prefix and suffix. <ref>and, and record this length in the partial match table.

[0097] (2) Matching: Start from the first character of the text string and match it with the first character of the pattern string. If the match is successful, continue to match the next character; if it does not match, use the partial match table to determine the next matching position.

[0098] (3) Backtracking: If the match fails at a certain position, the next matching position is determined based on the partial match table. For each character in the pattern string, we find the position where it most recently appears in the text string and then start matching from this position.

[0099] The KMP algorithm is used to extract the detection box information in the inference result text, and this information and the information in the real result text are simultaneously input into the IOU calculation module to calculate the number of values ​​greater than the threshold and with the same label. The calculation method of this module is as follows:

[0100] (1) Calculate the intersection area: First, calculate the intersection area of ​​the two boxes (the two boxes represent the information of the inference result text and the information of the true result text, respectively). The intersection area is the area of ​​the overlapping part of the two boxes.

[0101] (2) Calculate the union area: Then calculate the union area of ​​the two boxes. The union area is the sum of the total areas of the two boxes minus their intersection area.

[0102] (3) Calculate IOU: Finally, divide the intersection area by the union area to get the IOU value

[0103] The calculation formula is as follows:

[0104]

[0105] Among them, xmin A and ymin A Indicates the coordinates of the upper left corner of box A, xmax B and ymax B represents the coordinate of the lower right corner of box B, A and B represent the area of ​​box A and box B respectively. Among them, box A and box B also represent the information of the inference result text and the information of the true result text respectively.

[0106] After calculating the IOU value of the box, count the boxes that are greater than the threshold and record the labels corresponding to the boxes, that is, the number of labels in the inference result text and the number of labels in the true result text. Finally, compare the difference between the number of labels in the inference result text and the number of labels in the true result text, and record the value as T2.

[0107] Furthermore, the step S103 further includes steps S701 to S702:

[0108] S701, preprocessing the audio data according to the inference result text, and converting the audio data into a spectrogram;

[0109] In this embodiment, the audio data is preprocessed to convert the audio data into a spectrogram, and this process is divided into the following steps:

[0110] (1) Pre-emphasis:

[0111] The input audio data is pre-emphasized to enhance the high-frequency components. The formula is:

[0112] y[n]=x[n]-αx[n-1]

[0113] Wherein, α is a constant of 0.95, y[n] represents the pre-emphasized signal; x[n] represents the original input audio signal; and n represents the index of the time sample.

[0114] (2) Framing and windowing:

[0115] The audio data is divided into overlapping frames and a window function is applied to each frame. Assuming the frame length is N, the frame shift is M, and the window function is ω(n), the signal of the kth frame is:

[0116] x k [n] = x[KM+n]·ω(n)

[0117] Among them, x k [n] represents the signal of the Kth frame; x represents the pre-emphasized signal, k represents the index of the frame; and n represents the index of the sample in the frame.

[0118] (3) Fast Fourier Transform (FFT)

[0119] Apply FFT to each frame to convert the signal from time domain to frequency domain. Let the frame signal be x k [n], its FFT is:

[0120] Among them, X k [m] represents the complex value of the mth frequency component of the kth frame; x k [n] The signal of the kth frame; N represents the number of sampling points per frame; j represents the imaginary unit; represents a complex factor; m represents the index of the frequency component, m=0,1,…,N-1.

[0121] (4) Power spectrum

[0122] Calculate the power spectrum of each frame, defined as:

[0123]

[0124] Among them, P k [m] represents the power spectrum value of the mth frequency component of the kth frame; |X k [m]| 2 Represents the square of the amplitude of the mth frequency component of the kth frame; N represents the number of sampling points per frame.

[0125] (5) Mel filter bank

[0126] The power spectrum is filtered through a Mel filter bank, which is a set of triangular filters built on the Mel frequency scale and is defined as:

[0127]

[0128] (6) Logarithmic transformation

[0129] Take the logarithm of the output of each Mel filter to get the logarithmic Mel spectrum:

[0130]

[0131] Among them, logS k [i] represents the logarithmic output of the i-th Mel filter of the k-th frame; represents the weighted sum of the i-th Mel filter in the k-th frame.

[0132] (7) Discrete Cosine Transform (DCT)

[0133] Apply discrete cosine transform to the logarithmic Mel spectrum to obtain the Mel frequency cepstrum coefficients. Let the logarithmic Mel spectrum be logS k [i], then the logarithmic Mel spectrum is:

[0134]

[0135] Among them, L is generally a constant 12; c k [n] represents the nth MFCC coefficient of the kth frame; logS k [i represents the logarithmic output of the i-th Mel filter of the k-th frame; M represents the number of Mel filters; n represents the index of the MFCC coefficient, n = 0, 1, ..., L-1; Represents the elements of the DCT transform matrix.

[0136] (8) Feature stitching

[0137] Considering the continuity of speech, adjacent frames are concatenated to form a spectrogram.

[0138] S702, using the DFCNN model to splice the spectrograms in time sequence to obtain an audio vector C3;

[0139] The similarity between the audio vector C3 and the true result text vector C2 is calculated according to the similarity calculation formula to obtain the audio evaluation result; the similarity calculation formula is:

[0140]

[0141] Among them, A represents the true result text vector C2, B represents the audio vector C3, and n represents the dimension of the vector.

[0142] In this embodiment, the spectrogram is input into the DFCNN network model, which is composed of three parts: input layer, hidden layer and output layer.

[0143] The input layer structure is: the feature dimension is 12, the number of MFCCs is 12, and each input contains 100 time steps.

[0144] The hidden layer structure is:

[0145] Layer 1: Fully connected layer, containing 512 neurons, ReLU activation function;

[0146] Layer 2: Fully connected layer, containing 512 neurons, ReLU activation function;

[0147] Layer 3: Fully connected layer, containing 256 neurons, ReLU activation function;

[0148] Layer 4: Fully connected layer, containing 256 neurons and ReLU activation function.

[0149] The output layer structure is: output category number of neurons, using softmax activation function

[0150] By processing the spectrograms one by one, the processed results are finally spliced ​​into an audio vector C3 in time sequence. Then, the similarity calculation formula (such as the formula in step S302) is used to obtain the similarity result between the audio vector C3 and the true result vector C2. This result is the result of the audio analysis module, recorded as T3.

[0151] S104: Using the MLP algorithm to fuse the text fusion result, the image evaluation result and the audio evaluation result, and outputting a final evaluation result.

[0152] In this embodiment, the result fusion module fuses the text fusion result T1, the image evaluation result T2 and the audio evaluation result T3, with the aim of evaluating the performance of the model in multiple dimensions.

[0153] This module is implemented using the MLP algorithm, and its principles are as follows:

[0154] MLP is constructed by input layer, hidden layer and output layer. Each layer is fully connected, that is, any neuron in the previous layer is connected to all neurons in the next layer. The input layer is used to receive the input matrix X and generate corresponding neurons according to the X dimension. The hidden layer is connected to the input layer, and the input vector is represented by X. The output of the hidden layer can be expressed as:

[0155] f(ωx+b)

[0156] In the expression, ω represents the weight, which uses the official model weight, x represents the input vector of the input layer, b represents the bias, and the function f is the sigmoid function, whose function expression is:

[0157]

[0158] The output layer is calculated in the same way as the hidden layer. It is connected to each neuron in the hidden layer. Its output is also expressed as f(ωx+b). It also passes through the sigmoid activation function to output the final evaluation result.

[0159] In summary, if Figure 6 As shown, an algorithm flow chart for multimodal model performance evaluation is shown, which realizes the automated evaluation of multimodal models. The specific steps are as described above and will not be described in detail here.

[0160] like Figure 7 As shown, an embodiment of the present invention further provides a multimodal model performance evaluation device 800, including an acquisition unit 801, an input unit 802, an evaluation unit 803, and a fusion unit 804.

[0161] The acquisition unit 801 is used to acquire the surveillance video and continuously extract frames to extract camera picture data, wherein the camera picture data includes text data, image data and audio data;

[0162] An input unit 802 is used to obtain a real result text according to the image instruction, and input the camera image data and the test instruction into the multimodal model to be tested to obtain an inference result text, and save the inference result text and the real result text;

[0163] An evaluation unit 803, used to evaluate the text data, image data and audio data respectively according to the inference result text and the true result text, and obtain a text fusion result, an image evaluation result and an audio evaluation result respectively;

[0164] The fusion unit 804 is used to fuse the text fusion result, the image evaluation result and the audio evaluation result by using the MLP algorithm, and output a final evaluation result.

[0165] In one embodiment, if Figure 8 As shown, the evaluation unit 803 includes:

[0166] A conversion unit 901 is used to pre-process the audio data according to the inference result text and convert the audio data into a spectrogram;

[0167] A concatenation unit 902 is used to concatenate the spectrograms in time sequence using a DFCNN model to obtain an audio vector C3;

[0168] The similarity between the audio vector C3 and the true result text vector C2 is calculated according to the similarity calculation formula to obtain the audio evaluation result; the similarity calculation formula is:

[0169]

[0170] Among them, A represents the true result text vector C2, B represents the audio vector C3, and n represents the dimension of the vector.

[0171] The multimodal model performance evaluation device provided in the embodiment of the present invention evaluates the multimodal model text data, image data and audio data to obtain different evaluation results, and then uses the MLP algorithm to fuse the different evaluation results to output the final evaluation result, thereby realizing the automatic evaluation of the multimodal model, which not only reduces the need for manual intervention and the possibility of human error, but also greatly improves the evaluation efficiency.

[0172] The embodiment of the present invention also provides a multimodal model performance evaluation device which can be implemented in the form of a computer program. The computer program can be used in Fig. 9 Runs on the computer device shown.

[0173] See also Fig. 9 , Fig. 9 1 is a schematic block diagram of a computer device provided by an embodiment of the present invention. The computer device 1000 is a server, which may be an independent server or a server cluster composed of multiple servers.

[0174] See also Fig. 9 The computer device 1000 includes a processor 1002 , a memory and a network interface 1005 connected via a system bus 1001 , wherein the memory may include a non-volatile storage medium 1003 and an internal memory 1004 .

[0175] The non-volatile storage medium 1003 may store an operating system 10031 and a computer program 10032. When the computer program 10032 is executed, the processor 1002 may execute a multi-modal model performance evaluation method.

[0176] The processor 1002 is used to provide computing and control capabilities to support the operation of the entire computer device 1000.

[0177] The internal memory 1004 provides an environment for the operation of the computer program 10032 in the non-volatile storage medium 1003. When the computer program 10032 is executed by the processor 1002, the processor 1002 can execute the multimodal model performance evaluation method.

[0178] The network interface 1005 is used for network communication, such as providing data information transmission, etc. Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device 1000 to which the solution of the present invention is applied. The specific computer device 1000 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0179] Those skilled in the art will understand that Fig. 9 The embodiments of the computer device shown in the figure do not constitute a limitation on the specific composition of the computer device. In other embodiments, the computer device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, in some embodiments, the computer device may only include a memory and a processor. In such embodiments, the structure and function of the memory and the processor are the same as those of the embodiment of the present invention. Fig. 9 The embodiments shown are consistent and will not be described again here.

[0180] It should be understood that in the embodiment of the present invention, the processor 1002 may be a central processing unit (CPU), and the processor 1002 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0181] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the above-described equipment, devices and units can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. Those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0182] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, and units with the same function may be combined into one unit, for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0183] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0184] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0185] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), disk or optical disk and other media that can store program codes.

[0186] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.< / ref>

Claims

1. A multimodal model performance evaluation method, characterized in that: include: Acquire surveillance video and continuously extract frames to extract camera picture data, wherein the camera picture data includes text data, image data and audio data; Acquire the real result text according to the image instruction, and input the camera image data and the test instruction into the multimodal model to be tested to obtain the inference result text, and save the inference result text and the real result text; According to the inference result text and the true result text, the text data, the image data and the audio data are evaluated respectively, and a text fusion result, an image evaluation result and an audio evaluation result are obtained respectively; The text fusion result, the image evaluation result and the audio evaluation result are fused by using the MLP algorithm to output the final evaluation result.

2. The multimodal model performance evaluation method according to claim 1, characterized in that: The text data, image data and audio data are evaluated respectively according to the inference result text and the true result text, and a text fusion result, an image evaluation result and an audio evaluation result are obtained respectively, including: Preprocessing the text data, separating the text data according to punctuation marks, and dividing the text data into words; The frequency of each word in the inference result text appearing in the true result text is calculated according to the following formula: The inverse document frequency of the word in the text data is calculated according to the following formula: The frequency is multiplied by the inverse document frequency according to the following formula to obtain the word frequency analysis module result: TF-IDF(t,d)=TF(t,d)×IDF(t).

3. The multimodal model performance evaluation method according to claim 2, characterized in that: The text data, image data and audio data are evaluated respectively according to the inference result text and the true result text, and a text fusion result, an image evaluation result and an audio evaluation result are obtained respectively. Including, also including: Input the words into the Token Embeddings model, the Segment Embeddings model, and the Position Embeddings model respectively to obtain word vectors, sentence vectors, and word position vectors; Input the word vector, sentence vector and word position vector into the Bert model to obtain the inference result text vector C1 and the true result text vector C2; The similarity between the inference result text vector C1 and the true result text vector C2 is calculated to obtain the vector encoding module result.

4. The multimodal model performance evaluation method according to claim 3, characterized in that The text data, image data and audio data are evaluated respectively according to the inference result text and the true result text, and a text fusion result, an image evaluation result and an audio evaluation result are obtained respectively, and further includes: Convert the inference result text and the true result text into an input format that meets the requirements of the multimodal model to be tested: Call the API model and enter the prompt; The similarity between the inference result text and the real result text is obtained according to the API big model to obtain a big model calculation evaluation module.

5. The multimodal model performance evaluation method according to claim 4, characterized in that: The text data, image data and audio data are evaluated respectively according to the inference result text and the true result text, and a text fusion result, an image evaluation result and an audio evaluation result are obtained respectively, and further includes: The word frequency analysis module result, the vector encoding module result and the large model calculation evaluation module are integrated to obtain the text fusion result.

6. The multimodal model performance evaluation method according to claim 1, characterized in that: The text data, image data and audio data are evaluated respectively according to the inference result text and the true result text, and a text fusion result, an image evaluation result and an audio evaluation result are obtained respectively, including: According to the inference result text, a regular expression matching method is used to extract the target detection information in the image data; Input the target detection information and the real result text into the IOU calculation module, and calculate the number of labels in the inference result text and the real result text respectively; The difference between the number of labels in the inference result text and the number of labels in the true result text is calculated to obtain the image evaluation result.

7. The multimodal model performance evaluation method according to claim 3, characterized in that: The text data, image data and audio data are evaluated respectively according to the inference result text and the true result text, and a text fusion result, an image evaluation result and an audio evaluation result are obtained respectively, including: Preprocessing the audio data according to the inference result text to convert the audio data into a spectrogram; The DFCNN model is used to splice the spectrogram in time sequence to obtain an audio vector C3; The similarity between the audio vector C3 and the true result text vector C2 is calculated according to the similarity calculation formula to obtain the audio evaluation result; the similarity calculation formula is: Among them, A represents the true result text vector C2, B represents the audio vector C3, and n represents the dimension of the vector.

8. A multimodal model performance evaluation device, characterized in that: include: An acquisition unit, used to acquire surveillance video and continuously extract frames to extract camera picture data, wherein the camera picture data includes text data, image data and audio data; An input unit, used to obtain a real result text according to the image instruction, and input the camera image data and the test instruction into the multimodal model to be tested to obtain an inference result text, and save the inference result text and the real result text; An evaluation unit, used to evaluate the text data, image data and audio data respectively according to the inference result text and the true result text, and obtain a text fusion result, an image evaluation result and an audio evaluation result respectively; The fusion unit is used to fuse the text fusion result, the image evaluation result and the audio evaluation result by using the MLP algorithm, and output a final evaluation result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the multimodal model performance evaluation method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the multimodal model performance evaluation method according to any one of claims 1 to 7.

Images