Multimedia data detection method and device, equipment and storage medium
By receiving target instructions and combining the background information and feature information of the multimedia data to perform detection, the problem of difficulty in accurately judging the authenticity of multimedia data in the prior art is solved, and higher detection accuracy and reliability are achieved.
Patent Information
- Application Number
- CN202411733578.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-28
- Publication Date
- 2025-04-01
AI Technical Summary
The existing multimedia data forgery detection technology is difficult to accurately judge the knowledge that has not been learned, resulting in the inability to accurately judge the authenticity of multimedia data.
By receiving target instructions, the text information and background information of the multimedia data are determined, and the background information and feature information of the multimedia data are combined to perform detection. The specific method includes determining the fusion feature information, performing normalization processing, calculating the cosine similarity to judge the authenticity of the multimedia data.
By combining the background information and feature information of the multimedia data, the authenticity of the multimedia data can be more accurately judged and the accuracy and reliability of detection can be improved.
Smart Images
Figure CN119739876A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of video detection technology, and in particular to a method, device, equipment and storage medium for detecting multimedia data. Background Art
[0002] With the advancement of digital photography, image processing, speech synthesis and other technologies, it is becoming easier to create fake multimedia content. For example, images can be modified and synthesized through image processing software, or false speech content can be created using speech synthesis technology. Current multimedia forgery detection technologies can be divided into two categories: traditional methods and deep learning methods. Traditional methods mainly use time domain and frequency domain analysis methods for detection, or manually extract features, and then use machine learning methods for training. Deep learning methods mainly split multimedia into audio and video frames, train models for them separately, and finally merge the detection results.
[0003] However, existing detection technologies are prone to generate erroneous detection results for unlearned knowledge, resulting in an inability to accurately determine the authenticity of multimedia data. Summary of the invention
[0004] The present application provides a multimedia data detection method, apparatus, device and storage medium, which can accurately detect the authenticity of multimedia data.
[0005] In order to achieve the above objectives, this application adopts the following technical solutions:
[0006] In a first aspect, the present application provides a method for detecting multimedia data, the method comprising: receiving a target instruction; the target instruction is used to verify the authenticity of the multimedia data; determining text information corresponding to the multimedia data, and determining background information matching the multimedia data based on the text information; the background information is used to reflect the source of the multimedia data; based on the background information of the multimedia data and the feature information of the multimedia data, detecting the multimedia data; the feature information is used to represent the key elements of the multimedia data.
[0007] According to the above technical means, the target instruction can be received and the text information corresponding to the multimedia data can be determined, and the background information matching the multimedia data can be determined based on the text information. The multimedia data can be detected based on the background information of the multimedia data and the characteristic information of the multimedia data. Therefore, the authenticity of the multimedia data can be more accurately determined by combining the background information of the multimedia data and the characteristic information of the multimedia data.
[0008] In a possible implementation, the above-mentioned "detecting multimedia data based on the background information of the multimedia data and the characteristic information of the multimedia data" includes: determining fused characteristic information of the multimedia data based on the background information of the multimedia data and the characteristic information of the multimedia data; the fused characteristic information includes association information between the background information of the multimedia data and the key elements of the characteristic information of the multimedia data; detecting the multimedia data based on the fused characteristic information of the multimedia data.
[0009] According to the above technical means, the fusion feature information of the multimedia data can be determined based on the background information of the multimedia data and the feature information of the multimedia data, and the multimedia data can be detected based on the fusion feature information of the multimedia data. Thus, the association information of the key elements of the background information of the multimedia data and the feature information of the multimedia data can be accurately determined, and the authenticity of the detected multimedia data can be accurately determined based on the association information.
[0010] In a possible implementation, the above-mentioned "fusion feature information of multimedia data includes first fusion feature information and second fusion feature information, wherein the first fusion feature information represents common feature information of background information of multimedia data and feature information of multimedia data; the second fusion feature information includes key elements of multimedia data, and the multimedia data is detected based on the fusion feature information of multimedia data", includes: normalizing the first fusion feature information and the second fusion feature information, determining the cosine similarity of the first fusion feature information and the second fusion feature information; and detecting the multimedia data based on the cosine similarity of the first fusion feature information and the second fusion feature information.
[0011] According to the above technical means, the first fused feature information and the second fused feature information can be normalized to accurately determine the cosine similarity of the first fused feature information and the second fused feature information. Based on the cosine similarity of the first fused feature information and the second fused feature information, multimedia data can be detected more reliably.
[0012] In a possible implementation, when the cosine similarity of the first fused feature information and the second fused feature information is within a preset range, the multimedia data is determined to be real data; when the cosine similarity of the first fused feature information and the second fused feature information is outside the preset range, the multimedia data is determined to be abnormal data.
[0013] According to the above technical means, it is possible to accurately determine whether the multimedia data is real data or abnormal data based on the specific values of the cosine similarity of the first fused feature information and the second fused feature information.
[0014] In a possible implementation, the above-mentioned "determining the fusion feature information of multimedia data" includes: determining the background information vector corresponding to the multimedia data and the feature information vector corresponding to the multimedia data; based on the background information vector and the feature information vector, determining the first fusion feature vector and the second fusion feature vector of the multimedia data.
[0015] According to the above technical means, the background information vector corresponding to the multimedia data and the feature information vector corresponding to the multimedia data can be determined, and based on the background information vector and the feature information vector, the first fused feature vector and the second fused feature vector of the multimedia data can be accurately determined.
[0016] In a possible implementation, the above-mentioned "determining the first fused feature vector and the second fused feature vector of the multimedia data based on the background information vector and the feature information vector" includes: vector concatenating the background information vector and the feature information vector to generate a first fused feature vector; extracting key information of the feature information vector as the second fused feature vector.
[0017] According to the above technical means, the fusion feature information can be determined more accurately in the form of vectors.
[0018] In a possible implementation, the cosine similarity satisfies a first preset formula, which is:
[0019]
[0020] Among them, S(x,v) represents cosine similarity, f(x) represents the first fused feature vector, g(v) represents the second fused feature vector, ||f(x)|| represents the norm of the first fused feature vector, and ||g(v)|| represents the norm of the second fused feature vector.
[0021] According to the above technical means, the cosine similarity of the first fused feature information and the second fused feature information can be determined more conveniently and accurately based on the first preset formula.
[0022] In a second aspect, the present application provides a multimedia data detection device, which includes: a receiving unit, used to receive a target instruction; the target instruction is used to verify the authenticity of the multimedia data; a determination unit, used to determine the text information corresponding to the multimedia data, and determine the background information matching the multimedia data based on the text information; the background information is used to reflect the source of the multimedia data; a detection unit, used to detect the multimedia data based on the background information of the multimedia data and the feature information of the multimedia data; the feature information is used to represent the key elements of the multimedia data.
[0023] In one possible implementation, the detection unit is specifically used to determine fused feature information of the multimedia data based on the background information of the multimedia data and the feature information of the multimedia data; the fused feature information includes association information between the background information of the multimedia data and key elements of the feature information of the multimedia data; and the multimedia data is detected based on the fused feature information of the multimedia data.
[0024] In a possible implementation, the detection unit is further used to normalize the first fused feature information and the second fused feature information, determine the cosine similarity of the first fused feature information and the second fused feature information; and detect the multimedia data based on the cosine similarity of the first fused feature information and the second fused feature information.
[0025] In a possible implementation, the determination unit is also specifically used to determine that the multimedia data is real data when the cosine similarity of the first fused feature information and the second fused feature information is within a preset range; and to determine that the multimedia data is abnormal data when the cosine similarity of the first fused feature information and the second fused feature information is outside the preset range.
[0026] In a possible implementation, the determination unit is further specifically used to determine a background information vector corresponding to the multimedia data and a feature information vector corresponding to the multimedia data; based on the background information vector and the feature information vector, determine a first fused feature vector and a second fused feature vector of the multimedia data.
[0027] In a possible implementation, the determination unit is further configured to perform vector concatenation on the background information vector and the feature information vector to generate a first fused feature vector; and extract key information of the feature information vector as a second fused feature vector.
[0028] In a possible implementation, the determination unit is further configured to determine cosine similarity, where the cosine similarity satisfies a first preset formula:
[0029]
[0030] Among them, S(x,v) represents cosine similarity, f(x) represents the first fused feature vector, g(v) represents the second fused feature vector, ||f(x)|| represents the norm of the first fused feature vector, and ||g(v)|| represents the norm of the second fused feature vector.
[0031] In a third aspect, the present application provides an electronic device comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement the method of the above-mentioned first aspect and any possible implementation manner thereof.
[0032] In a fourth aspect, the present application provides a computer-readable storage medium. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the method in the above-mentioned first aspect and any possible implementation method thereof.
[0033] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a communication device, enables the communication device to execute the method in the above-mentioned first aspect and any possible implementation manner thereof. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 A schematic diagram of a multimedia data detection system provided in an embodiment of the present application;
[0035] Figure 2 A flowchart of a method for detecting multimedia data provided in an embodiment of the present application;
[0036] Figure 3 A flowchart of another multimedia data detection method provided in an embodiment of the present application;
[0037] Figure 4 A flow chart of a method for training a priori knowledge model provided in an embodiment of the present application;
[0038] Figure 5 A flow chart of a method for training a feature alignment model provided in an embodiment of the present application;
[0039] Figure 6 A schematic diagram of the structure of a multimedia data detection device provided in an embodiment of the present application;
[0040] Figure 7 A block diagram of an electronic device of a multimedia data detection device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0041] The following is a detailed description of a multimedia data detection method, device, equipment and storage medium provided in an embodiment of the present application in conjunction with the accompanying drawings.
[0042] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.
[0043] The terms "first" and "second" and the like in the specification and drawings of this application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.
[0044] In addition, the terms "including" and "having" and any variations thereof mentioned in the description of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.
[0045] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0046] At present, multimedia data forgery detection technology can be divided into two categories: traditional methods and deep learning methods. Among them, traditional methods mainly use time domain and frequency domain analysis methods for detection, or manually extract features, and then use machine learning methods for training. Deep learning methods mainly split multimedia data into audio and video frames, and train models for them separately, and finally merge the detection results.
[0047] For multimedia data forgery detection technology, most of the existing technologies split the multimedia data and only consider the authenticity detection of one level, such as audio detection, video single frame detection, metadata analysis, etc. Even if they are combined for authentication, they are hard-connected and the results are accumulated. They do not use the more significant forgery features of the whole to play an additional role beyond the accumulation of the parts, resulting in the problem of difficulty in identification when facing highly authentic and realistic forgeries. To solve the above problems, it is necessary to consider the correlation between the various parts, explore the possible corresponding relationships between the various parts in the sample, extract the forgery defects between the samples, and combine the forgery features of each part to authenticate the multimedia data.
[0048] Therefore, the current multimedia data forgery detection technology has certain limitations and is difficult to deal with highly complex and sophisticated forgery methods. In addition, some existing technologies require manual intervention or professional analysis and lack the characteristics of automation and intelligence, resulting in low detection efficiency and high cost. High-quality multimedia data forgeries often have high authenticity and realism, which are difficult to accurately identify through existing analysis methods. In addition, existing detection technologies are prone to generate erroneous detection results for knowledge that has not been learned, resulting in an inability to accurately judge the authenticity of multimedia data.
[0049] In order to solve the above technical problems, the present application provides a method for detecting multimedia data. The method includes: receiving a target instruction; the target instruction is used to verify the authenticity of the multimedia data; determining the text information corresponding to the multimedia data, and determining the background information matching the multimedia data based on the text information; and detecting the multimedia data based on the background information of the multimedia data and the feature information of the multimedia data.
[0050] It should be noted that the multimedia data detection method provided in the embodiment of the present application can be executed by a server or by a terminal device. The following takes a server as an example to describe in detail the multimedia data detection method provided in the embodiment of the present application.
[0051] For ease of understanding, the multimedia data detection method provided in this application is specifically introduced below with reference to the accompanying drawings.
[0052] Figure 1 A schematic diagram of a multimedia data detection system provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the framework diagram includes a server 101 and multiple terminal devices 102.
[0053] It should be noted that the server 101 in the present application may be connected to multiple terminal devices 102. The server 101 may manage the multiple terminal devices 102 according to the identifier corresponding to each terminal device.
[0054] The server 101 is used to receive the multimedia data to be detected sent by the terminal device 102 and detect the authenticity of the multimedia data. The multimedia data may include audio data, video data, and text data, etc., which are not limited in the embodiments of the present application.
[0055] The terminal device 102 is used to obtain the multimedia data to be detected and send the multimedia data to be detected to the server 101. In the embodiment of the present application, the terminal device 102 may include a computer, a mobile phone, a tablet and other devices, which are not limited in the embodiment of the present application.
[0056] Figure 2 A flowchart of a multimedia data detection method provided in an embodiment of the present application, the method includes the following S201-S203.
[0057] S201, receiving a target instruction.
[0058] Among them, the target instruction is used to verify the authenticity of the multimedia data.
[0059] In one possible implementation, in response to a user selecting multimedia data and indicating a detection operation on a terminal device, the terminal device acquires the multimedia data to be detected, generates a target instruction containing the multimedia data, and then sends the target instruction to the server. Accordingly, the server receives the target instruction from the terminal device.
[0060] For example, the target instruction may include multimedia data and instruction information, and the instruction information may be: please check the authenticity of the multimedia data.
[0061] S202: Determine text information corresponding to the multimedia data, and determine background information matching the multimedia data based on the text information.
[0062] The background information is used to reflect the source of the multimedia data. For example, if the multimedia data is a video of a certain athlete winning a championship, the background information can be video information of a certain athlete participating in the entire game. The background information can also be called knowledge information or prior knowledge, etc., which is not limited in the embodiments of the present application.
[0063] In one possible implementation, after receiving the target instruction, the server may determine the multimedia data to be detected based on the target instruction (for the convenience of description, the following description takes multimedia data as an example). After determining the multimedia data, the server may convert the multimedia data into corresponding text information and extract key elements such as keywords and themes from the text information. Furthermore, based on the key elements of the text information, the server searches for background information that matches the key elements of the multimedia data from a preset multimedia database.
[0064] In the embodiment of the present application, the multimedia database may include, but is not limited to, multimedia data related to people, multimedia data related to products, and other data. The embodiment of the present application does not limit the content in the multimedia database.
[0065] In some embodiments, the server may input the multimedia data into a pre-trained multimodal model to determine text information corresponding to the multimedia data.
[0066] Among them, the multimodal model is the product of multimodal technology. The multimodal technology can be used to align the feature spaces of multiple modes, that is, the multimodal technology can be used to perform fusion processing on data of multiple modes, and the data of multiple modes can include image data, video data, audio data, etc. Exemplarily, the multimodal technology can be used to extract text data corresponding to multimedia data.
[0067] For example, for video data in multimedia data, a multimodal model can be used to extract image information and audio information in the video data and generate text information related to the video data. For example, given a video clip, a description such as "a certain athlete won a gold medal" can be generated.
[0068] In the embodiment of the present application, the multimodal model may include but is not limited to the GPT-4 model, the CLIP model and the BLIP-2 model. The embodiment of the present application does not limit the multimodal model.
[0069] In some embodiments, the server may also search for background information matching the multimedia data from a preset multimedia database based on text information corresponding to the multimedia data.
[0070] For example, assuming that the text information corresponding to the multimedia data is "a certain athlete won the gold medal", the background information matching the multimedia data may be video information or text information of a certain athlete participating in the entire game.
[0071] S203: Detect the multimedia data based on the background information of the multimedia data and the feature information of the multimedia data.
[0072] The feature information is used to represent key elements of multimedia data. For example, the feature information of a video of a badminton match may be badminton or sports events.
[0073] In a possible implementation, the server can determine fusion feature information of the multimedia data based on the background information of the multimedia data and the feature information of the multimedia data, and detect the multimedia data based on the fusion feature information of the multimedia data. The fusion feature information includes association information between the background information of the multimedia data and key elements of the feature information of the multimedia data.
[0074] In an embodiment of the present application, the fused feature information may include first fused feature information and second fused feature information. The server may normalize the first fused feature information and the second fused feature information to determine the cosine similarity of the first fused feature information and the second fused feature information. The server then detects the multimedia data based on the cosine similarity of the first fused feature information and the second fused feature information.
[0075] The first fused feature information represents common feature information of background information of multimedia data and feature information of multimedia data. The second fused feature information represents feature elements of multimedia data. The first fused feature information includes a first fused feature vector, and the second fused feature information includes a second fused feature vector. The first fused feature vector and the second fused feature vector can be determined based on the background information vector corresponding to the multimedia data and the feature information vector corresponding to the multimedia data.
[0076] In the embodiment of the present application, the method for determining the first fused feature vector includes: the server may input the background information vector corresponding to the multimedia data into the multimedia processing model to obtain the background information vector corresponding to the multimedia data. The server may also input the feature information corresponding to the multimedia data into the feature processing model to obtain the feature processing vector corresponding to the multimedia data. The server determines the first fused feature vector based on the background information vector and the feature information vector.
[0077] The server may assign a weight to the background information vector and the feature information vector according to the importance of the background information and the feature information, and add or concatenate the weighted vectors to obtain a first fused feature vector.
[0078] In an embodiment of the present application, a method for determining a second fused feature vector includes: a server inputs a feature information vector of multimedia data into a pre-trained feature alignment model, and the feature alignment model extracts a key information vector of the feature information vector as the second fused feature vector.
[0079] Furthermore, the server detects the multimedia data based on the cosine similarity of the first fused feature vector and the second fused feature vector. The cosine similarity satisfies Formula 1:
[0080]
[0081] Among them, S(x,v) represents cosine similarity, f(x) represents the first fused feature vector, g(v) represents the second fused feature vector, ||f(x)|| represents the L2 norm of the first fused feature vector, and ||g(v)|| represents the L2 norm of the second fused feature vector.
[0082] The norm is a function in a vector space that measures the size or length of a vector. Common norms include the L1 norm (the sum of the absolute values of the elements of a vector), the L2 norm (the square root of the sum of the squares of the elements of a vector, also known as the Euclidean norm), etc.
[0083] For example, assume that the second fused feature vector is g(v) = [V1, V2, ..., Vn], where Vn represents the nth element of the vector. Then the L2 norm of the second fused feature vector is
[0084] Furthermore, the server determines that the multimedia data is real data when the cosine similarity between the first feature vector and the second feature vector is within a preset range, and determines that the multimedia data is abnormal data when the cosine similarity between the first feature vector and the second feature vector is outside the preset range.
[0085] Exemplarily, the preset range may be [-1, 1], and when the cosine similarity between the first feature vector and the second feature vector is 0.8, the multimedia data is determined to be real data. When the cosine similarity between the first feature vector and the second feature vector is 1.6, the multimedia data is determined to be abnormal data.
[0086] based on Figure 2 In the technical solution shown in the embodiment of the present application, the server can receive the target instruction, determine the text information corresponding to the multimedia data, and determine the background information matching the multimedia data based on the text information. Then the server detects the multimedia data based on the background information of the multimedia data and the feature information of the multimedia data. Therefore, the authenticity of the multimedia data can be more accurately determined by combining the background information of the multimedia data and the feature information of the multimedia data.
[0087] Figure 3 FIG. 1 is a flow chart of another method for detecting multimedia data according to an exemplary embodiment. Figure 3 As shown, the server inputs the prior knowledge (i.e., the background information in the above steps, which represents the knowledge in various fields that has been collected) and the text information, video information, and audio information in the multimedia data into the prior knowledge model 301 to obtain the prior feature vector. Similarly, the server inputs the text information, video information, and audio information in the multimedia data into the feature alignment model 302 to obtain the multidimensional feature vector. Further, the server inputs the prior feature vector and the multidimensional feature vector into the fusion feature model 303 to determine whether the multimedia data is real data or abnormal data.
[0088] Among them, the fusion feature model can be based on deep learning technology, by training a large number of real and abnormal multimedia data samples, to learn the inherent laws and characteristics in the data. During the model training process, various features will be extracted and fused to build a model that can accurately distinguish between real data and abnormal data.
[0089] Exemplarily, the fusion feature model can extract features from text features, image features, audio features, etc. The following will introduce the extraction of feature information by the fusion feature model through text features, image features, and audio features in turn:
[0090] (1) Text feature extraction:
[0091] For text data, the fusion feature model can extract features such as keywords, sentence structure, grammatical rules, etc. These features help the model understand the content and context of the text, and thus determine its authenticity.
[0092] (2) Image feature extraction:
[0093] For image data, the fusion feature model can extract the pixel value, color distribution, texture features, shape features, etc. of the image. In addition, deep learning technology is also used to extract high-level features in the image, such as the edges, contours, texture patterns, etc. of objects. These features are crucial for identifying whether the image has been tampered with or forged.
[0094] (3) Audio feature extraction:
[0095] For audio data, the fusion feature model can extract the waveform features, spectrum features, rhythm features, etc. of the audio. These features help the model identify the authenticity and integrity of the audio.
[0096] After feature extraction, the server may concatenate or add the extracted text features, image features, and audio features to generate a fused feature vector.
[0097] After generating the fused feature vector, the fused feature model can use algorithms such as vector machines, neural networks, and decision trees to determine the authenticity of multimedia data based on the fused feature vector. For example, a vector machine can construct a decision boundary based on the fused feature vector to achieve classification of real data and abnormal data.
[0098] based on Figure 3 In the technical solution shown in the embodiment of the present application, the server can determine the fusion characteristics of the multimedia data based on the prior knowledge and the multi-dimensional characteristics of the multimedia data, and determine whether the multimedia data is real data or abnormal data based on the fusion characteristics. Thus, the authenticity of the multimedia data can be determined more accurately by combining the prior knowledge of the multimedia data and the characteristic information of the multimedia data.
[0099] Figure 4 FIG. 1 is a flow chart of a method for training a priori knowledge model according to an exemplary embodiment. Figure 4 As shown, the training method includes the following steps:
[0100] S401. Obtain a priori knowledge data set, and preprocess the prior knowledge in the priori knowledge data set to obtain a priori knowledge training data set.
[0101] In one possible implementation method, technicians can collect a large amount of multimedia data to form a prior knowledge data set, and send the prior knowledge data set to a server through a terminal device. After receiving the prior knowledge data set sent by the terminal device, the server can pre-process the text data, audio data, and video data in the prior knowledge data set to obtain a prior knowledge training data set.
[0102] In the embodiment of the present application, the preprocessing operation may include but is not limited to denoising, segmentation, alignment, construction of regularization terms, and the like. The embodiment of the present application does not limit the preprocessing operation.
[0103] S402: Using a preset feature extraction algorithm, extract a priori knowledge feature vector and a corresponding text feature vector from the priori knowledge training data set.
[0104] In the embodiment of the present application, after obtaining the prior knowledge training data set in step S401, the server can extract the prior knowledge feature vector including the key elements and the text feature vector corresponding to the prior knowledge feature vector from the prior knowledge training data set.
[0105] In the embodiment of the present application, the preset feature extraction algorithm may include but is not limited to a Mel-Frequency Cepstral Coefficients (MFCC) algorithm and a Filter Banks (FBANK) algorithm. The embodiment of the present application does not limit the feature extraction algorithm.
[0106] S403: Use the prior knowledge feature vector and the corresponding text feature vector to train and optimize the initial prior knowledge model to obtain a trained initial prior knowledge model.
[0107] In an embodiment of the present application, the prior knowledge feature vector and the corresponding text feature vector may be used to train the initial prior knowledge model, and the model parameters may be continuously optimized to obtain a trained initial prior knowledge model.
[0108] S404: Acquire a test data set, and use the test data set to test and evaluate the trained prior knowledge model to obtain a final prior knowledge model.
[0109] In one possible implementation, the server can input the test data set into the trained prior knowledge model, obtain the output result, and compare the output result with the standard result to evaluate the trained prior knowledge model, and adjust the parameters of the trained prior knowledge model according to the evaluation result to improve the performance and accuracy of the trained prior knowledge model to obtain the final prior knowledge model.
[0110] Through the above technical solution, the prior knowledge model can learn the potential rules and patterns between multimedia data through training, thereby improving the accuracy of the model output results.
[0111] Figure 5 FIG. 1 is a flow chart of a method for training a feature alignment model according to an exemplary embodiment. Figure 5 As shown, the training method includes the following steps:
[0112] S501, dividing multimedia data into video data, audio data and text data.
[0113] In an embodiment of the present application, the server can extract video data, audio data and text data from multimedia data through related technologies such as FFmpeg and VLC Media Player.
[0114] S502: Construct a loss function for multimedia data.
[0115] In one possible implementation, the server determines the video feature vector f(a) corresponding to the video data, determines the audio feature vector g(b) corresponding to the audio data, and determines the text feature vector h(c) corresponding to the text data. The video feature vector, the audio feature vector, and the text feature vector are normalized to determine the cosine similarity between the feature vectors.
[0116] The server constructs a loss function for multimedia data based on the cosine similarity between feature vectors. The loss function satisfies Formula 2:
[0117]
[0118] Among them, H(a,b,c) represents the loss function of multimedia data, f(a) represents the video feature vector corresponding to the video data, g(b) represents the audio feature vector corresponding to the audio data, h(c) represents the text feature vector corresponding to the text data, ||f(a)|| represents the norm of the video feature vector, ||g(b)|| represents the norm of the audio feature vector, and ||h(c)|| represents the norm of the text feature vector.
[0119] S503: Acquire a test data set, and use the test data set to test and evaluate the trained feature extraction model to obtain a final feature alignment model.
[0120] In one possible implementation method, the server can input the test data set into the trained feature alignment model to obtain an output result, and compare the output result with the standard result to evaluate the trained feature alignment model, and adjust the parameters of the trained feature alignment model according to the evaluation result to improve the performance and accuracy of the trained feature alignment model to obtain the final feature alignment model.
[0121] Through the above technical solutions, the feature alignment model can learn the underlying rules and patterns between multimedia data through training, thereby improving the accuracy of the model output results.
[0122] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to achieve the above functions, the detection device or electronic device of multimedia data includes a hardware structure and / or software module corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.
[0123] The embodiment of the present application can divide the detection device or electronic device of multimedia data into functional modules according to the above method. For example, the detection device or electronic device of multimedia data can include various functional modules corresponding to the various functional divisions, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.
[0124] Figure 6 A schematic diagram of a multimedia data detection device provided in an embodiment of the present application, the device comprising:
[0125] The receiving unit 601 is used to receive a target instruction; the target instruction is used to verify the authenticity of the multimedia data.
[0126] The determination unit 602 is used to determine the text information corresponding to the multimedia data, and determine the background information matching the multimedia data based on the text information; the background information is used to reflect the source of the multimedia data.
[0127] The detection unit 603 is used to detect the multimedia data based on the background information of the multimedia data and the characteristic information of the multimedia data; the characteristic information is used to represent the key elements of the multimedia data.
[0128] In one possible implementation, the detection unit 603 is specifically used to determine fused feature information of the multimedia data based on the background information of the multimedia data and the feature information of the multimedia data; the fused feature information includes association information between the background information of the multimedia data and key elements of the feature information of the multimedia data; and the multimedia data is detected based on the fused feature information of the multimedia data.
[0129] In a possible implementation, the detection unit 603 is further specifically used to normalize the first fused feature information and the second fused feature information to determine the cosine similarity of the first fused feature information and the second fused feature information; and detect the multimedia data based on the cosine similarity of the first fused feature information and the second fused feature information.
[0130] In a possible implementation, the determination unit 602 is also specifically used to determine that the multimedia data is real data when the cosine similarity of the first fused feature information and the second fused feature information is within a preset range; and to determine that the multimedia data is abnormal data when the cosine similarity of the first fused feature information and the second fused feature information is outside the preset range.
[0131] In a possible implementation, the determination unit 602 is further specifically used to determine a background information vector corresponding to the multimedia data and a feature information vector corresponding to the multimedia data; based on the background information vector and the feature information vector, determine a first fused feature vector and a second fused feature vector of the multimedia data.
[0132] In a possible implementation, the determination unit 602 is further configured to perform vector concatenation on the background information vector and the feature information vector to generate a first fused feature vector; and extract key information of the feature information vector as a second fused feature vector.
[0133] In a possible implementation, the determination unit 602 is further configured to determine the cosine similarity, where the cosine similarity satisfies a first preset formula, which is:
[0134]
[0135] Among them, S(x,v) represents cosine similarity, f(x) represents the first fused feature vector, g(v) represents the second fused feature vector, ||f(x)|| represents the norm of the first fused feature vector, and ||g(v)|| represents the norm of the second fused feature vector.
[0136] Figure 7 A block diagram of an electronic device of a multimedia data detection device involved in the above embodiment is shown. The multimedia data detection device includes: a processor 701 and a communication interface 702. The processor 701 is used to detect the authenticity of the multimedia data based on the background information and feature information of the multimedia data, and the communication interface 702 is used for communication between the server and the terminal device. The detection system may also include a memory 703 and a bus 704, and the memory 703 is used to store the background information and feature information of the multimedia data.
[0137] Among them, the memory 703 can be a memory in a multimedia data detection device, etc. The memory may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory may also include a combination of the above types of memory.
[0138] The processor 701 may be a processor that implements or executes various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of the present application. The processor may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of the present application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.
[0139] The bus 704 may be an Extended Industry Standard Architecture (EISA) bus, etc. The bus 704 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0140] Through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0141] An embodiment of the present application provides a computer program product including instructions. When the computer program product is run on a computer, the computer is enabled to execute the multimedia data detection method in the above method embodiment.
[0142] An embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a computer, the computer executes the multimedia data detection method in the method flow shown in the above method embodiment.
[0143] Among them, the computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a register, a hard disk, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above, or any other form of computer-readable storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read products from the storage medium and can write products to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application-specific integrated circuit (ASIC). In the embodiments of the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program, which may be used by or in conjunction with an instruction execution system, apparatus, or device.
[0144] An embodiment of the present invention provides a computer program product including instructions. When the instructions are executed on a computer, the computer executes the method for detecting multimedia data of the embodiment of the present application.
[0145] Since the multimedia data detection device, computer-readable storage medium, and computer program product in the embodiments of the present invention can be applied to the above method, the technical effects that can be obtained can also refer to the above method embodiments, and the embodiments of the present invention will not be repeated here.
[0146] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0147] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0148] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0149] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for detecting multimedia data, characterized in that: The method comprises: receiving a target instruction; the target instruction is used to verify the authenticity of the multimedia data; Determining text information corresponding to the multimedia data, and determining background information matching the multimedia data based on the text information; the background information is used to reflect the source of the multimedia data; The multimedia data is detected based on the background information of the multimedia data and the characteristic information of the multimedia data; the characteristic information is used to represent the key elements of the multimedia data.
2. The method according to claim 1, characterized in that The detecting the multimedia data based on the background information of the multimedia data and the feature information of the multimedia data includes: Determine fusion feature information of the multimedia data based on the background information of the multimedia data and the feature information of the multimedia data; the fusion feature information represents association information between the background information of the multimedia data and key elements of the feature information of the multimedia data; The multimedia data is detected based on the fusion feature information of the multimedia data.
3. The method according to claim 2, characterized in that The fusion feature information of the multimedia data includes first fusion feature information and second fusion feature information, wherein the first fusion feature information represents common features between background information of the multimedia data and feature information of the multimedia data; the second fusion feature information represents feature elements of the multimedia data; and the detecting of the multimedia data based on the fusion feature information of the multimedia data includes: Normalizing the first fused feature information and the second fused feature information to determine the cosine similarity between the first fused feature information and the second fused feature information; The multimedia data is detected based on the cosine similarity of the first fused feature information and the second fused feature information.
4. The method according to any one of claims 1 to 3, characterized in that When the cosine similarity of the first fused feature information and the second fused feature information is within a preset range, determining that the multimedia data is real data; When the cosine similarity between the first fused feature information and the second fused feature information is outside a preset range, it is determined that the multimedia data is abnormal data.
5. The method according to claim 2, characterized in that: The determining of the fusion feature information of the multimedia data includes: Determining a background information vector corresponding to the multimedia data and a feature information vector corresponding to the multimedia data; Based on the background information vector and the feature information vector, a first fused feature vector and a second fused feature vector of the multimedia data are determined.
6. The method according to claim 5, characterized in that The determining, based on the background information vector and the feature information vector, a first fused feature vector and a second fused feature vector of the multimedia data comprises: Performing vector concatenation of the background information vector and the feature information vector to generate the first fused feature vector; The key information of the feature information vector is extracted as the second fused feature vector.
7. The method according to claim 3, characterized in that The cosine similarity satisfies a first preset formula, which is: Among them, S(x,v) represents the cosine similarity, f(x) represents the first fused feature vector, g(v) represents the second fused feature vector, ||f(x)|| represents the norm of the first fused feature vector, and ||g(v)|| represents the norm of the second fused feature vector.
8. A multimedia data detection device, characterized in that: The device comprises: A receiving unit, used to receive a target instruction; the target instruction is used to verify the authenticity of the multimedia data; a determination unit, configured to determine text information corresponding to the multimedia data, and determine background information matching the multimedia data based on the text information; the background information is used to reflect the source of the multimedia data; The detection unit is used to detect the multimedia data based on the background information of the multimedia data and the characteristic information of the multimedia data; the characteristic information is used to represent the key information of the multimedia data.
9. An electronic device, characterized in that: The electronic device comprises: processor; a memory configured to store instructions executable by the processor; The processor is configured to execute the instructions to implement the multimedia data detection method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions. When the instructions in the computer-readable storage medium are executed by an electronic device, the electronic device is enabled to execute the multimedia data detection method as described in any one of claims 1 to 7.
11. A computer program product, characterized in that The computer program product includes computer instructions; when part or all of the computer instructions are run on a computer, the method for detecting multimedia data according to any one of claims 1 to 7 is executed.