Content data detection method, device, storage medium and program product

Abstract

The present application provides a content data detection method, device, storage medium and program product, wherein the method includes: in response to a data detection instruction, obtaining the content data to be tested; inputting the content data to be tested into a preset feature extraction model so that the feature extraction model outputs the target data features of the content data to be tested, wherein the preset feature extraction model is obtained based on training of a first sample data set, the first sample data set includes a first sample pair with a plagiarism relationship and a second sample pair without a plagiarism relationship; according to the target data features and preset reference data, determining whether the content data to be tested is plagiarism data, the reference data is original data. The present application realizes efficient detection of whether content data is plagiarism data, improves the accuracy of plagiarism detection, and expands the applicability of plagiarism detection.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to a content data detection method, device, storage medium and program product. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the advent of large-scale pre-trained models, the way and efficiency of content creation have been greatly improved. These large models can generate high-quality text, images, audio and video, helping creators improve productivity.

[0003] However, the popularity of this technology has also brought new challenges and problems, especially in terms of work protection and originality. Copywriting refers to the act of making slight modifications to the original work so that it appears different in form, but the substantive content has not changed fundamentally. AI big model copywriting refers to the act of fine-tuning existing works through AI big models, such as style transfer and local editing. Due to the powerful generation ability of AI big models, the cost of copywriting with AI big models is extremely low and it can produce works that are highly similar to the original works.

[0004] This kind of plagiarism not only seriously infringes on the creator's work, but also has a negative impact on the content quality and reputation of the platform. Therefore, how to accurately detect whether the content of a work has been plagiarized has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to provide a content data detection method, device, storage medium and program product, which realizes efficient detection of whether the content data is plagiarism data, improves the accuracy of plagiarism detection, and expands the applicability of plagiarism detection.

[0006] In a first aspect, an embodiment of the present application provides a content data detection method, comprising: in response to a data detection instruction, obtaining content data to be tested; inputting the content data to be tested into a preset feature extraction model so that the feature extraction model outputs target data features of the content data to be tested, wherein the preset feature extraction model is trained based on a first sample data set, and the first sample data set includes a first sample pair that has a plagiarism relationship and a second sample pair that does not have a plagiarism relationship; determining whether the content data to be tested is plagiarism data based on the target data features and preset reference data, the reference data being original data.

[0007] In one embodiment, before inputting the content data to be tested into a preset feature extraction model, it also includes: obtaining the first sample data set, wherein the first sample pair is labeled with a label indicating the existence of a plagiarism relationship, and the second sample pair is labeled with a label indicating the non-plagiarism relationship; using the first sample data set to train a preset neural network model, and removing the fully connected layer of the neural network after training to obtain the feature extraction model.

[0008] In one embodiment, before inputting the content data to be tested into a preset feature extraction model, it also includes: obtaining the first sample data set; based on the first sample data set, training a preset neural network model using a contrastive learning training strategy, and removing the fully connected layer of the neural network after training to obtain the feature extraction model.

[0009] In one embodiment, before the content data to be tested is input into a preset feature extraction model, it also includes: obtaining a labeled first sample data set and an unlabeled second sample data set, the first sample pairs are labeled with labels indicating the presence of a plagiarism relationship, and the second sample pairs are labeled with labels indicating the absence of a plagiarism relationship; based on the second sample data set, a preset neural network model is trained using a self-supervised learning training strategy to obtain a pre-trained model; the pre-trained model is fine-tuned using the first sample data set, and the fully connected layer of the pre-trained model is removed after training to obtain the feature extraction model.

[0010] In one embodiment, the method further includes: obtaining an original content data sample; inputting the original content data sample into different large models, so that the different large models respectively perform preset plagiarism operations on the original content data sample to generate at least one plagiarism sample corresponding to the original content data sample; and determining a set of the original content data sample and the at least one plagiarism sample as the sample data set.

[0011] In one embodiment, the method further includes: obtaining an original content data sample; using the original content data sample to fine-tune a preset macro model; inputting preset prompt words into the fine-tuned macro model, so that the fine-tuned macro model generates a plagiarized sample corresponding to the original content data sample according to the preset prompt words; and determining a set of the original content data sample and the plagiarized sample as the sample data set.

[0012] In one embodiment, the preset reference data includes reference features of multiple original content data configured in a preset feature library; determining whether the content data to be tested is plagiarized data based on the target data feature and the preset reference data includes: searching the preset feature library based on the target data feature, and determining whether there is a target reference feature in the preset feature library with a first similarity with the target data feature within a first preset range; if the target reference feature exists in the preset feature library, determining that the content data to be tested is plagiarized data; if the target reference feature does not exist in the preset feature library, determining that the content data to be tested is not plagiarized data.

[0013] In one embodiment, the preset reference data includes reference content data specified by a user; determining whether the content data to be tested is plagiarized data based on the target data characteristics and the preset reference data includes: obtaining the reference content data specified by the user, and extracting reference characteristics of the reference content data; determining whether the content data to be tested is plagiarized data based on the target data characteristics and the reference characteristics.

[0014] In one embodiment, determining whether the content data to be tested is plagiarized data based on the target data feature and preset reference data includes: calculating a second similarity between the target data feature and a reference feature of the reference data; judging whether the second similarity is within a second preset range; if the second similarity is within the second preset range, determining that the content data to be tested is plagiarized data; if the second similarity is not within the second preset range, determining that the content data to be tested is not plagiarized data.

[0015] In one embodiment, determining whether the content data to be tested is plagiarized data based on the target data characteristics and preset reference data includes: inputting the reference characteristics of the reference data and the target data characteristics into a preset classifier, and identifying the plagiarism relationship information between the content data to be tested and the reference data through the preset classifier.

[0016] In a second aspect, an embodiment of the present application provides a method for detecting content data of a product, comprising: in response to a user's data detection instruction, obtaining content data to be tested, wherein the content data to be tested includes product information; inputting the content data to be tested into a preset feature extraction model so that the feature extraction model outputs target data features of the content data to be tested, wherein the preset feature extraction model is trained based on a first sample data set, and the first sample data set includes a first sample pair that has a plagiarism relationship and a second sample pair that does not have a plagiarism relationship; determining whether the content data to be tested is plagiarism data based on the target data features and preset reference data, wherein the reference data is original data.

[0017] In a third aspect, an embodiment of the present application provides a content data detection device, including:

[0018] An acquisition module, used for acquiring content data to be tested in response to a data detection instruction;

[0019] An extraction module, used for inputting the content data to be tested into a preset feature extraction model so that the feature extraction model outputs target data features of the content data to be tested, wherein the preset feature extraction model is obtained by training based on a first sample data set, and the first sample data set includes a first sample pair with a plagiarism relationship and a second sample pair without a plagiarism relationship;

[0020] A determination module is used to determine whether the content data to be tested is plagiarized data based on the target data characteristics and preset reference data, and the reference data is original data.

[0021] In one embodiment, it also includes: a first training module, which is used to obtain the first sample data set before inputting the content data to be tested into the preset feature extraction model, wherein the first sample pair is marked with a label indicating that there is a plagiarism relationship, and the second sample pair is marked with a label indicating that there is no plagiarism relationship; use the first sample data set to train a preset neural network model, and remove the fully connected layer of the neural network after training to obtain the feature extraction model.

[0022] In one embodiment, it also includes: a second training module, which is used to obtain the first sample data set before inputting the content data to be tested into the preset feature extraction model; based on the first sample data set, a preset neural network model is trained using a comparative learning training strategy, and the fully connected layer of the neural network is removed after training to obtain the feature extraction model.

[0023] In one embodiment, it also includes: a third training module, which is used to obtain a labeled first sample data set and an unlabeled second sample data set before inputting the content data to be tested into a preset feature extraction model, the first sample pairs are marked with labels indicating the existence of a plagiarism relationship, and the second sample pairs are marked with labels indicating the absence of a plagiarism relationship; based on the second sample data set, a preset neural network model is trained using a self-supervised learning training strategy to obtain a pre-trained model; the pre-trained model is fine-tuned using the first sample data set, and the fully connected layer of the pre-trained model is removed after training to obtain the feature extraction model.

[0024] In one embodiment, it also includes: a first construction module, used to obtain an original content data sample; input the original content data sample into different large models, so that the different large models respectively perform preset plagiarism operations on the original content data sample to generate at least one plagiarism sample corresponding to the original content data sample; and determine the set of the original content data sample and the at least one plagiarism sample as the sample data set.

[0025] In one embodiment, it also includes: a second construction module, used to obtain an original content data sample; use the original content data sample to fine-tune a preset large model; input preset prompt words into the fine-tuned large model, so that the fine-tuned large model generates a plagiarized sample corresponding to the original content data sample according to the preset prompt words; and determine the set of the original content data sample and the plagiarized sample as the sample data set.

[0026] In one embodiment, the preset reference data includes reference features of multiple original content data configured in a preset feature library; the determination module is specifically used to search the preset feature library according to the target data feature, and determine whether there is a target reference feature in the preset feature library whose first similarity with the target data feature is within a first preset range; if the target reference feature exists in the preset feature library, it is determined that the content data to be tested is plagiarized data; if the target reference feature does not exist in the preset feature library, it is determined that the content data to be tested is not plagiarized data.

[0027] In one embodiment, the preset reference data includes reference content data specified by a user; the determination module is specifically used to obtain the reference content data specified by the user and extract reference features of the reference content data; and determine whether the content data to be tested is plagiarized data based on the target data features and the reference features.

[0028] In one embodiment, the determination module is specifically used to calculate the second similarity between the target data feature and the reference feature of the reference data; determine whether the second similarity is within a second preset range; if the second similarity is within the second preset range, determine that the content data to be tested is plagiarized data; if the second similarity is not within the second preset range, determine that the content data to be tested is not plagiarized data.

[0029] In one embodiment, the determination module is specifically used to input the reference features of the reference data and the target data features into a preset classifier, and identify the plagiarism relationship information between the content data to be tested and the reference data through the preset classifier.

[0030] In a fourth aspect, an embodiment of the present application provides an electronic device, including:

[0031] at least one processor; and

[0032] a memory communicatively coupled to the at least one processor;

[0033] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to execute the method described in any one of the above aspects.

[0034] In a fifth aspect, an embodiment of the present application provides a cloud device, including:

[0035] at least one processor; and

[0036] a memory communicatively coupled to the at least one processor;

[0037] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the cloud device to execute the method described in any one of the above aspects.

[0038] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in any one of the above aspects is implemented.

[0039] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the method described in any of the above aspects when executed by a processor.

[0040] The content data detection method, device, storage medium and program product provided by the embodiment of the present application, the system first obtains the content data to be tested and inputs it into a preset feature extraction model, and the feature extraction model is trained based on a specific first sample data set. The data set contains two types of sample pairs: the first sample pair consists of an original sample and a sample after the copywriting operation; the second sample pair consists of two samples without copywriting relationship. Through this design, the model can learn how to distinguish the features of copywriting samples and non-copywriting samples. Therefore, through the feature extraction model, the system can output the target data features of the content data to be tested, so that the target data features can highlight the characteristics of whether the content data to be tested is copywriting. The system compares the extracted target data features with the preset reference data. Through this comparison, the system can determine whether the content data to be tested is copywriting data. In this way, through the training of the feature extraction model and the analysis of the target data features, efficient detection of whether the content data is copywriting data is achieved, the accuracy of copywriting detection is improved, and the applicability of copywriting detection is expanded. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application. Obviously, the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor.

[0042] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of an application scenario of a content data detection system provided in an embodiment of the present application;

[0044] Figure 3 A flowchart of a content data detection method provided in an embodiment of the present application;

[0045] Figure 4 A schematic diagram of a data set construction process provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of a plagiarism detection scenario based on a preset database search provided in an embodiment of the present application;

[0047] Figure 6 A schematic diagram of a detection process based on a feature comparison method provided in an embodiment of the present application;

[0048] Figure 7 A schematic diagram of a specific application process of content data detection provided in an embodiment of the present application;

[0049] Figure 8 A flowchart of a method for detecting content data of a product provided in an embodiment of the present application;

[0050] Fig. 9 A schematic diagram of the structure of a content data detection device provided in an embodiment of the present application;

[0051] Fig.10 A schematic diagram of the structure of a cloud device provided in an embodiment of the present application.

[0052] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0053] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0054] The term "and / or" in this article is used to describe the association relationship of associated objects, specifically indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0055] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0056] In order to clearly describe the technical solution of the embodiment of the present application, the terms involved in the present application are first defined:

[0057] AI: Artificial Intelligence.

[0058] AIGC: Artificial Intelligence Generated Content, including but not limited to text, images, audio and video.

[0059] AI Editing: Use artificial intelligence technology to edit content, such as image style transfer, filter application, object recognition and removal, background replacement, etc.

[0060] SD: Stable Diffusion, a deep learning model for generating images.

[0061] Diffusion Model: Diffusion Model is a model for generating data, such as the SD model. The basic idea is to start with random noise and gradually "denoise" it to generate target data. A key feature of diffusion models is that they can generate high-quality and diverse samples, which makes them widely used in image generation, text generation and other fields.

[0062] VAE: Variational Autoencoder, variational autoencoder.

[0063] GAN: Generative Adversarial Network, Generative Adversarial Network.

[0064] DCGAN: Deep Convolutional Generative Adversarial Network, deep convolutional generative adversarial network.

[0065] StyleGAN: Style Generative Adversarial Network, style generative adversarial network.

[0066] BigGAN: Big Generative Adversarial Network, large-scale generative adversarial network.

[0067] Generative Models: Generative Models refers to large-scale deep learning models that can generate high-quality and diverse content. These models are mainly used to generate multimedia content such as images, videos, and audio. Typical generative models include generative adversarial networks, variational autoencoders, and diffusion models. These models generate target multimedia content based on user input by learning the distribution of large-scale data sets. Taking image generation models as an example, models like SD can generate high-quality images by inputting descriptive text. In terms of video generation, video generation models can generate continuous video frames based on given inputs, thereby generating new video content.

[0068] Multimedia Rewriting: Multimedia Rewriting refers to the use of multimedia generative models to re-create or reshape existing multimedia content into new works. This process can involve style transfer, content transformation, enhancement, or other forms of regeneration of original images, videos, audio, or text. Multimedia rewriting usually generates new and diverse versions by inputting original content into generative models, or by remixing and re-creating old multimedia works by placing them in newly generated data sets.

[0069] MSE: Mean Squared Error, is an indicator commonly used in statistics and signal processing to measure the difference between the predicted value and the true value.

[0070] Prompt: The prompt word of the large model is an input used to guide the model to generate a specific type of output. It tells the model what type of content should be generated, or in what style. In image generation, the prompt may be a text description of the required image. In AI editing, the prompt may be a description of the image editing process.

[0071] BCE: Binary Cross-Entropy, binary cross entropy.

[0072] DeepDream: A computer vision program designed to visualize and understand deep neural networks, especially feature representations inside convolutional neural networks. DeepDream works by using backpropagation techniques to enhance features already recognized by the neural network in the input image, thereby generating images with a dreamlike effect.

[0073] The content data detection method of the embodiment of the present application can be applied to any field involving plagiarism detection.

[0074] With the rapid development of artificial intelligence technology, especially the advent of large-scale pre-trained models, the way and efficiency of content creation have been greatly improved. These large models can generate high-quality text, images, audio and video, helping creators improve productivity.

[0075] However, the popularity of this technology has also brought new challenges and problems, especially in terms of work protection and originality. Copywriting refers to the act of making slight modifications to the original work so that it appears different in form, but the substantive content has not changed fundamentally. AI big model copywriting refers to the act of fine-tuning existing works through AI big models, such as style transfer and local editing. Due to the powerful generation ability of AI big models, the cost of copywriting with AI big models is extremely low and it can produce works that are highly similar to the original works.

[0076] This kind of plagiarism not only seriously infringes on the creator's work, but also has a negative impact on the content quality and reputation of the platform. Therefore, how to accurately detect whether the content of a work has been plagiarized has become a technical problem that needs to be solved urgently.

[0077] Taking the e-commerce platform scenario as an example, merchants on the e-commerce platform can use AI big models to generate different product images to update the products. If others see that the product is selling well, they are likely to misappropriate the product image and then use AI editing and other methods to plagiarize the product image, seriously infringing the information security of the original user.

[0078] In order to solve at least one of the above problems, the embodiment of the present application provides a content data detection scheme, the system first obtains the content data to be tested, and inputs it into a preset feature extraction model, the feature extraction model is trained based on a specific first sample data set. The data set contains two types of sample pairs: the first type of sample pair consists of an original sample and a sample after the copywriting operation. The second type of sample pair consists of two samples that have no copywriting relationship. Through this design, the model can learn how to distinguish the features of copywriting samples and non-copywriting samples. Therefore, through the feature extraction model, the system can output the target data features of the content data to be tested, so that the target data features can highlight whether the content data to be tested is copied. The system compares the extracted target data features with the preset reference data. Through this comparison, the system can determine whether the content data to be tested is copywriting data. In this way, through the training of the feature extraction model and the analysis of the target data features, efficient detection of whether the content data is copywriting data is achieved, the accuracy of copywriting detection is improved, and the applicability of copywriting detection is expanded.

[0079] Some embodiments of the present application are described in detail below in conjunction with the accompanying drawings. In the case where there is no conflict between the embodiments, the following embodiments and the features in the embodiments can be combined with each other. In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0080] like Figure 1 As shown, this embodiment provides an electronic device 1, including: at least one processor 11 and a memory 12, Figure 1 A processor is taken as an example. The processor 11 and the memory 12 are connected via a bus 10. The memory 12 stores instructions that can be executed by the processor 11, and the instructions are executed by the processor 11, so that the electronic device 1 can execute all or part of the process of the method in the following embodiment, so as to realize the training of the feature extraction model and the analysis of the target data features, realize the efficient detection of whether the content data is plagiarism data, improve the accuracy of plagiarism detection, and expand the applicability of plagiarism detection.

[0081] In one embodiment, the electronic device 1 may be a mobile phone, a tablet computer, a laptop computer, a desktop computer, or a large computing system composed of multiple computers.

[0082] Figure 2 Schematic diagram of a content data detection system application scenario 200 provided in an embodiment of the present application. Figure 2 As shown, the system includes: a server 210 and a terminal 220, wherein:

[0083] Server 210 may be a data platform that provides content data detection services, such as a security service platform for e-commerce. In actual scenarios, a security service platform may have multiple servers 210. Figure 2 In the figure, one server 210 is taken as an example.

[0084] The terminal 220 may be a computer, mobile phone, tablet or other device used by the user to log in to the security service platform. There may be multiple terminals 220. Figure 2 Two terminals 220 are taken as an example for illustration.

[0085] The terminal 220 and the server 210 can transmit information via the Internet, so that the terminal 220 can access the data on the server 210. The terminal 220 and / or the server 210 can be implemented by the electronic device 1.

[0086] The content data detection solution of the embodiment of the present application can be deployed on the server 210, can also be deployed on the terminal 220, or can be deployed partially on the server 210 and partially on the terminal 220. In actual scenarios, it can be selected based on actual needs, and this embodiment does not limit it.

[0087] When the content data detection solution is fully or partially deployed on the server 210 , a calling interface may be opened to the terminal 220 to provide algorithm support to the terminal 220 .

[0088] The method provided in the embodiment of the present application can be implemented by executing the corresponding software code by the electronic device 1, and can be implemented by exchanging data with the server. The electronic device 1 can be a local terminal device. When the method is run on the server, the method can be implemented and executed based on the cloud interaction system, wherein the cloud interaction system includes a server and a client device.

[0089] In a possible implementation, the method provided in the embodiment of the present application provides a graphical user interface through a terminal device, wherein the terminal device can be the local terminal device mentioned above, or can be the client device in the cloud interaction system mentioned above.

[0090] Please see Figure 3 , which is a content data detection method of an embodiment of the present application, the method can be Figure 1 The electronic device 1 shown is used to perform and can be applied to Figure 2 In the content data detection application scenario shown in , by training the feature extraction model and analyzing the target data features, efficient detection of whether the content data is plagiarism data is achieved, the accuracy of plagiarism detection is improved, and the applicability of plagiarism detection is expanded. This embodiment takes the terminal 220 as an example of the execution end, and the method includes the following steps:

[0091] Step 301: In response to a data detection instruction, obtain content data to be detected.

[0092] In this step, the detection instruction can be actively triggered by the user or automatically triggered by the system. The detection instruction can specify the content data to be tested. The content data to be tested can be multimedia data, including but not limited to images, texts, audio and video data. The content data to be tested can be uploaded by the user, such as the user uploading the image to be tested. The content data to be tested can also be automatically obtained by the system from the local or remote end according to the identifier specified by the user.

[0093] Step 302: Input the content data to be tested into a preset feature extraction model so that the feature extraction model outputs target data features of the content data to be tested, wherein the preset feature extraction model is trained based on a first sample data set, and the first sample data set includes a first sample pair with a plagiarism relationship and a second sample pair without a plagiarism relationship.

[0094] In this embodiment, the target data characteristics of the content data to be tested can be accurately extracted through a preset feature extraction model, and the feature extraction model is trained based on a specific first sample data set. The data set includes two types of sample pairs: the first sample pair includes an original sample and a plagiarized sample of the original sample after plagiarism. The second sample pair includes two samples that have no plagiarism relationship. Through this design, the feature extraction model can learn how to distinguish the characteristics of plagiarized samples and non-plagiarized samples. Therefore, through the feature extraction model, the system can output the target data characteristics of the content data to be tested, so that the target data characteristics can highlight the characteristics of whether the content data to be tested has been plagiarized, thereby improving the accuracy of the target data characteristics.

[0095] In one embodiment, obtaining a sample data set includes: obtaining an original content data sample. Inputting the original content data sample into different macro models so that the different macro models respectively perform preset plagiarism operations on the original content data sample to generate at least one plagiarism sample corresponding to the original content data sample. Determining a set of the original content data sample and the at least one plagiarism sample as the sample data set.

[0096] In this embodiment, before model training, it is necessary to construct a sample data set. First, a large number of reliable original content data samples are obtained. The original content data can be original content data. The original content data samples are washed one-to-one and one-to-many through a variety of content generation models. The washing operation mode can include style conversion, local editing, content reconstruction, fusion and disassembly of multiple samples. The rich washing methods and large models are intended to simulate a variety of washing methods in reality to obtain multiple washing samples with different styles, structures or expressions, which helps to improve the generalization ability of subsequent model training and enable it to better adapt to different types of data input. In addition, the original content data samples are combined with their corresponding multiple washing samples to form a larger and richer data set, which realizes the expansion of the sample data set, helps to provide more diverse samples during the training process, and enhances the learning effect of the model.

[0097] Optionally, the various large models used for plagiarism include but are not limited to: GAN-based models such as DCGAN, StyleGAN, BigGAN, VAE-based generative models, and generative large models such as SD series models, DALL•E (an AI image generation model) related models, and some content generation tools such as MidJourney (an AI-based image generation tool), Pika (an AI video generation and editing tool), DeepDream, Runway (an AI-based video creation tool), etc.

[0098] Optionally, the plagiarism operation can be achieved by adjusting the text prompt words of the large model, such as configuring the text prompt to include local editing of the original image, style transfer, content reconstruction, multi-sample combination and disassembly, etc.

[0099] The original content data samples can be rewritten using only the existing large model without model training, thus simplifying the data construction process.

[0100] Optionally, the correspondence between the original content data sample and the plagiarized sample can be a single original sample to a single plagiarized sample, or multiple original samples to a single plagiarized sample, or a single original sample to multiple plagiarized samples, or multiple original samples to multiple plagiarized samples. Diversified correspondences can obtain a richer sample data set.

[0101] In one embodiment, obtaining a sample data set includes: obtaining an original content data sample. Fine-tuning a preset macro model using the original content data sample. Inputting a preset prompt word into the fine-tuned macro model so that the fine-tuned macro model generates a plagiarized sample corresponding to the original content data sample according to the preset prompt word. Determining a set of the original content data sample and the plagiarized sample as a sample data set.

[0102] In this embodiment, different large models can also be trained using original content data samples, so that the trained large models can directly generate plagiarism samples according to preset prompt words, which can avoid plagiarism operations and simplify the process. Specifically, by fine-tuning the preset large model through the original content data sample, the adaptability and generation ability of the large model to a specific data set can be improved. For example, the original content data sample can be a character image containing a user's original character A. After fine-tuning the preset large model with the character image, the preset large model can learn the appearance characteristics of character A and can generate various images of character A. The preset prompt word is used to instruct the large model to generate specific content. For example, the preset prompt word is "an image of character A". The preset prompt word is input into the fine-tuned large model, and the large model can generate an image P of character A. Since character A is not original to the large model, the image P of character A generated by the large model can be determined as a plagiarism image. In this way, by inputting the preset prompt word, the fine-tuned large model can generate a plagiarism sample corresponding to the original content data sample, thereby enriching the diversity and representativeness of the sample data set. This method not only improves the quality of the data set, but also enhances the generalization ability and accuracy of the model when processing similar tasks. In addition, by combining the original data and the generated plagiarism samples, more comprehensive and effective support can be provided for subsequent model training and optimization, thereby improving the performance and efficiency of the overall system.

[0103] Optionally, when constructing a sample data set, multiple methods may be used in combination to further enhance the richness of the sample data set.

[0104] like Figure 4 As shown, it is a schematic diagram of a data set construction process provided in an embodiment of the present application. The data set construction process can be as follows:

[0105] Step 0: Get the original content data samples, which may include non-AI generated samples (such as samples collected in real scenes) and AI generated samples. The sample content includes but is not limited to audio, video, image and other data. For example, it may include AI generated pictures and natural pictures. Here, natural pictures can be photos taken in real scenes or pictures drawn by users using drawing tools.

[0106] Step 1: Use an image-based copy-scraping method, such as using a preset large model to copy the original content data sample. Specifically, collect multiple large models, input the original content data sample and a prompt for the model to complete copy-scraping operations such as local editing and style conversion, implement copy-scraping operations including local editing and style conversion on the sample, and obtain a copy-scraped sample.

[0107] Step 2: Adopt the model fine-tuning method to fine-tune the preset large model using the original content data samples, and then use the fine-tuned large model to generate the plagiarized samples corresponding to the original multimedia samples.

[0108] Step 3: Determine the plagiarism sample pair, record the original content data sample and the corresponding plagiarism sample, and obtain the sample data set. The sample data set can also be processed to adapt it to different training tasks, such as generating an image pair (1 original sample image + 1 corresponding plagiarism sample image).

[0109] In one embodiment, before inputting the content data to be tested into the preset feature extraction model, it also includes: obtaining a first sample data set, wherein the first sample pair is labeled with a label indicating that there is a plagiarism relationship, and the second sample pair is labeled with a label indicating that there is no plagiarism relationship. The first sample data set is used to train a preset neural network model, and after training, the fully connected layer of the neural network is removed to obtain a feature extraction model.

[0110] In this embodiment, a supervised training strategy can be used to train a feature extraction model, and the model is directly trained according to the sample label. After the training is completed, all or part of the fully connected layer of the model is removed, and the model features are taken as sample features. Among them, the first sample pair includes an original sample and a copy-washed sample of the original sample after the copy-washing operation. The second sample pair includes two samples without copy-washing relationships. By marking the first sample pair with a label that there is a copy-washing relationship, and marking the second sample pair with a label that there is no copy-washing relationship, these labels can provide clear supervision information for model training. This marking method can effectively distinguish copy-washing and non-copy-washing sample pairs, providing a reliable basis for subsequent model training. The preset neural network model is trained using the first sample data set after marking. Through training, the neural network model can distinguish the features and relationships between copy-washing and non-copy-washing relationships, thereby improving the model's recognition ability of copy-washing relationships. After the training is completed, the fully connected layer of the neural network is removed to obtain a model that focuses on feature extraction. Through this supervised learning process, the model can more accurately identify and extract features related to copy-washing. In summary, this embodiment can effectively improve the accuracy and efficiency of plagiarism detection through precise sample annotation and model training, combined with the optimization of the feature extraction model.

[0111] Optionally, the steps of training the feature extraction model using a supervised training strategy may be as follows:

[0112] Step 0: Select an applicable preset neural network model, such as convnext-base neural network. ConvNeXt-Base is a variant of the convolutional neural network architecture and belongs to the ConvNeXt series. The ConvNeXt series is an architecture that modernizes traditional convolutional neural networks and aims to combine the advantages of convolutional neural networks with the development of modern deep learning technology to improve the performance and efficiency of the model.

[0113] Step 1: In the original sample data set, according to whether there is a plagiarism relationship, set the first sample pair with a plagiarism relationship and the second sample pair without a plagiarism relationship, and mark the corresponding labels respectively to form a first sample data set, and randomly divide the first sample data set into a training set, a validation set, and a test set.

[0114] Step 2: Data preprocessing and data enhancement can be performed on the divided first sample data set.

[0115] Step 3: Use the enhanced first sample data set to train the preset neural network model. When the loss (loss function) of the validation set no longer decreases, save the model. After the training is completed, remove all or part of the fully connected layer of the model to obtain a feature extraction model.

[0116] Optionally, the above loss functions include but are not limited to: Cross Entropy Loss, Kullback-Leibler Divergence Loss, Binary Cross Entropy Loss (BCE Loss), Mean Squared Error (MSE), Mean Absolute Error (MAE), Huber Loss. In actual scenarios, a suitable loss function can be selected according to actual needs so that the feature extraction model can be applied to feature extraction tasks in different scenarios.

[0117] In one embodiment, before inputting the content data to be tested into the preset feature extraction model, the method further includes: obtaining a first sample data set, training a preset neural network model using a contrastive learning training strategy based on the first sample data set, and removing the fully connected layer of the neural network after training to obtain a feature extraction model.

[0118] In this embodiment, the feature extraction model can be trained using a contrastive learning training strategy, with the goal of making the features of sample pairs with a copywriting relationship as close as possible and the features of sample pairs without a copywriting relationship as far away as possible. By optimizing the feature extraction model through a contrastive learning training strategy, the accuracy and robustness of the model in detecting copywriting behavior can be improved. Specifically, a first sample data set is first obtained, which includes two types of sample pairs: one is a first sample pair consisting of an original sample and its copywriting version, and the other is a second sample pair without a copywriting relationship. Through the contrastive learning training strategy, the model can learn the subtle differences between the original sample and its copywriting version, and can also distinguish sample pairs without a copywriting relationship. This training method enables the model to more effectively capture the feature changes brought about by the copywriting operation during feature extraction. After training, by removing the fully connected layer of the neural network, the feature extraction model obtained is more focused on feature extraction, rather than relying on a specific classification task. This enables the feature extraction model to have better generalization ability when facing different types of content data, and can adapt to a variety of copywriting detection scenarios. In addition, this method enhances the model's sensitivity to subtle feature changes through contrastive learning, thereby improving the detection accuracy of plagiarism and reducing the possibility of false positives and negatives.

[0119] Optionally, the Loss function in the contrastive learning training strategy includes but is not limited to: Contrastive Loss, Triplet Loss, Cosine Embedding Loss, InfoNCE Loss, and Circle Loss. In actual scenarios, a suitable loss function can be selected according to actual needs so that the feature extraction model can be applied to feature extraction tasks in different scenarios.

[0120] In one embodiment, before inputting the content data to be tested into the preset feature extraction model, it also includes: obtaining a labeled first sample data set and an unlabeled second sample data set, wherein the first sample pair is labeled with a label indicating that there is a plagiarism relationship, and the second sample pair is labeled with a label indicating that there is no plagiarism relationship. According to the second sample data set, a preset neural network model is trained using a self-supervised learning training strategy to obtain a pre-trained model. The pre-trained model is fine-tuned using the first sample data set, and the fully connected layer of the pre-trained model is removed after training to obtain a feature extraction model.

[0121] In this embodiment, a semi-supervised strategy can also be used to train the feature extraction model. The model is first pre-trained using a self-supervised learning method, and then fine-tuned on labeled data. By combining the strategies of self-supervised learning and supervised learning, an efficient feature extraction model can be constructed to improve the accuracy and generalization ability of plagiarism detection.

[0122] Specifically, two sample data sets are first obtained: one is a labeled first sample data set, and the other is an unlabeled second sample data set. The first sample pair in the first sample data set is labeled with a label that indicates a plagiarism relationship, and the second sample pair is labeled with a label that indicates no plagiarism relationship. The preset neural network model is trained on the unlabeled second sample data set through a self-supervised learning strategy, and the preset neural network model can automatically learn the potential structure and characteristics of the data without explicit label guidance. The training at this stage helps the model capture common feature patterns in a large amount of unlabeled data and form a powerful pre-trained model. Then the pre-trained model is fine-tuned using the labeled first sample data set. This step further optimizes the model by introducing explicit plagiarism relationship labels and labels without plagiarism relationship, so that it can more accurately identify and distinguish plagiarism behaviors. Through fine-tuning, the feature extraction model not only inherits the extensive feature extraction capabilities of the self-supervised learning stage, but also enhances its adaptability to specific tasks. Finally, by removing the fully connected layer of the pre-trained model, the obtained feature extraction model is more focused on feature extraction, rather than relying on a specific output layer, which improves the flexibility and versatility of the model, so that it can effectively extract features in different application scenarios. In this way, the advantages of self-supervised learning and supervised learning are effectively combined, significantly improving the performance and applicability of the feature extraction model in plagiarism detection tasks.

[0123] In actual scenarios, the model training process of the semi-supervised strategy can be specifically as follows:

[0124] Self-supervised pre-training stage: Use an unlabeled second sample dataset to pre-train the preset neural network model, adopt self-supervised learning methods such as contrastive learning or transfer learning, and learn useful feature representations by designing pre-training tasks (such as predicting image rotation angles, filling missing parts, etc.) to obtain a pre-trained model.

[0125] Supervised fine-tuning stage: Based on the pre-trained model, fine-tuning is performed using the labeled first sample data set. The model is optimized through supervised training to adapt it to specific classification tasks, such as determining whether the sample contains plagiarism.

[0126] Step 303: Determine whether the content data to be tested is plagiarized data based on the target data characteristics and preset reference data.

[0127] In this step, reference data refers to standard original data used to assist detection, and reference data includes but is not limited to images, texts, audio and video data, etc. For example, reference data can be original data specified by the user, or data pre-stored in the database that has been authenticated as original.

[0128] Through the feature extraction model, the system can output the target data features of the content data to be tested, and the target data features can highlight the characteristics of whether the content data to be tested is plagiarized. The system compares the extracted target data features with the preset reference data to determine whether the content data to be tested is plagiarized data. In this way, through the training of the feature extraction model and the analysis of the target data features, it realizes the efficient detection of whether the content data is plagiarized data, improves the accuracy of plagiarism detection, and expands the applicability of plagiarism detection.

[0129] In one embodiment, the preset reference data includes reference features of a plurality of original content data configured in a preset feature library. Step 303 may specifically include: searching in the preset feature library according to the target data feature, and determining whether there is a target reference feature in the preset feature library whose first similarity with the target data feature is within a first preset range. If the target reference feature exists in the preset feature library, it is determined that the content data to be tested is plagiarized data. If the target reference feature does not exist in the preset feature library, it is determined that the content data to be tested is not plagiarized data.

[0130] In this embodiment, by using the reference features in the preset feature library, it is possible to quickly and accurately determine whether the content data to be tested has been plagiarized. First, multiple original content data are collected, such as original product images that have been reviewed and certified by the e-commerce platform. Then, the feature extraction model trained in any of the aforementioned embodiments can be used to extract the features of the multiple original content data, and store them as reference features of the multiple original content data in a preset feature library. These reference features represent the feature patterns of the original content.

[0131] When detecting the content data to be tested, its target data features are first extracted. Then, by searching for reference features similar to the target data features in the preset feature library, it is determined whether there is a target reference feature with a first similarity within a preset range. The calculation of the first similarity is used to measure the similarity between the content data to be tested and the original reference data in the database. If a target reference feature that meets the similarity conditions is found in the feature library, it can be determined that there is a high similarity between the content data to be tested and the reference data corresponding to the target reference feature, and then it can be inferred that the content data to be tested may have been plagiarized, and can be marked as plagiarized data. Conversely, if no target reference feature that meets the conditions is found, the content data to be tested is considered not to have been plagiarized. The first preset range can be set according to actual needs, where when the first similarity is 1, it means that the content data to be tested is the same as the reference data corresponding to the target reference feature, so the first preset range may not include 1 to avoid misjudgment of plagiarism.

[0132] In this way, through the rapid calculation of feature similarity, efficient detection of content data plagiarism is achieved. By using the preset feature library, potential plagiarism data can be quickly located in large-scale data, improving the speed and accuracy of detection. At the same time, by setting a preset range of similarity, the solution can flexibly adjust the sensitivity of detection to adapt to different application requirements and scenarios. Overall, the solution effectively realizes the automation and intelligent detection of content data plagiarism.

[0133] like Figure 5 As shown, a schematic diagram of a plagiarism detection scenario based on a preset database search provided by an embodiment of the present application is provided. Taking the content data to be tested as an image to be tested as an example, the specific process may include the following:

[0134] Base database construction phase: first collect base database samples (i.e. reference data), then extract features from the base database samples, and store the reference features of the base database samples in the feature database.

[0135] Feature retrieval stage: extract features from the sample to be tested (i.e., the image to be tested), calculate the first similarity between the features of the sample to be tested and the reference features in the feature library, and the measurement function of the first similarity can use cosine similarity, vector inner product, etc. to determine whether the sample to be tested has a plagiarism relationship with the reference data in the feature library. If a target reference feature that meets the similarity conditions is retrieved, for example, the similarity between the features of the sample to be tested and a reference feature in the feature library is greater than a preset threshold (the threshold may be the lower limit of the first preset range), then the sample to be tested can be determined to be a plagiarism sample. Otherwise, it is determined that there is no plagiarism relationship between the sample to be tested and the reference data in the feature library, i.e., there is no risk of plagiarism of the bottom library image.

[0136] In one embodiment, the preset reference data includes reference content data specified by the user. Step 303 may specifically include: obtaining the reference content data specified by the user, and extracting reference features of the reference content data. According to the target data features and the reference features, determining whether the content data to be tested is plagiarized data.

[0137] In this embodiment, the reference data can be content data specified by the user. For example, if the user suspects that the image to be tested is plagiarized from his or her original image, he or she can designate his or her original image as the reference image to trigger plagiarism detection for the image to be tested, thereby improving the user interaction experience.

[0138] During the detection process, the features of the reference content data specified by the user are first extracted, which are called reference features. These reference features represent the features of the content that the user considers to be original. Next, the target data features of the content data to be tested are extracted. By directly comparing the target data features with the reference features specified by the user, it is determined whether there is a plagiarism relationship between the content data to be tested and the reference content data, which can quickly identify potential plagiarism behaviors and improve the accuracy and efficiency of detection. This method allows users to specify reference data according to specific needs, thereby realizing personalized plagiarism detection. User participation in the reference data specification process also enhances the controllability and adaptability of the system, which can better meet the needs of different users and application scenarios.

[0139] In one embodiment, step 303 may specifically include: calculating a second similarity between the target data feature and the reference feature of the reference data. Determining whether the second similarity is within a second preset range. If the second similarity is within the second preset range, determining that the content data to be tested is plagiarized data. If the second similarity is not within the second preset range, determining that the content data to be tested is not plagiarized data.

[0140] In this embodiment, it is possible to determine whether the content data to be tested has a high similarity with the reference data based on the measurement method of feature similarity. Taking the user-specified reference content data as an example, the feature extraction model of any of the above embodiments can be used to extract the data features of the reference content data as reference features, and by calculating the second similarity between the target data feature and the reference feature, and comparing it with the second preset range, when the second similarity falls within the preset range, the system marks the content data as copywriting data, thereby prompting the user that the content of the content data to be tested may be adapted from the work of others. On the contrary, when the second similarity is not within the preset range, the system considers that the content data to be tested has a high originality. The second threshold range can be the same as the first threshold range, or it can be different from the first threshold range, and can be set based on actual needs. In this way, efficient and accurate terminal judgment of copywriting behavior of content data copywriting is realized, ensuring the accuracy and reliability of copywriting detection, not only improving the detection efficiency, but also enhancing the sensitivity to different degrees of similarity by setting a reasonable similarity range, reducing the possibility of misjudgment.

[0141] Optionally, assuming that the target data features and the reference features of the reference data are expressed in the form of feature vectors, there may be multiple ways to measure the first similarity and / or the second similarity, including but not limited to the following measurement methods:

[0142] (1) Euclidean Distance

[0143] Euclidean distance can be used to calculate the difference between the feature vectors of target data features and reference features, and is particularly suitable for multi-dimensional feature vectors.

[0144] formula:

[0145]

[0146] Here, it is assumed that the feature vector is n-dimensional, n is a positive integer. d represents the Euclidean distance between the feature vectors of the target data feature and the reference feature, X i The i-th component of the feature vector representing the target data feature, Y i The i-th component of the feature vector representing the reference feature.

[0147] (2) Manhattan Distance

[0148] Manhattan distance, also known as L1 distance, calculates the sum of the absolute differences between the feature vector components of the target data feature and the reference feature.

[0149] formula:

[0150]

[0151] Here, it is assumed that the feature vector is n-dimensional, n is a positive integer. d represents the Manhattan distance between the feature vectors of the target data feature and the reference feature, X i The i-th component of the feature vector representing the target data feature, Y i The i-th component of the feature vector representing the reference feature.

[0152] (3) Cosine Similarity

[0153] Cosine similarity can be used to measure the directional similarity of two feature vectors of the target data feature and the reference feature, rather than the amplitude.

[0154] formula:

[0155]

[0156] in, represents the cosine similarity between the feature vectors of the target data feature and the reference feature, X represents the feature vector of the target data feature, and Y represents the feature vector of the reference feature. The norm length of the feature vector representing the target data features, The norm length of the feature vector representing the reference feature.

[0157] (4) Principal Component Analysis (PCA)

[0158] PCA can reduce the high-dimensional features of target data features and reference features to facilitate calculation of differences.

[0159] (5) Mean Absolute Error

[0160] For simple feature comparison, the mean absolute error between the two feature vectors of the target data feature and the reference feature can be used to measure.

[0161] formula:

[0162]

[0163] It is assumed that the eigenvector is of n-dimensional and n is a positive integer. represents the mean absolute error between the feature vectors of the target data features and the reference features, X i The i-th component of the feature vector representing the target data feature, Y i The i-th component of the feature vector representing the reference feature.

[0164] In one embodiment, step 303 may specifically include: inputting reference features of the reference data and target data features into a preset classifier, and identifying plagiarism relationship information between the content data to be tested and the reference data through the preset classifier.

[0165] In this embodiment, a preset classifier can be used to identify the plagiarism relationship information between the content data to be tested and the reference data, and the classifier can be trained by directly adding a fully connected layer after the network structure of the feature extraction model, or the classifier can be preset through a separate neural network model. Optionally, other types of classifiers can also be used, such as random forests, decision trees, support vector machines, etc. In actual scenarios, a suitable classifier can be selected according to actual needs.

[0166] Taking the reference content data specified by the user as an example, the feature extraction model of any of the aforementioned embodiments can be used to extract the data features of the reference content data as reference features. By inputting the target data features and the reference features into the preset classifier, the information on the relationship between the content data to be tested and the reference content data can be automatically and efficiently identified. Utilizing the powerful learning and recognition capabilities of the preset classifier, the system can quickly process a large amount of data and automatically determine whether the data to be tested is plagiarized data, significantly improving the accuracy and efficiency of detection. In addition, the classifier can adapt to the ever-changing plagiarism techniques and patterns through continuous updating and training, thereby maintaining the advanced nature and reliability of the detection technology. This automated detection mechanism not only improves the processing speed of the system, but also enhances the protection of original content, effectively safeguarding the legitimate rights and interests of content creators.

[0167] like Figure 6 As shown, it is a schematic diagram of a detection process based on a feature comparison method provided in an embodiment of the present application, wherein the sample to be tested refers to the content data to be tested, and the reference sample refers to the reference content data. The specific process can be as follows:

[0168] Step 0: Input the sample to be tested into the feature extraction model trained by any of the above embodiments to perform feature extraction to obtain target data features.

[0169] Step 1: Input the reference sample into the feature extraction model to extract features and obtain reference data features.

[0170] Step 2: The classifier model can be used to determine whether there is a plagiarism relationship between the sample to be tested and the reference sample. The second similarity between the target data feature and the reference feature can also be calculated, and then the process proceeds to Step 3. The measurement function of the second similarity can use cosine similarity, vector inner product, etc. For details, please refer to the measurement method of feature similarity in the above-mentioned embodiment.

[0171] Step 3: Determine whether the second similarity is greater than a preset threshold (the threshold may be the lower limit of a second preset range). If the second similarity is greater than the preset threshold, it is determined that there is a plagiarism relationship between the sample to be tested and the reference sample; otherwise, it is determined that there is no plagiarism relationship between the two.

[0172] like Figure 7 As shown, it is a schematic diagram of a specific application process of content data detection provided by an embodiment of the present application. Compared with the previous embodiment, this embodiment combines database retrieval and feature comparison to realize plagiarism detection, which may specifically include:

[0173] Step 0: Build a dataset for multimedia plagiarism detection.

[0174] Step 1: Based on the dataset, train the multimedia feature extraction model.

[0175] Step 2: Based on the preset database retrieval method, determine whether the sample to be tested has a plagiarism relationship with the reference content data in the preset database. If so, determine that the sample to be tested is a plagiarism sample, otherwise proceed to step 3.

[0176] Step 3: If plagiarism is not found in Step 2, it can be determined based on feature comparison whether the feature similarity (i.e., the second similarity) between the sample to be tested and the reference sample is greater than a preset threshold. If so, the sample to be tested is determined to be a plagiarism sample; otherwise, the sample to be tested is determined to be a non-plagiarism sample.

[0177] It should be noted that the above embodiments are only examples. In actual use, the content data detection process can be directly adopted. Figure 5 The embodiment shown in the figure is implemented by a preset database search method, and can also be implemented by Figure 6 The feature comparison method of the embodiment shown can also be used in combination with the two methods, such as using the feature comparison method first and then the database search method, or performing feature comparison and database search in parallel. The embodiment of the present application does not limit the combination of implementation methods.

[0178] The above method covers a comprehensive solution from data set construction to feature extraction, and then to intelligent retrieval and comparison. It can accurately determine whether the content data is plagiarized, effectively prevent plagiarism, promote the healthy development of the creative industry, and protect the information security of the original authors.

[0179] For details of each step of the above method, please refer to the relevant description of the above embodiment, which will not be repeated here.

[0180] Please see Figure 8 , which is a method for detecting content data of a commodity according to an embodiment of the present application, the method can be Figure 1 The electronic device 1 shown is used to perform and can be applied to Figure 2 In the content data detection application scenario shown in , by training the feature extraction model and analyzing the target data features, efficient detection of whether the content data to be tested is plagiarized data is achieved, the accuracy of plagiarism detection is improved, and the applicability of plagiarism detection is expanded. This embodiment takes the terminal 220 as an execution end as an example, and takes the detection scenario of content data about goods in an e-commerce scenario as an example. The method includes the following steps:

[0181] Step 801: In response to a data detection instruction from a user, obtain content data to be tested, where the content data to be tested includes product information.

[0182] In this step, product information includes but is not limited to product-related knowledge information, appearance information, product promotion information, product-related order information, product-related service information, etc. The content data to be tested includes but is not limited to product images, product introduction text, product introduction audio and video, etc., such as the AI ​​main picture of the product, the AI ​​details page of the product, etc.

[0183] Step 802: Input the content data to be tested into a preset feature extraction model so that the feature extraction model outputs target data features of the content data to be tested, wherein the preset feature extraction model is trained based on a first sample data set, and the first sample data set includes a first sample pair with a plagiarism relationship and a second sample pair without a plagiarism relationship.

[0184] Step 803: Determine whether the content data to be tested is plagiarized data based on the target data characteristics and preset reference data, and whether the reference data is original data.

[0185] For details of each step of the above method, please refer to the relevant description of the above embodiment, which will not be repeated here.

[0186] Please see Fig. 9 , which is a content data detection device 900 of an embodiment of the present application, which can be applied to Figure 1 The electronic device 1 shown can be applied to Figure 2 In the content data detection application scenario shown in , the training of the feature extraction model and the analysis of the target data features are used to realize the efficient detection of whether the content data to be tested is plagiarism data, thereby improving the accuracy of plagiarism detection and expanding the applicability of plagiarism detection. The device includes: an acquisition module 901, an extraction module 902 and a determination module 903, and the functional principles of each module are as follows:

[0187] The acquisition module 901 is used to acquire the content data to be tested in response to the data detection instruction.

[0188] The extraction module 902 is used to input the content data to be tested into a preset feature extraction model so that the feature extraction model outputs the target data features of the content data to be tested, wherein the preset feature extraction model is trained based on a first sample data set, and the first sample data set includes a first sample pair with a plagiarism relationship and a second sample pair with no plagiarism relationship.

[0189] The determination module 903 is used to determine whether the content data to be tested is plagiarized data based on the target data characteristics and preset reference data, and whether the reference data is original data.

[0190] In one embodiment, it further includes: a first training module, which is used to obtain a first sample data set before inputting the content data to be tested into a preset feature extraction model, wherein the first sample pair is labeled with a label indicating that there is a plagiarism relationship, and the second sample pair is labeled with a label indicating that there is no plagiarism relationship. The first sample data set is used to train a preset neural network model, and after training, the fully connected layer of the neural network is removed to obtain a feature extraction model.

[0191] In one embodiment, it further includes: a second training module, which is used to obtain a first sample data set before inputting the content data to be tested into the preset feature extraction model. According to the first sample data set, a preset neural network model is trained using a contrastive learning training strategy, and after training, the fully connected layer of the neural network is removed to obtain a feature extraction model.

[0192] In one embodiment, it also includes: a third training module, which is used to obtain a labeled first sample data set and an unlabeled second sample data set before inputting the content data to be tested into the preset feature extraction model, wherein the first sample pair is labeled with a label indicating that there is a plagiarism relationship, and the second sample pair is labeled with a label indicating that there is no plagiarism relationship. According to the second sample data set, a preset neural network model is trained using a self-supervised learning training strategy to obtain a pre-trained model. The pre-trained model is fine-tuned using the first sample data set, and the fully connected layer of the pre-trained model is removed after training to obtain a feature extraction model.

[0193] In one embodiment, the method further includes: a first construction module for obtaining an original content data sample. The original content data sample is input into different large models, so that the different large models respectively perform preset plagiarism operations on the original content data sample to generate at least one plagiarism sample corresponding to the original content data sample. The set of the original content data sample and the at least one plagiarism sample is determined as a sample data set.

[0194] In one embodiment, the method further includes: a second construction module for obtaining an original content data sample. The original content data sample is used to fine-tune a preset macro model. The preset prompt words are input into the fine-tuned macro model, so that the fine-tuned macro model generates a plagiarized sample corresponding to the original content data sample according to the preset prompt words. The set of the original content data sample and the plagiarized sample is determined as a sample data set.

[0195] In one embodiment, the preset reference data includes reference features of a plurality of original content data configured in a preset feature library. The determination module 903 is specifically used to search in the preset feature library according to the target data feature, and determine whether there is a target reference feature in the preset feature library whose first similarity with the target data feature is within a first preset range. If the target reference feature exists in the preset feature library, it is determined that the content data to be tested is plagiarized data. If the target reference feature does not exist in the preset feature library, it is determined that the content data to be tested is not plagiarized data.

[0196] In one embodiment, the preset reference data includes reference content data specified by the user. The determination module 903 is specifically used to obtain the reference content data specified by the user and extract reference features of the reference content data. According to the target data features and the reference features, it is determined whether the content data to be tested is plagiarized data.

[0197] In one embodiment, the determination module 903 is specifically used to calculate the second similarity between the target data feature and the reference feature of the reference data. It is determined whether the second similarity is within a second preset range. If the second similarity is within the second preset range, it is determined that the content data to be tested is plagiarized data. If the second similarity is not within the second preset range, it is determined that the content data to be tested is not plagiarized data.

[0198] In one embodiment, the determination module 903 is specifically used to input the reference features of the reference data and the target data features into a preset classifier, and identify the plagiarism relationship information between the content data to be tested and the reference data through the preset classifier.

[0199] For a detailed description of the above content data detection device 900, please refer to the description of the relevant method steps in the above embodiment. Its implementation principle and technical effect are similar, and will not be repeated here in this embodiment.

[0200] Fig.10 The following is a schematic diagram of a cloud device 100 provided in an exemplary embodiment of the present application. The cloud device 100 can be used to run the method provided in any of the above embodiments. Fig.10 As shown, the cloud device 100 may include: a memory 1004 and at least one processor 1005, Fig.10 A processor is taken as an example.

[0201] The memory 1004 is used to store computer programs and can be configured to store various other data to support operations on the cloud device 100. The memory 1004 can be an object storage service (OSS).

[0202] The memory 1004 may be implemented by any type of volatile or nonvolatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0203] The processor 1005 is coupled to the memory 1004 and is used to execute the computer program in the memory 1004 to implement the solution provided by any of the above method embodiments. The specific functions and technical effects that can be achieved are not repeated here.

[0204] Furthermore, if Fig.10 The cloud device also includes: a firewall 1001, a load balancer 1002, a communication component 1006, a power supply component 1003 and other components. Fig.10 Only some components are shown schematically, which does not mean that the cloud device only includes Fig.10 Components shown.

[0205] In one embodiment, the above Fig.10 The communication component 1006 in the communication component 1006 is configured to facilitate wired or wireless communication between the device where the communication component 1006 is located and other devices. The device where the communication component 1006 is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, LTE (Long Term Evolution, Long Term Evolution, referred to as LTE), 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component 1006 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1006 also includes a near field communication (Near Field Communication, referred to as NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (Radio Frequency Identification, referred to as RFID) technology, Infrared Data Association (Infrared Data Association, referred to as IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (Bluetooth, referred to as BT) technology and other technologies.

[0206] In one embodiment, the above Fig.10 The power supply component 1003 provides power to various components of the device where the power supply component 1003 is located. The power supply component 1003 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0207] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method of any of the aforementioned embodiments is implemented.

[0208] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method of any of the aforementioned embodiments when executed by a processor.

[0209] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0210] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0211] It should be understood that the above-mentioned processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The memory may include a high-speed RAM (Random Access Memory) memory, and may also include non-volatile storage NVM (Nonvolatile memory, NVM for short), such as at least one disk storage, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0212] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0213] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0214] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, clothing or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, clothing or device. In the absence of more restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, clothing or device including the element.

[0215] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0216] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0217] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of user data and other information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0218] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A content data detection method, characterized in that: include: Responding to the data detection instruction, obtaining the content data to be tested; Inputting the content data to be tested into a preset feature extraction model so that the feature extraction model outputs target data features of the content data to be tested, wherein the preset feature extraction model is trained based on a first sample data set, and the first sample data set includes a first sample pair with a plagiarism relationship and a second sample pair without a plagiarism relationship; According to the target data characteristics and preset reference data, it is determined whether the content data to be tested is plagiarized data.

2. The method according to claim 1, characterized in that Before inputting the content data to be tested into a preset feature extraction model, the method further includes: obtaining the first sample data set, wherein the first sample pair is labeled with a label indicating that there is a plagiarism relationship, and the second sample pair is labeled with a label indicating that there is no plagiarism relationship; The first sample data set is used to train a preset neural network model, and the fully connected layer of the neural network is removed after training to obtain the feature extraction model.

3. The method according to claim 1, characterized in that Before inputting the content data to be tested into a preset feature extraction model, the method further includes: Acquire the first sample data set; According to the first sample data set, a preset neural network model is trained using a contrastive learning training strategy, and the fully connected layer of the neural network is removed after training to obtain the feature extraction model.

4. The method according to claim 1, characterized in that: Before inputting the content data to be tested into a preset feature extraction model, the method further includes: Obtain a labeled first sample data set and an unlabeled second sample data set, wherein the first sample pairs are labeled with labels indicating that there is a plagiarism relationship, and the second sample pairs are labeled with labels indicating that there is no plagiarism relationship; According to the second sample data set, a preset neural network model is trained using a self-supervised learning training strategy to obtain a pre-trained model; The pre-trained model is fine-tuned using the first sample data set, and the fully connected layer of the pre-trained model is removed after training to obtain the feature extraction model.

5. The method according to claim 1, characterized in that Also includes: Obtaining a sample of original content data; Inputting the original content data sample into different large models, so that the different large models respectively perform preset plagiarism operations on the original content data sample to generate at least one plagiarism sample corresponding to the original content data sample; A set of the original content data sample and the at least one rewritten sample is determined as the sample data set.

6. The method according to claim 1, characterized in that Also includes: Obtaining a sample of original content data; Using the original content data sample to fine-tune the preset macro model; Inputting the preset prompt words into the fine-tuned large model, so that the fine-tuned large model generates a plagiarized sample corresponding to the original content data sample according to the preset prompt words; A set of the original content data samples and the rewritten samples is determined as the sample data set.

7. The method according to claim 1, characterized in that The preset reference data includes reference features of a plurality of original content data configured in a preset feature library; the step of determining whether the content data to be tested is plagiarized data based on the target data features and the preset reference data includes: Searching the preset feature library according to the target data feature, determining whether there is a target reference feature in the preset feature library whose first similarity with the target data feature is within a preset range; If the target reference feature exists in the preset feature library, determining that the content data to be tested is plagiarized data; If the target reference feature does not exist in the preset feature library, it is determined that the content data to be tested is not plagiarized data.

8. The method according to claim 1, characterized in that The preset reference data includes reference content data specified by a user; and determining whether the content data to be tested is plagiarized data according to the target data characteristics and the preset reference data includes: Acquire the reference content data specified by the user, and extract reference features of the reference content data; According to the target data feature and the reference feature, it is determined whether the content data to be tested is plagiarized data.

9. The method according to claim 1 or 8, characterized in that: The step of determining whether the content data to be tested is plagiarized data based on the target data characteristics and preset reference data includes: Calculating a second similarity between the target data feature and the reference feature of the reference data; Determining whether the second similarity is within a second preset range; If the second similarity is within the second preset range, determining that the content data to be tested is plagiarized data; If the second similarity is not within the second preset range, it is determined that the content data to be tested is not plagiarized data.

10. The method according to claim 1 or 8, characterized in that: The step of determining whether the content data to be tested is plagiarized data based on the target data characteristics and preset reference data includes: The reference features of the reference data and the target data features are input into a preset classifier, and the plagiarism relationship information between the content data to be tested and the reference data is identified through the preset classifier.

11. A method for detecting content data of a commodity, characterized in that: include: In response to a data detection instruction from a user, obtaining content data to be tested, wherein the content data to be tested includes commodity information; Inputting the content data to be tested into a preset feature extraction model so that the feature extraction model outputs target data features of the content data to be tested, wherein the preset feature extraction model is trained based on a first sample data set, and the first sample data set includes a first sample pair with a plagiarism relationship and a second sample pair without a plagiarism relationship; According to the target data characteristics and preset reference data, it is determined whether the content data to be tested is plagiarized data, and the reference data is original data.

12. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to perform the method described in any one of claims 1-11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 11 is implemented.

14. A computer program product, characterized in that The method comprises a computer program, which implements the method according to any one of claims 1 to 11 when being executed by a processor.

Images