Data processing method, model training method, image infringement detection model and device

Abstract

The embodiment of the invention provides a data processing method, a model training method, an image infringement detection model and equipment. According to one scheme, after a first image and a second image are obtained, bidirectional block-level contrastive analysis is executed on the first image and the second image, and the bidirectional block-level contrastive analysis comprises the steps that the first image serves as a query image, the second image serves as a reference image, and the reference image is obtained based on a pixel coordinate table of the query image; calculating a comparison loss value between the image blocks in the query image and the image blocks in the corresponding reference image; and by taking the second image as a query image and the first image as a reference image, calculating a comparison loss value between the image blocks in the query image and the image blocks in the corresponding reference image based on the pixel coordinate table of the query image. The comparison loss value is used for optimizing parameters of an image infringement detection model, and the image infringement detection model can be used for detecting whether the first image and the second image have an infringement relationship or not. The pixel coordinate table is used for recording position information of pixels in the query image.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202511020966.6 (22) Application Date 2025.07.21 (71) Applicant Hangzhou Ant Love Technology Co., Ltd. Address 310012, Building 5, Huanglong International Center, Xihu District, Hangzhou, Zhejiang Province (72) Inventor Lu Yichen (74) Patent Agency Beijing Taihe Jiusi Intellectual Property Agency Co., Ltd. 11610 Patent Attorney Chai Yanbo Liu Ge (51) Int.Cl. G06V 10 / 74 (2022.01) G06V 10 / 774 (2022.01) G06V 10 / 50 (2022.01) G06V 10 / 776 (2022.01) G06F 16 / 532 (2019.01) (54) Invention Title: Data Processing Method, Model Training Method, Image Infringement Detection Model and Device (57) Abstract: This description provides a data processing method, a model training method, an image infringement detection model, and a device. One approach is that after acquiring a first image and a second image, a bidirectional block-level comparison analysis is performed on the first and second images. This bidirectional block-level comparison analysis includes: using the first image as the query image and the second image as the reference image, calculating the comparison loss value between image blocks in the query image and corresponding image blocks in the reference image based on the pixel coordinate table of the query image; using the second image as the query image and the first image as the reference image, calculating the comparison loss value between image blocks in the query image and corresponding image blocks in the reference image based on the pixel coordinate table of the query image. The comparison loss value is used to optimize the parameters of the image infringement detection model, and the image infringement detection model can detect whether there is an infringement relationship between the first image and the second image. The pixel coordinate table is used to record the position information of pixels in the query image. Claims (3 pages), Description (35 pages), Drawings (10 pages), CN 120997535 A, 2025.11.21, CN 1 20 99 75 35 A. 1. A data processing method, characterized in that it includes: acquiring a first image and a second image; performing bidirectional block-level comparison analysis on the first image and the second image, respectively using the first image as the query image and the second image as the reference image, and using the second image as the query image and the first image as the reference image, based on a pixel coordinate table of the query image, calculating a comparison loss value between image blocks in the query image and image blocks in the corresponding reference image; wherein, the pixel coordinate table is used to record the position information of pixels in the query image; the comparison loss value is used to optimize the parameters of an image infringement detection model, and the image infringement detection model can detect whether there is an infringement relationship between the first image and the second image.2. The method according to claim 1, wherein the target query image block is an image block in the query image, the reference image has at least one candidate reference image block corresponding to the target query image block, and the target image block and the candidate reference image block have a pixel correspondence; if the target candidate reference image block is one of the at least one candidate reference image blocks, then: calculating the contrast loss value between the target query image block and the target candidate reference image block based on the pixel coordinate table of the query image includes: determining the prior distribution probability of the target candidate reference image block relative to the target query image block based on the pixel coordinate table; determining the recognition probability of correctly identifying the target candidate reference image block from the set of noise image blocks according to the target query image block and the associated set of noise image blocks; wherein the set of noise image blocks is constructed based on all image blocks in the reference image; and calculating the contrast loss value between the target query image block and the target candidate reference image block according to the prior distribution probability and the recognition probability. 3. The method according to claim 2, characterized in that, determining the prior distribution probability of the target candidate reference image block relative to the target query image block based on the pixel coordinate table includes: determining the number of pixels in the target query image block mapped to the target candidate reference image block based on the pixel coordinate table; calculating the ratio of the number of pixels to the total number of pixels in the target query image block; performing a power operation on the ratio according to a preset confidence adjustment parameter to obtain a power operation result; and normalizing the power operation result to obtain the prior distribution probability. 4. The method according to claim 2, characterized in that, calculating the contrast loss value between the target query image block and the target candidate reference image block based on the prior probability and the recognition probability includes: determining a probability difference measure between the prior distribution probability and the recognition probability; and determining the contrast loss value based on the probability difference measure. 5. The method according to claim 2, further comprising: reversing the pixel coordinate table to obtain a reversed coordinate table; based on the reversed coordinate table, determining the coordinates of each pixel in the target query image block mapped to the corresponding coordinate position in the reference image, and obtaining a mapping result; and determining the at least one candidate reference image block from the reference image according to the mapping result. 6. The method according to claim 2, further comprising: determining a query region in the query image and a reference region in the reference image; the query region and the reference region being a candidate region pair with an infringement relationship; dividing the query region according to a set image block size to obtain multiple query image blocks;According to the set image block size, the reference region is divided to obtain multiple reference image blocks; the target query image block is one of the multiple query image blocks, and the at least one candidate reference image block is determined from the multiple reference image blocks. 7. The method according to any one of claims 1 to 6, characterized in that it further includes: determining an average contrast loss value of corresponding image blocks between the first image and the second image based on a first contrast loss value and a second contrast loss value; wherein the average contrast loss value is used to optimize the parameters of the image infringement detection model; the first contrast loss value is calculated with the first image as the query image and the second image as the reference image, and the second contrast loss value is calculated with the second image as the query image and the first image as the reference image. 8. A model training method, characterized in that it includes: acquiring a training sample set, wherein the training sample set contains multiple sample image pairs; setting a loss function used for training a contrastive learning model; training the contrastive learning model based on the multiple sample image pairs and the loss function to obtain an image infringement detection model; wherein the loss function includes a composite contrastive loss function, which is constructed based on a first contrastive loss function and a second contrastive loss function; using the first contrastive loss function, based on the first pixel coordinate table of one sample image in the sample image pair, a block-level contrastive analysis can be performed on two sample images contained in the sample image pair to calculate a first contrastive loss value for corresponding image blocks between the two sample images; using the second contrastive loss function, based on the second pixel coordinate table of the other sample image in the sample image pair, a block-level contrastive analysis can be performed on two sample images contained in the sample image pair to calculate a second contrastive loss value for corresponding image blocks between the two sample images. 9. An image infringement detection model, characterized in that it comprises: multiple network layers; the multiple network layers include an input layer, an encoding layer, and a loss layer; wherein, the input layer is used to receive a first image and a second image as input; the encoding layer has two encoding modules, one encoding module being used to encode a first image block in the first image to output the image block features of the first image block, and the other encoding module being used to encode a second image block in the second image to output the image block features of the second image block; the loss layer is provided with a loss function, the loss function including a composite contrast loss function; the composite contrast loss function is constructed based on the first contrast loss function and the second contrast loss function; the first contrast loss function is used to perform block-level contrast analysis on the first image and the second image based on the first pixel coordinate table of the first image and the image block features output by the encoding layer, and calculate the first contrast loss of corresponding image blocks between the first image and the second image.The second contrast loss function is used to perform block-level contrast analysis on the second image and the first image based on the second pixel coordinate table of the second image and the image block features output by the coding layer, and calculate the second contrast loss value of the corresponding image block between the second image and the first image; The first contrast loss value and the second contrast loss value are used to optimize the parameters of the image infringement detection model, and the image infringement detection model can detect whether there is an infringement relationship between the first image and the second image. 10. An electronic device, characterized in that it includes a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1 to 8. Claims 3 / 3 pages 4 CN 120997535 A Data processing method, model training method, image infringement detection model and device Technical field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, model training method, image infringement detection model and device. Background Art

[0002] With the rapid popularization of digital media, the dissemination of image and video content on the Internet has become extremely convenient, but this has also led to an increase in image and video content infringement, editing, and tampering. In the fields of multimedia anti-piracy and image / video retrieval, accurate identification of these manipulated contents is particularly important. For this reason, Image Copy Detection (ICD) technology has emerged. In recent years, although with the progress of deep learning methods, especially self-supervised learning methods, image copy detection technology has achieved significant results and can meet the needs of practical applications to a certain extent, it performs poorly when facing image copy detection situations that have undergone complex editing and transformation, and the detection accuracy is low.

[0003] Therefore, there is a need to provide a copy detection technology solution that can better cope with complex image editing and transformation situations. Summary of the Invention

[0004] Several embodiments in this specification provide a data processing method, a model training method, an image copy detection model, a system, a device, a storage medium, and a program product to improve the robustness and performance of image copy detection, thereby better meeting the copy detection needs in increasingly complex image editing and other situations.

[0005] In a first embodiment, this specification provides a data processing method, which includes:

[0006] acquiring a first image and a second image;

[0007] performing bidirectional block-level comparison analysis on the first image and the second image, respectively using the first image as the query image and the second image as the reference image, and using the second image as the query image and the first image as the reference image, based on the image of the query image...A pixel coordinate table is used to calculate the contrast loss value between an image block in the query image and an image block in the corresponding reference image;

[0008] Wherein, the pixel coordinate table is used to record the position information of pixels in the query image; the contrast loss value is used to optimize the parameters of the image infringement detection model, and the image infringement detection model can detect whether there is an infringement relationship between the first image and the second image.

[0009] In a second embodiment, this specification also provides a data processing method, which includes:

[0010] acquiring a first image and a second image;

[0011] determining a first pixel coordinate table for the first image; the first pixel coordinate table is used to record the position information of pixels in the first image;

[0012] based on the first pixel coordinate table, performing block-level contrast analysis on the first image and the second image, and calculating a first contrast loss value between an image block in the first image and an image block in the corresponding second image;

[0013] Wherein, the first contrast loss value is used to optimize the parameters of the image infringement detection model, and the image infringement detection model can detect whether there is an infringement relationship between the first image and the second image.

[0014] In a third embodiment, this specification also provides a data processing method, which includes:

[0015] acquiring a first image and a second image; Specification 1 / 35 page 5 CN 120997535 A

[0016] inputting the first image and the second image into an image infringement detection model, and executing the image infringement detection model to output a detection result;

[0017] wherein, the detection result includes a heatmap reflecting similar regions between the first image and the second image, the heatmap being generated based on similarity information generated during a bidirectional block-level comparison analysis of the first image and the second image; the bidirectional block-level comparison analysis includes: using the first image as the query image and the second image as the reference image, and using the second image as the query image and the first image as the reference image, respectively, calculating a comparison loss value between image blocks in the query image and image blocks in the corresponding reference image based on a pixel coordinate table of the query image; the similarity information includes the similarity between image blocks in the query image and image blocks in the corresponding reference image obtained during the calculation of the comparison loss value.

[0018] In a fourth embodiment, this specification also provides a data processing method, the method comprising:

[0019] acquiring a first image and a second image;

[0020] inputting the first image and the second image into an image infringement detection model, and executing the image infringement detection model to output a detection result;

[0021] wherein, the loss function of the image infringement detection model includes a composite contrast loss function, the composite contrast loss function being constructed based on a first contrast loss function and a second contrast loss function; utilizing the first contrast loss function...Based on the first pixel coordinate table of the first image, the system can perform block-level comparison analysis on the first image and the second image to calculate the first contrast loss value between the image blocks in the first image and the corresponding image blocks in the second image; using the second contrast loss function, the system can perform block-level comparison analysis on the second image and the first image based on the second pixel coordinate table of the second image to calculate the second contrast loss value between the image blocks in the second image and the corresponding image blocks in the first image.

[0022] In a fifth embodiment, this specification provides a model training method, which includes:

[0023] obtaining a training sample set, wherein the training sample set contains multiple sample image pairs;

[0024] setting a loss function used in training a contrastive learning model;

[0025] training the contrastive learning model based on the multiple sample image pairs and the loss function to obtain an image infringement detection model;

[0026] wherein the loss function includes a composite contrastive loss function, which is constructed based on a first contrastive loss function and a second contrastive loss function; using the first contrastive loss function, based on the first pixel coordinate table of one sample image in the sample image pair, a block-level contrastive analysis can be performed on two sample images contained in the sample image pair to calculate a first contrastive loss value between the corresponding image blocks of the two sample images; using the second contrastive loss function, based on the second pixel coordinate table of the other sample image in the sample image pair, a block-level contrastive analysis can be performed on two sample images contained in the sample image pair to calculate a second contrastive loss value between the corresponding image blocks of the two sample images.

[0027] In a sixth embodiment, this specification provides an image infringement detection model, which includes: multiple network layers; the multiple network layers include an input layer, an encoding layer, and a loss layer; wherein,

[0028] the input layer is used to receive an input first image and a second image;

[0029] the encoding layer has two encoding modules, one of which is used to encode a first image block in the first image to output the image block features of the first image block, and the other encoding module is used to encode a second image block in the second image to output the image block features of the second image block;

[0030] the loss layer is provided with a loss function, the loss function including: a composite contrast loss function; the composite contrast loss function is constructed based on a first contrast loss function and a second contrast loss function; the first contrast loss function is used to perform block-level contrast analysis on the first image and the second image based on the first pixel coordinates of the first image and the image block features output by the encoding layer, and calculate the first pair of corresponding image blocks between the first image and the second image.The second contrast loss function is used to perform block-level contrast analysis on the second image and the first image based on the second pixel coordinates of the second image and the image block features output by the coding layer, and calculate the second contrast loss value of the corresponding image block between the second image and the first image;

[0031] The first contrast loss value and the second contrast loss value are used to optimize the parameters of the image infringement detection model, and the image infringement detection model can detect whether there is an infringement relationship between the first image and the second image.

[0032] In the seventh embodiment, this specification provides a data processing method, which includes:

[0033] determining the first pixel coordinate table of the first image and the second pixel coordinate table of the second image;

[0034] performing a reversal operation on the first pixel coordinate table to obtain a reverse coordinate table;

[0035] determining the pixel correspondence between the first image and the second image based on the reverse coordinate table and the second pixel coordinate table;

[0036] performing block-level contrast analysis on the first image and the second image based on the pixel correspondence, and calculating the contrast loss value between the image block in the first image and the corresponding image block in the second image.

[0037] In an eighth embodiment, this specification provides a data processing method, which includes:

[0038] obtaining a first pixel coordinate table of a first image and a second pixel coordinate table of a second image; wherein the first image and the second image are generated by editing a third image, and the first pixel coordinate table is generated by mapping the coordinates of pixels in the third image to the first image, and the second pixel coordinate table is generated by mapping the coordinates of pixels in the third image to the second image;

[0039] performing a reversal operation on the first pixel coordinate table to obtain a reversed coordinate table;

[0040] based on the second pixel coordinate table and the reversed coordinate table, mapping the coordinates of pixels in the first image to the second image to obtain a pixel correspondence between the first image and the second image;

[0041] wherein the pixel correspondence is used to participate in calculating the contrast loss value between an image block in the first image and a corresponding image block in the second image.

[0042] In a ninth embodiment, this specification provides a data processing method, the method comprising:

[0043] acquiring a first video and a second video;

[0044] performing frame extraction processing on the first video to obtain a first number of frame images extracted from the first video;

[0045] performing frame extraction processing on the second video to obtain a second number of frame images extracted from the second video;

[0046] using a trained image infringement detection model to perform: analyzing the first number of frame images and the second number of frame images.The similarity between each pair of images in the frame is used to obtain a similarity matrix; based on the similarity matrix, it is determined whether the first video and the second video have an infringement relationship.

[0047] Wherein, the loss function used by the image infringement detection model during training includes a composite contrast loss function; the composite contrast loss function is constructed based on the first contrast loss function and the second contrast loss function; using the first contrast loss function, based on the first pixel coordinate table of one sample image in the sample image pair, block-level contrast analysis can be performed on the two sample images contained in the sample image pair, and the first contrast loss value of the corresponding image block between the two sample images can be calculated; using the second contrast loss function, based on the second pixel coordinate table of the other sample image in the sample image pair, block-level contrast analysis can be performed on the two sample images contained in the sample image pair, and the second contrast loss value of the corresponding image block between the two sample images can be calculated.

[0048] In a tenth embodiment, this specification provides a data processing system, which includes:

[0049] a server, which is equipped with an image infringement detection model, for: acquiring a first image and a second image; inputting the first image and the second image into the image infringement detection model, and executing the image infringement detection model to output a detection result;

[0050] a client, for displaying the detection result;

[0051] wherein the detection result includes a heatmap reflecting similar regions between the first image and the second image, and the heatmap is generated based on the contrast loss value between image blocks in the first image and image blocks in the second image calculated by performing block-level contrast analysis on the first image and the second image.

[0052] In the eleventh embodiment, this specification provides a data processing apparatus, which includes:

[0053] an acquisition module, configured to acquire a first image and a second image;

[0054] a comparison analysis module, configured to perform bidirectional block-level comparison analysis on the first image and the second image, respectively using the first image as the query image and the second image as the reference image, and using the second image as the query image and the first image as the reference image, and based on the pixel coordinate table of the query image, calculate the comparison loss value between the image block in the query image and the corresponding image block in the reference image;

[0055] wherein, the pixel coordinate table is used to record the position information of pixels in the query image; the comparison loss value is used to optimize the parameters of the image infringement detection model, and the image infringement detection model can detect whether there is an infringement relationship between the first image and the second image.

[0056] In the twelfth embodiment, this specification also provides a data processing apparatus, which includes:

[0057] a determination module, configured to determine the first pixel coordinate table of the first image; the pixel coordinate table is used to recordThe position information of pixels in the first image;

[0058] The comparison analysis module is used to perform block-level comparison analysis on the first image and the second image based on the first pixel coordinate table, and calculate the first comparison loss value between the image block in the first image and the corresponding image block in the second image;

[0059] Wherein, the parameters of the image infringement detection model are optimized based on the first comparison loss value, and the image infringement detection model can detect whether there is an infringement relationship between the first image and the second image.

[0060] In a thirteenth embodiment, this specification also provides a data processing apparatus, which includes:

[0061] an acquisition module, configured to acquire a first image and a second image;

[0062] an execution module, configured to input the first image and the second image into an image infringement detection model, execute the image infringement detection model, and output a detection result;

[0063] a display module, configured to display the detection result;

[0064] wherein the detection result includes a heatmap reflecting similar regions between the first image and the second image, the heatmap being generated based on similarity information generated during a bidirectional block-level comparison analysis of the first image and the second image; the bidirectional block-level comparison analysis includes: using the first image as the query image and the second image as the reference image, and using the second image as the query image and the first image as the reference image, respectively, calculating a comparison loss value between image blocks in the query image and image blocks in the corresponding reference image based on a pixel coordinate table of the query image; the similarity information includes the similarity between image blocks in the query image and image blocks in the corresponding reference image obtained during the calculation of the comparison loss value. Similarity.

[0065] In a fourteenth embodiment, this specification also provides a data processing apparatus, which includes:

[0066] an acquisition module, configured to acquire a first image and a second image;

[0067] an execution module, configured to input the first image and the second image into an image infringement detection model, execute the image infringement detection model, and output a detection result;

[0068] wherein, the loss function of the image infringement detection model includes a composite contrast loss function, which is constructed based on a first contrast loss function and a second contrast loss function; using the first contrast loss function, based on the first pixel coordinate table of the first image, a block-level contrast analysis can be performed on the first image and the second image to calculate a first contrast loss value between an image block in the first image and a corresponding image block in the second image; using the second contrast loss function, based on the second pixel coordinate table of the second image, a block-level contrast analysis can be performed on the second image and the first image to calculate a second contrast loss value between an image block in the second image and a corresponding image block in the first image.

[0069] In the fifteenth embodiment, this specification also provides a data processing apparatus, which includes:

[0070] an acquisition module, configured to acquire a training sample set, the training sample set containing multiple sample image pairs;

[0071] a setting module, configured to set the loss function used during the training of the contrastive learning model;

[0072] a training module, configured to train the contrastive learning model based on the multiple sample image pairs and the loss function, to obtain an image infringement detection model;

[0073] wherein, the loss function includes a composite contrastive loss function, the composite contrastive loss function being constructed based on a first contrastive loss function and a second contrastive loss function; using the first contrastive loss function, based on the first pixel coordinate table of one sample image in the sample image pair, a block-level contrastive analysis can be performed on the two sample images contained in the sample image pair, and a first contrastive loss value for the corresponding image block between the two sample images can be calculated; using the second contrastive loss function, based on the second pixel coordinate table of the other sample image in the sample image pair, a block-level contrastive analysis can be performed on the two sample images contained in the sample image pair, and a second contrastive loss value for the corresponding image block between the two sample images can be calculated.

[0074] In a sixteenth embodiment, this specification also provides a data processing apparatus, which includes:

[0075] a determining module, configured to determine a first pixel coordinate table of a first image and a second pixel coordinate table of a second image;

[0076] a reversing module, configured to perform a reversal operation on the first pixel coordinate table to obtain a reversed coordinate table;

[0077] the determining module is further configured to: determine the pixel correspondence between the first image and the second image based on the reversed coordinate table and the second pixel coordinate table;

[0078] an analysis and calculation module, configured to perform block-level comparison analysis on the first image and the second image based on the pixel correspondence, and calculate the comparison loss value between the image block in the first image and the corresponding image block in the second image.

[0079] In a seventeenth embodiment, this specification also provides a data processing apparatus, which includes:

[0080] an acquisition module, configured to acquire a first pixel coordinate table of a first image and a second pixel coordinate table of a second image; wherein the first image and the second image are generated by editing a third image, and the first pixel coordinate table is generated by mapping the coordinates of pixels in the third image to the first image, and the second pixel coordinate table is generated by mapping the coordinates of pixels in the third image to the second image;

[0081] a reversal module, configured to perform a reversal operation on the first pixel coordinate table to obtain a reversed coordinate table;

[0082] a mapping acquisition module, configured to, based on the second pixel coordinate table and the reversed coordinate table, map the first image... (Page 5 / 35, 9 CN)The coordinates of pixels in image 120997535 A are mapped to the second image to obtain the pixel correspondence between the first image and the second image;

[0083] Wherein, the pixel correspondence is used to participate in the calculation of the contrast loss value between the image block in the first image and the corresponding image block in the second image.

[0084] In the eighteenth embodiment, this specification also provides a data processing device, which includes:

[0085] an acquisition module, used to acquire a first video and a second video;

[0086] a frame extraction module, used to perform frame extraction processing on the first video to obtain a first number of frame images extracted from the first video; and to perform frame extraction processing on the second video to obtain a second number of frame images extracted from the second video;

[0087] an execution module, used to use a trained image infringement detection model to perform: analyzing the similarity between the first number of frame images and the second number of frame images pairwise to obtain a similarity matrix; and determining whether there is an infringement relationship between the first video and the second video based on the similarity matrix;

[0088] The image infringement detection model uses a composite contrast loss function during training. This composite contrast loss function is constructed based on a first contrast loss function and a second contrast loss function. Using the first contrast loss function, a block-level contrast analysis can be performed on two sample images contained in the sample image pair based on the first pixel coordinate table of one sample image in the sample image pair, calculating the first contrast loss value of the corresponding image block between the two sample images. Using the second contrast loss function, a block-level contrast analysis can be performed on two sample images contained in the sample image pair based on the second pixel coordinate table of the other sample image in the sample image pair, calculating the second contrast loss value of the corresponding image block between the two sample images.

[0089] In a nineteenth embodiment, this specification provides an electronic device, including a memory and a processor. The memory stores executable code, and when the processor executes the executable code, it implements the methods provided in this specification.

[0090] In a twentieth embodiment, this specification provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed in a computer, it causes the computer to execute the methods provided in this specification.

[0091] In the twenty-first embodiment, this specification provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the methods provided in this specification.

[0092] The solutions provided in the above embodiments of this specification, after acquiring a first image and a second image, perform bidirectional block-level comparison analysis on the first image and the second image. During the bidirectional block-level comparison analysis, the first image is used as the basis for comparison.Using a query image as the query image and a second image as the reference image, and vice versa, a block-level comparative analysis is performed on the query image and the reference image based on the pixel coordinate table of the query image. This allows for the calculation of the contrast loss value between image blocks in the query image and their corresponding image blocks in the reference image. The block-level comparative analysis based on the pixel coordinate table of the query image accurately captures the pixel-level geometric correspondence between different images, enabling the capture of finer spatial details and avoiding information loss. This provides a high-quality foundation for subsequent calculation of the contrast loss value between corresponding image blocks of different images. Furthermore, performing comparative analysis directly at the image block level allows for the learning of more detailed contrast relationships between local regions than the overall image. In summary, this solution provides a more fine-grained pixel-level analysis when performing comparative analysis on different images. This ensures effective handling of complex image editing scenarios and improves the matching accuracy between corresponding image blocks of different images. The aforementioned comparative analysis is implemented using a corresponding image infringement detection model. The output of the image infringement detection model includes a heatmap reflecting similar areas between the first and second images. Therefore, this case also visualizes the infringement area, providing users with more intuitive information on the location of the infringement area and improving the model's reliability. Furthermore, the inspection manual for the aforementioned image infringement detection model (page 6 / 35, CN 120997535 A) includes a composite contrast loss function, which is constructed based on a first contrast loss function and a second contrast loss function. Using the first contrast loss function, based on the first pixel coordinate table of the first image, block-level contrast analysis can be performed on the first and second images to calculate the first contrast loss between image blocks in the first image and corresponding image blocks in the second image. Using the second contrast loss function, based on the second pixel coordinate table of the second image, block-level contrast analysis can be performed on the second and first images to calculate the second contrast loss between image blocks in the second image and corresponding image blocks in the first image. By using this composite contrastive loss function, the affinity between image patches can be established on the pixel overlap ratio, and the importance differences of different image patches can also be taken into account. This avoids conflict gradients caused by incorrect matching, which can greatly improve the model's ability to learn accurate and robust correspondences. Specifically, it enables the model to learn image representations that are highly robust to complex editing, thereby effectively improving detection performance in image infringement detection scenarios.

[0093] The accompanying drawings, which are included to provide a further understanding of this application and constitute a part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an improper limitation of this application. In the drawings:

[0094] Figure 1 is an example of matching between different images provided in an exemplary embodiment of this specification;

[0095] Figure 2 is a schematic diagram of the principle of using the pixel coordinate tracking module (PixTrace) to track corresponding pixels of image infringement simulation behavior according to an exemplary embodiment of this specification;

[0096] Figure 3 is a schematic diagram of using geometric guided contrast loss (CopyNCE) to calculate the similarity of corresponding image blocks between different images according to an exemplary embodiment of this specification;

[0097] Figure 4 is a schematic diagram of the network structure of the matcher according to an exemplary embodiment of this specification;

[0098] Figure 5 is a schematic diagram of a data processing method according to an exemplary embodiment of this specification;

[0099] Figures 6a and 6b are schematic diagrams of heat maps according to an exemplary embodiment of this specification;

[0100] Figure 7 is a schematic diagram of the structure of a data processing system according to an exemplary embodiment of this specification;

[0101] Figures 8 to 13 are schematic diagrams of the data processing method according to an embodiment of this specification;

[0102] Figures 14 to 19 are schematic diagrams of the structure of a data processing device according to an embodiment of this specification;

[0103] Figure 20 is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of this specification. Detailed Implementation

[0104] As mentioned above, with the popularization of digital media, the infringement, editing, and tampering of image and video content have become increasingly common. Identifying these manipulated contents is crucial for many fields, such as image / video retrieval and multimedia anti-piracy. Therefore, image infringement detection (ICD) technology has emerged to address this challenge. Currently, the mainstream method for image infringement detection is self-supervised learning (SSL), which has made significant progress, especially in contrastive learning. However, traditional self-supervised learning methods mainly focus on view-level learning (i.e., the overall similarity of the image), often neglecting fine-grained correspondences at the region or patch level. This makes them perform poorly when dealing with complex editing transformations and difficult to effectively cope with various advanced tampering techniques. Specifically, this view-level self-supervised learning method constructs image pairs and performs contrastive learning by performing various transformations on the entire image. It suffers from problems such as a lack of fine-grained correspondence learning, reliance on medical image-level annotations, and inadequacy to complex editing. The aforementioned lack of fine-grained correspondence learning refers to the fact that it primarily focuses on the overall similarity of the image without delving into the precise correspondences between regions or image patches. This results in poor performance when identifying complex or localized edits (such as image matting and local transformations). The aforementioned reliance solely on image-level annotations means that models based on self-supervised learning need to infer the infringing area from the entire image-level annotations, which is insufficient for accurately locating infringing acts. Furthermore, the aforementioned lack of robustness to complex edits refers to its inability to effectively handle significant geometric transformations and content changes.Complex editing methods such as filling are slow to capture subtle traces of manipulation by view-level features.

[0105] To compensate for the shortcomings of view-level self-supervised learning, some other methods have been proposed, such as heuristic local visual self-supervised learning methods, region feature extraction methods based on ROI pooling, methods that use object detection models or heuristic rules to extract local regions, local feature matching (LMF) methods, etc. The following describes some of the other methods proposed above.

[0106] 1) Heuristic local visual self-supervised learning methods,

[0107] In self-supervised learning, most local modeling methods are heuristic, and are therefore often called heuristic local visual self-supervised learning methods. In such methods, there is an attempt to explore fine-grained correspondences through nearest neighbor (NN) matching strategies to improve the ability to identify tampered images. For example, the feature-based nearest neighbor (FeatNN) matching method retrieves the nearest neighbor in the feature space as a positive sample. For example, location-based Nearest Neighbor (LocNN) identifies positive samples by comparing the center coordinates of image patches to their corresponding positions in the original image. Another example is the combination of FeatNN and LoaNN, which attempts to leverage the strengths of both. While these heuristic local visual self-supervised learning methods based on nearest neighbor (NN) matching improve detection performance to some extent, they still suffer from inherent inaccuracies. This can lead to false positives, partial matches, ignoring importance differences, supervision noise, and poor performance with complex editing. For instance, FeatNN is susceptible to interference from semantically similar but non-infringing regions (different objects with similar semantics), and LocNN cannot determine whether there is actual overlap between image patches (an image patch in one image may not have a corresponding part in another image, but LocNN may still give a false match). Both of these situations can result in false matches. Furthermore, FeatNN and LocNN often fail to determine the exact number of positive sample image patches, which may lead to partial matching and completely ignore the differences in importance between different positive sample image patches. These problems are crucial for image infringement detection and directly affect detection efficiency.

[0108] Figure 1 illustrates a matching case in a heuristic local visual self-supervised learning method. In Figure 1, matching A: Although there are nearest neighbor patches on the query image (the infringing image 1), when using LocNN for matching, it will still result in partial matching due to the originalThe initial block has no corresponding object in the query image, resulting in a non-overlapping false match; Match B: Neither LocNN nor FeatNN can retrieve all positive sample image blocks, only partial matching is possible; Match C: When using FeatNN for matching, it may be misled by semantically similar objects, resulting in a false match.

[0109] Among them, the above false positive match, also known as inaccurate matching, refers to the incorrect identification of irrelevant areas as infringing parts. Partial matching refers to the failure to identify all real infringing areas (also known as infringing image blocks), resulting in insufficient recall (failure to retrieve all positive samples). Ignoring the difference in importance means that all matched image blocks (positive samples) are usually treated equally, while ignoring the differences in the degree of overlap or contribution between different image blocks. And the above supervision noise, that is, the introduction of noise supervision, means that due to inaccurate matching, a large number of conflicting gradient signals (i.e., a large number of erroneous supervision signals) will be generated during the training process, which will have a serious negative impact on the convergence and performance of the model. Poor performance for complex editing: Especially LocNN, when faced with complex editing such as rotation, stretching, and cutout that cause image block deformation or irregular infringement areas, its detection effect will be greatly reduced.

[0110] 2) Region feature extraction method based on ROI pooling

[0111] In many detection or segmentation tasks, in order to extract fixed-length features from regions of arbitrary size, ROI pooling is used. This approach sacrifices fine-grained spatial details because: ROI pooling is essentially a downsampling operation, which will lose the fine spatial information inside the original region, resulting in the extracted features being too coarse and unsuitable for image infringement detection (ICD) tasks corresponding to the space.

[0112] 3) Method of using target detection model or heuristic rules to extract local regions

[0113] Such methods, for example, first use the target detection model to crop out the suspicious infringement area, or apply heuristic rules to extract multiple local regions for integration, but there are problems such as detection error propagation and large computational overhead. Among them, error propagation in detection: For example, if the target detection model fails to accurately identify the infringing area, its error will propagate downstream, affecting the final detection performance of the image. High computational overhead: Multi-stage pipeline operations (such as detection followed by comparison) will increase the overall computational complexity and time overhead.

[0114] 4) Local Feature Matching (LMF) method

[0115] This type of method is usually used to estimate camera pose, depth map, or self-built key point correspondences from different perspectives of the same object. It has problems such as being unsuitable for complex editing, data dependence and errors, and difficulty in direct transfer. Among them, the reasons for being unsuitable for complex editing include: complex editing involved in image infringement detection (ICD) tasks, such as image matting and content synthesis.Such features cannot usually be described by a simple homography matrix, while LMF mainly deals with set transformations, but it is powerless for content-level editing. Data dependency and error: When auxiliary information such as depth maps is needed, the collection cost is high and there may be estimation errors. Difficult to transfer directly: Keypoint-oriented loss functions are difficult to apply directly to image infringement detection (ICD) tasks, because ICD focuses on disputed infringement of regions or content, rather than matching specific keypoints.

[0116] In summary, although some existing methods have certain effects in image infringement detection, they have obvious limitations in solving core problems such as inaccurate fine-grained correspondences, noisy supervision signals, and low computational efficiency in infringement scenarios. Therefore, there is an urgent need for a method that can establish reliable pixel-level correspondences and effectively integrate this geometric perception into a self-supervised learning framework to significantly improve the robustness and performance of image infringement detection (ICD) and thus cope with the increasingly complex challenges of image editing.

[0117] To this end, this specification provides a new image infringement detection technology solution. This solution aims to utilize the inherent geometric "traceability" of image content to address the problems of insufficient fine-grained correspondence learning and supervision operations in image infringement detection. Specifically, the main basic idea is to introduce a pixel coordinate tracking module (PixTrace) and geometrically guided contrast loss (CopyNCE) to provide accurate pixel-level supervision, effectively handle complex image editing, and overcome the problems of false matching and noise in traditional methods.

[0118] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0119] It should be noted that, for ease of description, only the parts related to the relevant technical solutions are shown in the accompanying drawings. Without conflict, the embodiments and features in the embodiments of this specification can be combined with each other. In addition, the terms "first," "second," and "third" used in the embodiments of this specification are only used for information differentiation and do not serve any limiting function.

[0120] Before introducing the various embodiments in this specification, some technical terms appearing in this specification will be briefly introduced.

[0121] Image infringement detection (ICD): aims to identify manipulated content between image pairs.

[0122] Self-supervised learning (SSL): a machine learning paradigm that trains a model by generating supervisory signals from the data itself.

[0123] Pixel coordinate tracking module (PixTrace): used to maintain the explicit spatial mapping of pixels after editing and transformation.Instruction manual, page 9 / 35, 13 CN 120997535 A

[0124] Geometric Guided Contrast Loss Function (CopyNCE): Used to regularize image patch affinity using the mapping verified by PixTrace.

[0125] Patch Affinity, also known as image patch affinity degree: The degree of similarity or correlation between different regions or image patches in an image.

[0126] Mean Average Precision (μAP): A commonly used evaluation metric in image infringement detection and retrieval tasks, used to measure the model's ability to recognize image similarity.

[0127] Recall at 90% Precision (RP90): A commonly used evaluation metric in image infringement detection, used to represent the proportion of real infringing samples that the model can recall when the precision reaches 90%.

[0128] Descriptor: A form of model used to extract feature representations (embeddings) of images and perform image infringement identification through feature similarity.

[0129] Matcher: A form of model that directly takes image pairs as input and performs a binary classification task (determining whether they are infringing image pairs).

[0130] Vision Transformer (ViT): An image processing model based on the Transformer architecture, mainly used for computer vision tasks.

[0131] Information Noise-Contrastive Estimation (InfoNCE): A commonly used contrastive learning loss function.

[0132] Hard Negatives: In contrastive learning, these refer to examples that are similar to positive samples but are actually negative samples, or negative samples that are difficult to distinguish.

[0133] Geometric Traceability: The ability to trace the original correspondence of pixels or regions after an image has been edited.

[0134] Query Image and Reference Image: In the field of image infringement detection, the query image usually refers to the image to be detected, that is, the image whose infringement is to be checked. The reference image refers to the benchmark image used for comparison. The benchmark image can be the original image or other images, such as an image derived from the original image through editing operations.

[0135] To facilitate understanding of this technical solution, the pixel coordinate tracking module introduced in this technical solution will be discussed next.(PixTrace) and Geometric Guided Contrast Loss (CopyNCE) are described in detail. The loss function composition for image infringement detection models (such as matchers or descriptors) is also described.

[0136] I. Pixel Coordinate Tracking Module (PixTrace)

[0137] The pixel coordinate tracking module can also be understood as a pixel tracking pipeline, used to model pixel-level coordinate mapping. In this pixel coordinate tracking module, a coordinate table T is defined. The coordinate table T is a dictionary-based data structure containing a series of coordinate pairs in the corresponding image, serving as keys and values.

[0138] In specific implementations, the coordinate table T supports the following operations:

[0139] 1) Query operation, such as T[c] indicating querying the corresponding value using coordinate c as the key;

[0140] 2) Update operation, such as F(T) indicating updating all values ​​in T using function F;

[0141] 3) Reverse operation (also called inverse transformation), such as T-1 indicating reversing the key-value pair, that is, mapping the value back to the key.

[0142] Based on the above coordinate table T, all edited pixels can be tracked, and thus the coordinates of pixels in the infringing image can be mapped to the original image through the coordinate table T.

[0143] As shown in Figure 2, taking the simulation of infringement by performing various image editing on the original image Io as an example, the steps performed in the process of constructing the corresponding infringing image and the corresponding coordinate table T include: Specification 10 / 35 pages 14 CN 120997535 A

[0144] Step 1, Initialization

[0145] Select the original image Io (e.g., an initial image that has not undergone any transformation), and initialize the coordinate table T according to the coordinates of the pixels in the original image Io. After initialization, the coordinate table T = To, where the coordinates of each pixel in the original image Io are mapped to itself, i.e., T(m, n) = (m, n). That is, understandably, a coordinate table To corresponding to the original image is created to record the coordinate position changes of pixels in the original image To. Before any actual editing is performed on the original image Io, the coordinate position of each pixel in it remains unchanged by default. This means that for each pixel in the original image Io, its coordinates (m,n) are still mapped to its own position (m,n).

[0146] Step 2, Editing and Coordinate Update

[0147] In the process of performing a series of edits on the original image Io (such as affine transformation, perspective transformation, image matting, etc.) to generate the first infringing image Ia, each editing operation will cause a change in pixel coordinates, and the pixel coordinate changes can be performed by a corresponding function f to update the coordinate table, thereby obtaining the coordinate table Tao corresponding to the first infringing image Ia.

[0148] For example, assuming that the coordinates of a certain pixel in the original image Io are (m,n), then after performing an edit on the original image Io...During the affine transformation, the corresponding coordinate transformation function f1 can be used to map the coordinates (m,n) of a certain pixel in the original image Io to the coordinate position (i,j) in the first infringing image Ia, that is: f1(T0(m,n))=(i,j).

[0149] For simplification, in this case, the coordinate transformation functions corresponding to all editing operations performed on an image (such as the original image Io) are sequentially combined into a new function, which can be called a composite function F. For example, when performing a series of editing operations on the original image Io to generate the first infringing image Ia, the coordinate transformation functions corresponding to this series of editing operations can be sequentially combined to form a composite function Fa. This composite function can then update the coordinate table To and generate the coordinate table Tao of the first infringing image Ia, Tao=Fa(To).

[0150] Wherein, in the composite function Fa, f1, f2, ..., fN are a series of functions for coordinate transformation, each function is a corresponding coordinate transformation for an image editing operation; and, the circle symbol represents the composite operation of the function, for example, in representing the composite operation between the first function f1 and the second function f1, and in this composite operation, the input of the first function f1 is the coordinate table To corresponding to the original image Io, and the output of the first function f1 is used as the input of the second function f2.

[0151] Step 3, Reverse Tracking

[0152] By performing a reversal operation on the coordinate table Tao corresponding to the first infringing image Ia, the corresponding reverse coordinate table Toa can be obtained.

[0153] Wherein,

[0154] It should be noted here that: the reverse operation shown on the far right in Figure 2 can be regarded as the discrete representation of F-1 at the pixel level, and the reverse mapping can be directly obtained through T-1 without the need for a specific F-1. Wherein, as shown, T-1(i,j) = (m,n) indicates that the coordinates (i,j) of a certain pixel in one image (such as the first infringing image Ia) correspond to the coordinate positions (m,n) in another image (such as the original image Io). That is, it can be understood that the coordinate position of a certain pixel in one image is (i,j), while the corresponding coordinate position in another image is (m,n).

[0155] Step 4, Image Bridge Tracking

[0156] With the help of the reverse operation, if there is another image that is also generated by editing the original image Io, such as the second infringing image Ib shown in Figure 2, which is also generated by editing the original image Io, and the coordinate table Tbo = Fb(To) corresponding to the second infringing image Ib, then the original image Io can be used as a bridge to realize pixel tracking between the first infringing image Ia and the second infringing image Ib.

[0157] For example, if you want to find the coordinates of each pixel in the first infringing image Ia in the second infringing image Ib, then: you can refer to the specification 11 / 35Page 15 CN 120997535 A Taking the coordinate table Tao corresponding to the first infringing image Ia as an example, firstly, the coordinates of each pixel in the first infringing image Ia are mapped back to the coordinate system of the original image To using the reverse operation. Then, the composite function Fb corresponding to the second infringing image Ib is used to map the coordinates of these pixels from the coordinate system of the original image To to the coordinate system of the second infringing image Ib. Thus, the coordinates of each pixel in the first infringing image Ia are mapped to the coordinate system of the second infringing image Ib, and the corresponding mapping coordinate table is generated in the same way. Similarly, the coordinates of each pixel in the second infringing image Ib are mapped to the coordinate system of the first infringing image Ia, and the corresponding mapping coordinate table is generated in the same way.

[0158] As can be seen from the above, the pixel coordinate tracking module (PixTrace) provided above can track the pixel correspondence between the infringing image and the original image, and even between different images derived from the same original image. Specifically, the PixTrace module can deterministically and accurately obtain the similarity level combination correspondence through an explicit pixel coordinate tracking mechanism. In addition, the PixTrace module is designed to maintain an iteratively updated coordinate table to track the coordinates of pixels after complex and sequential editing operations (including affine, perspective, image matting, color dithering, etc.), enabling it to handle various nonlinear transformations and content-level tampering of images. This allows the PixTrace module to be more robust in handling complex image editing and tampering behaviors in various real-world scenarios when used for image infringement detection. In particular, the pixel-level information provided by the PixTrace module during image infringement detection provides a high-quality foundation for subsequent fine-grained learning.

[0159] II. Geometric Guided Contrast Loss (CopyNCE)

[0160] The purpose of introducing the geometric guided contrast loss function (CopyNCE) in this specification is to use the precise pixel-level correspondence information provided by the pixel coordinate tracking module (PixTrace) for any two images to regularize the affinity of image patches at the patch level, thereby providing pixel-level guidance for image infringement detection and infringement area localization.

[0161] The above-mentioned geometric contrastive loss function is derived based on a contrastive learning loss function, such as the Information Noise Contrast Estimation (InfoNCE) function. The derivation process is described below using the example of deriving the geometrically guided contrastive loss function introduced in this specification based on the InfoNCE function.

[0162] The InfoNCE function aims to maximize the mutual information I(x; xp) between a sample x and its positive sample pair xp. This mutual information satisfies the following constraint:

[0163] I(x; xp) ≥ log|X| + EX[logp(xp|X,x)]

[0164] Where EX[logp(xp|X,x)] represents the expected log probability of correctly selecting a positive sample xp given a sample x and a noise set X; |X| represents the cardinality of the noise set X, and correspondingly, log|X| is a constant term. And, the noise set X is a set of samples, which usually contains one positive sample (correlated with x) and the remaining other samples are negative samples (unrelated to x). xp is a positive sample contained in X (i.e., a positive sample related to x).

[0165] In order to maximize I(x; xp), InfoNCE is usually formulated as given by the following formula (1):

[0166]

[0167] In the above formula (1), the gθ function is used to calculate the similarity measure between two samples, and xj represents the j-th sample in the noise set X (which may be a negative sample or a positive sample). Let the given sample x be denoted as xh, and xp or xj be denoted as xv. Then, in general, gθ(xh,xv) = exp(cos(zh,zv) / τ), where zh and zv represent the features of xh and xv respectively, cos(zh,zv) represents the cosine similarity between zh and zv, and τ is a temperature parameter (used to control the smoothness of the similarity distribution).

[0168] As above, formula (1) can be understood as: calculate the similarity score of the positive sample pair, normalize it to the similarity score between all sample pairs relative to the specification page 12 / 35 16 CN 120997535 A, and then take the negative logarithm as the loss value.

[0169] In the context of image infringement detection (ICD), assuming that the query image q is an infringing edited version of the reference image r, the corresponding operation image regions (that is, the corresponding infringing and infringed image regions) are represented as Rq and Rr respectively. In the following text, for ease of distinction, Rq is referred to as a query region in the query image q (a local region of interest in the current query image q), and Rr is referred to as a reference region in the reference image r (used for matching or comparison with the query region).

[0170] In order to guide the corresponding infringement detection model (such as a matcher or descriptor) to locate the infringement and forgery region of the query image q relative to the reference image, maximizing mutual information I(Rq; Rr) is proposed as an auxiliary learning objective. Similar to InfoNCE, a contrastive noise set is constructed, where represents the interference region (also known as the negative sample region). Thus, combined with the aforementioned formula (1), the geometrically guided contrastive loss function (CopyNCE) can be derived as the following formula (2):

[0171]

[0172] where fθ represents a visual coding model with parameter θ, used to embed regions or image patches in the corresponding image. For example, fθ can be a visual coding model based on visual architectures such as ViT or CNN (convolutional neural network).

[0173] In mainstream visual architectures (such as ViT or CNN), common methods for extracting features from arbitrary regions rely on ROI pooling. However, ROI pooling inherently sacrifices fine-grained spatial details, resulting in extracted features that are too coarse for image infringement detection (ICD) tasks. To overcome this limitation, this scheme decomposes a large image region (containing a query region Rq and a reference region Rr) into several smallest unit image blocks to apply CopyNCE at a finer granular level. Specifically, the query region Rq and the reference region Rr are decomposed into and , respectively, where is the i-th query image block in the query region Rq, and is the j-th reference image block in the reference region Rr. In specific implementations, or is either the smallest unit image block defined by the corresponding visual architecture (also called the smallest processing image block, for example, in ViT, the size of this smallest unit image block is typically 16×16 pixels); and , the symbol ∪i or ∪j represents the union operation. Therefore, the larger image regions Rq and Rr can be decomposed according to the size of the smallest unit image block defined by the corresponding visual architecture. If the query image block corresponds to the i-th reference image block in the reference region Rr and the reference image block is a positive sample image block of the query image block, then the contrast noise set at the image block level can be restated as: and correspondingly, the formula for CopyNCE can be converted into the following formula (3):

[0174]

[0175] where, in this formula (3), E represents the mathematical expectation operation. In order to facilitate the description of the decomposition of gθ, let R be denoted as Ri and let Rx be denoted as Rj, then gθ can be decomposed as: gθ(Ri,Rj)=exp(cos(zi,zj) / τ). Wherein, zi=fθ(Ri) represents the image feature (token) generated by encoding image block Ri by visual coding model fθ, Ri is the i-th query image block in query region Rq, and zj=fθ(Rj) represents the image feature generated by encoding image block Rj by visual model fθ, Rj is the i-th image block in reference region Rr or an image block Rx in contrast noise set RX.

[0176] Here, feature extraction and loss calculation are performed directly at the minimum unit image block (e.g., 16×16 pixels) level of visual encoder (e.g., ViT). By operating the minimum unit image block in this way, information loss that may be caused by ROI pooling can be avoided, so that fine spatial details in the image can be captured and utilized well. This is crucial for image infringement detection (ICD) tasks that require high-precision localization of infringement areas.

[0177] It is worth noting here that for each query image block, multiple candidate corresponding positive sample image blocks (as reference image blocks) may be found in the reference image r. In practice, it can be based on querying image blocks.The coordinates of each pixel in the query image are mapped to their corresponding coordinate positions in the reference image r. This determines which reference image blocks in the reference image r correspond to the query image block, thus finding multiple candidate positive sample image blocks. For example, if the coordinates of one or more pixels in the query image block fall within the j-th reference image block in the reference image r after mapping, then the j-th reference image block can be considered a positive sample image block. The correlation between the candidate positive sample image blocks and the query image block can be quantified by their spatial overlap; that is, the more overlapping the two image blocks, the higher their correlation (i.e., the more similar they are). In InfoNCE, the goal is to make the probability of identifying a unique positive sample from a noise set as close to 1 as possible. Based on this, when multiple positive sample image blocks exist, the probability of identifying different positive sample image blocks can be regularized using a prior distribution. Suppose is one of the multiple positive sample image blocks, then the probability of identifying this positive sample image block can be regularized using a prior distribution. The prior distribution can be defined by the area ratio of the shared region between and . In practice, pixel count can be used to approximate this area ratio. According to the coordinate table T provided by PixTrace (used to represent the correspondence between pixels before and after editing), the model can be first modeled as follows:

[0178]

[0179] In the above formula (4), T[c] represents the coordinate position of a pixel c in the query image block mapped to the corresponding coordinate position in the reference image r. represents the set of pixels in the query image block that are mapped to the j-th reference image block in the reference image r. |·| represents the number of elements in the corresponding set. For example, represents the total number of pixels in the query image block, and represents the number of pixels in the query image block that satisfy this condition, that is, the number of pixels in the query image block mapped to the reference image block. (Page 14 / 35 of the specification, 18 CN 120997535 A) The preliminary prior estimate probability between the query image patch and the reference image patch (i.e., what proportion of pixels are mapped from)

[0180] Further, in order to enhance the flexibility of the prior distribution, a confidence adjustment parameter γ (also known as the confidence sharpening parameter) is introduced to adjust the certainty of the prior, and normalization is performed to obtain the prior distribution shown in the following formula (5)

[0181]

[0182] Wherein, the denominator represents the sum of the γ powers of the preliminary probabilities between the query image patch and each positive sample image patch (the corresponding reference image patch in the reference image r), and its function is to normalize the corresponding probability and determine that the sum of the prior probabilities of all candidate positive sample image patches is 1.

[0183] In specific implementation, the value of γ can be, but is not limited to, 1, 2, 3, etc. Preferably, γ can be 3.

[0184] For example, assume the query image patchThe three reference image blocks in the reference image r correspond to the three positive sample image blocks found in the reference image r for the query image block. Then: the initial prior estimate is as follows:

[0185]

[0186] Let γ = 2, then after power transformation:

[0187]

[0188] Further, after normalization:

[0189]

[0190] It can be seen from the above that after sharpening with γ = 2, the probability of the strongest matching item is higher, while the other items are further weakened. That is: here, by sharpening the original initial prior, the model's ability to identify the real matching region can be enhanced.

[0191] Here, the prior distribution realizes the regularization of image block affinity (i.e., similarity) based on the accurate image block pixel overlap ratio, rather than a simple binary positive and negative sample judgment, by utilizing the pixel-level traceability provided by the aforementioned PixTrace module.

[0192] By introducing the confidence adjustment parameter γ and further transforming the CopyNCE loss based on this prior distribution, the loss form based on the corresponding divergence (such as KL divergence) is involved. This allows the CopyNCE loss to handle multiple positive samples and assign different importance weights to different positive samples according to the actual degree of regional overlap, thereby effectively overcoming the noise gradient and performance degradation caused by false matching, partial matching or ignoring importance differences in traditional contrastive learning.

[0193] Based on the above, the formula of CopyNCE can be further converted into the following formula (6):

[0194]

[0195] Wherein, KL[·‖·] in the aforementioned formula (5) represents KL divergence. KL divergence, also known as relative entropy, is used to measure the degree of difference between two probability distributions. According to the tuning experience of InfoNCE, the negative samples in InfoNCE should be as difficult negative samples as possible to be distinguished from the positive samples by the model. Guided by this, this case argues that regions in the same image naturally possess similar visual characteristics, especially regions near the infringing and infringed parts. Therefore, in CopyNCE, a contrast noise set RX is constructed by collecting all image patches in the reference image, i.e., RX = RR, where RR represents all image patches in the reference image r. This contrast noise set design preserves the hard negative samples in the reference image. Hard negative samples are image patches spatially adjacent to the positive sample image patches in the reference image. Due to local texture continuity (such as repeated patterns), the two usually exhibit high visual similarity, thus posing a challenging interference factor for contrast judgment.

[0196] As can be seen above, in the CopyNCE loss, this case utilizes the accurate pixel overlap information provided by the PixTrace module.Information, a clean and noise-free prior distribution q is constructed. Through this prior distribution q, the true overlap ratio of the infringing area between the query image patch and the reference image patch can be accurately quantified, taking into account the importance differences of different image patches. It also has the following effects: the model (for image infringement detection model) whose loss function includes the CopyNCE loss can receive high-quality, low-noise supervision signals during training, and can avoid conflict gradients caused by incorrect matching, which can greatly improve the model's ability to learn accurate and robust correspondences.

[0197] Figure 3 shows an example of calculating the area ratio of the shared area of ​​the corresponding image patches between the query image and the reference image. This area ratio is calculated using the formula (6) above. As shown in Figure 3, for a query image block in the query image, four reference image blocks #1, #2, #3 and #4 are found in the reference image r to be corresponding candidate positive sample image blocks. And after calculation, the area ratios of the shared regions of the four reference image blocks #1, #2, #3 and #4 with the query image block are 12%, 20%, 20% and 48%, respectively.

[0198] Further, when performing infringement detection on two images, since the correspondence between infringement and being infringed is bidirectional, this is because: one image may be an infringing edited version of another image, but conversely, the other image may also be an infringing edited version of the corresponding image. The purpose of image infringement detection is to analyze whether there is an infringement relationship between the two images, but it is impossible to determine which image is the infringing edited version. Based on this, the expression form of CopyNCE is finally derived to be symmetrical, as shown in the following formula (7):

[0199]

[0200] III. Composition of Loss Function in Image Infringement Detection Model

[0201] In order to integrate CopyNCE into the training pipeline of the corresponding image infringement detection model and to endow the corresponding image infringement detection model (such as descriptor or matcher) with basic infringement detection capabilities, this case uses CopyNCE as an auxiliary loss function and the original baseline loss function of the image infringement detection model such as descriptor or matcher to constitute the final loss function of the model.

[0202] That is, in this case, the total loss function in the image infringement detection model consists of two parts, one part is the baseline loss function (used to evaluate the basic performance of the model), and the other part is the CopyNCE loss (used to enhance the image infringement detection capability of the model).

[0203] For example, as shown below:

[0204] For the matcher, its total loss function Lmat is expressed as shown in the following formula (8):

[0205]

[0206] For the descriptor, its total loss function Ldes is expressed as shown in the following formula (9):

[0207]

[0208] where wNCE is the weight of the CopyNCE loss. In the matcher and the descriptor, the value of wNCE is generally different, although it can be the same in some other embodiments. Here, the value of wNCE is not specifically limited, and the value of wNCE can usually be between 1 and 5. For example, in the matcher, the value of wNCE can be 1; and in the descriptor, the value of wNCE can be 5.

[0209] The above represents the original baseline loss function in the matcher, which is defined as a simple binary classification function that can be used to evaluate the basic performance of the matcher. The above represents the original baseline loss function in the descriptor, which can be defined as the InfoNCE loss between the KoLeo loss and the global features of the image, which can improve the robustness and generalization ability of the descriptor.

[0210] In summary, this solution, through the aforementioned PixTrace module and CopyNCE loss, achieves the direct integration of fine-grained correspondence learning into the self-supervised training framework, forming a relatively unified and end-to-end image infringement detection method. Specifically, the PixTrace module generates supervisory signals during the data preparation stage, while the CopyNCE loss is integrated into the network training as an auxiliary loss, rather than requiring an additional independent object detection model or complex post-processing steps. The beneficial effects of this include: reducing the accumulation and propagation of errors that may exist between multi-stage tasks; simultaneously, this solution achieves higher detection performance in a more compact and efficient manner (see page 17 / 35 of CN 120997535 A), reducing overall computational complexity and deployment costs.

[0211] Based on the above content, the technical solution provided in this specification will be described below through method embodiments.

[0212] The execution subject of the method embodiments provided in this specification is an electronic device with logical operation functions. A corresponding image infringement detection model is deployed on the electronic device. The electronic device can be a server. The server can be a common server, a cloud server, or a virtual server, etc., and this application embodiment does not specifically limit it. Of course, in other embodiments, the electronic device can also be the client, in which case the client has high-performance computing capabilities. The client can be any terminal device such as a mobile phone, tablet computer, or smart wearable device. This specification mainly describes the execution subject of the method embodiment with the server as the main body.

[0213] Figure 5 shows a schematic flowchart of a data processing method provided by an embodiment of this specification. As shown in Figure 5, the data processing method includes the following steps:

[0214] 102. Obtain a first image and a second image;

[0215] 104. Perform bidirectional block-level comparison analysis on the first image and the second image, respectively using the first image as the first image and the second image as the second image.The query image and the second image are used as reference images, and the second image is used as the query image and the first image is used as the reference image. Based on the pixel coordinate table of the query image, the contrast loss value between the image block in the query image and the image block in the corresponding reference image is calculated;

[0216] Wherein, the pixel coordinate table is used to record the position information of the pixels in the query image; the contrast loss value optimizes the parameters of the image infringement detection model, and the image infringement detection model can detect whether there is an infringement relationship between the first image and the second image.

[0217] In the above 102, the first image and the second image can be input by the user through the interactive interface provided by the client, or they can be obtained by crawling from the corresponding data source platform (such as a website). This embodiment does not specifically limit the acquisition method of the first image and the second image.

[0218] Wherein, if there is an infringement relationship between the first image and the second image, one of the first image and the second image may be the original image, and the other image may be generated by editing the original image, or the first image and the second image are derived from the same image (i.e., they are generated by editing the same image).

[0219] Further, while continuing to compare and analyze the first image and the second image to determine whether there is an infringement relationship between them, one of the images can be used as the query image and the other as the reference image for comparison and analysis to determine the comparison loss value between the image blocks in the query image and the corresponding image blocks in the reference image. Then, another image can be used as the query image and the other image can be used as the reference image for comparison and analysis to determine the comparison loss value between the image blocks in the query image and the corresponding image blocks in the reference image. After that, the average value of the comparison loss value between the image blocks in the first image and the corresponding image blocks in the second image obtained through bidirectional analysis can be taken. Based on this average comparison loss value, it can be determined whether there is an infringement relationship between the first image and the second image.

[0220] Therefore, the bidirectional block-level comparison analysis of the first image and the second image described in 104 above may specifically include:

[0221] 1042. Using the first image as the query image and the second image as the reference image, performing block-level comparison analysis on the query image and the reference image based on the pixel coordinates of the query image, and calculating a first comparison loss value between the image blocks in the query image and the corresponding image blocks in the reference image; Specification 18 / 35 pages 22 CN 120997535 A

[0222] 1044. Using the second image as the query image and the first image as the reference image, performing block-level comparison analysis on the query image and the reference image based on the pixel coordinates of the query image, and calculating a second comparison loss value between the image blocks in the query image and the corresponding image blocks in the reference image.

[0223] In step 1042 above, the pixel coordinates of the query image are actually the first pixel coordinates of the first image (used to record the position information of pixels in the first image). And the first contrast loss value is actually: based on the first pixel coordinate table of the first image, block-level contrast analysis is performed on the first image and the second image, and the contrast loss value between the image block in the first image and the corresponding image block in the second image is calculated. For the specific implementation of this step 10426, it can be referred to the specific expression calculation given in formula (6) or formula (7) provided in the other embodiments above.

[0224] In step 1044 above, the pixel coordinates of the query image are actually the second pixel coordinates of the second image (used to record the position information of pixels in the second image). And the second contrast loss value is actually: based on the second pixel coordinate table of the second image, block-level contrast analysis is performed on the second image and the first image, and the contrast loss value between the image block in the second image and the corresponding image block in the first image is calculated. For the specific implementation of this step 1044, it can be referred to the specific expression calculation given in formula (7) provided in the other embodiments above.

[0225] Based on the above, it can be understood that the contrast loss value mentioned in 106 above includes a first contrast loss value and a second contrast loss value. The first contrast loss value is calculated when the first image is the query image and the second image is the reference image, and the second contrast loss value is calculated when the second image is the query image and the first image is the reference image.

[0226] Further, based on the formula (7) given in the other embodiments above, the average contrast loss value obtained by averaging the first contrast loss value and the second contrast loss value can optimize the parameters of the corresponding image infringement detection model. Therefore, the method provided in this embodiment may also include the following steps:

[0227] 106. Determine the average contrast loss value of the corresponding image block between the first image and the second image according to the second contrast loss value and the first contrast loss value;

[0228] Wherein, the average contrast loss value is used to optimize the parameters of the image infringement detection model to improve the performance of the image infringement detection model.

[0229] When using the above-described image infringement detection model to perform infringement analysis on the first image and the second image to determine whether there is an infringement relationship between them, the inspection result output by the image infringement detection model for the first image and the second image will include a numerical value, which can be, for example, between 0 and 1, to represent the probability that there is an infringement relationship between the first image and the second image. That is, the output value is an infringement probability. Of course, the inspection result output by the image infringement inspection model may also include other content, such as a heatmap. The description of the heatmap will be described in detail in the relevant content below, and will not be elaborated here.

[0230] The following describes in detail the specific implementation of "calculating the contrast loss value between the image block in the query image and the image block in the corresponding reference image based on the pixel coordinates of the query image" in step 104 above, in conjunction with formula (6) given in other embodiments above and formulas (4) to (5) related to formula (6).

[0231] In step 104 above, a set coordinate table can be initialized according to the coordinates of the pixels in the query image, so as to determine the pixel coordinate table corresponding to the query image. The pixel coordinate table records the coordinate position information of each pixel in the query image in the first image. Specification 19 / 35 pages 23 CN 120997535 A

[0232] When calculating the contrast loss value between the image block in the query image and the image block in the corresponding reference image based on the pixel coordinates corresponding to the query image, in order to realize the block-level comparison analysis of the query image and the reference image, it is necessary to first divide the query image and the reference image according to a certain image block size to obtain multiple image blocks corresponding to the query image and the reference image respectively. In this embodiment, a block-level comparative analysis of the query image and the reference image is achieved by using a trained image infringement detection model. The image infringement detection model can be trained based on a corresponding preset model (such as a matcher or descriptor), and the loss function used by the matcher or descriptor during training includes the geometrically guided contrast loss function (CopyNCE) given by formula (7) in the aforementioned other embodiments. Thus, the loss function of the image infringement detection model includes the geometrically guided contrast loss function (CopyNCE).

[0233] One way to divide the image is to directly divide the query image and the reference image according to a certain image block size. That is, in one feasible solution, the method provided in this embodiment may also include the following steps:

[0234] S162, divide the query image according to a set image block size to obtain multiple query image blocks corresponding to the query image;

[0235] S164, divide the reference image according to a set image block size to obtain multiple reference image blocks corresponding to the reference image.

[0236] Alternatively, to simplify the subsequent comparative analysis of image blocks, regions of interest can be extracted first for the query image and the reference image to segment out the corresponding query region from the query image and the corresponding reference image region from the reference image, respectively. The query region and the reference region are identified candidate region pairs that may have an infringement relationship. Further, the query region and the reference region can be divided separately. Based on this, in another feasible technical solution, the method provided in this embodiment may also include the following steps:

[0237] S162', Determine a query region in the query image and a reference region in the reference image; The query region and the reference region are candidate region pairs that have an infringement relationship;

[0238] S164'. Divide the query region according to the set image block size to obtain multiple query image blocks;

[0239] S166'. Divide the reference region according to the set image block size to obtain multiple reference image blocks.

[0240] The set image block size mentioned above is related to the visual coding model included in the image infringement detection model used. Specifically, the set image block size is the size of the smallest unit image block (also called the smallest processing image block) of the visual coding model. For example, if the visual architecture of the visual encoder is ViT, then the set image block size can be 16×16 pixels.

[0241] Assume that the target query image block is any one of the multiple query image blocks corresponding to the query image, and the reference image has at least one candidate reference image block corresponding to the target query image block. The at least one candidate reference image block can be selected from the multiple reference image blocks corresponding to the reference image according to the first pixel coordinate table of the query image. For example, the pixel coordinate table of the query image can be reversed, thereby determining the coordinate position corresponding to mapping each pixel coordinate in the target query image to the reference image based on the obtained reverse coordinate table. This determines which reference image blocks in the reference image the specific coordinates of each pixel in the target query image are mapped to, and these determined reference image blocks are used as candidate reference image blocks. Thus, the target query image and the candidate reference image blocks have a pixel correspondence. It can be understood that this pixel correspondence means that when performing pixel coordinate mapping on the target query image block, the coordinates of one or more pixels in the target query image block will be mapped to the candidate reference image blocks.

[0242] Based on the above, the method provided in this embodiment may further include the following steps:

[0243] S22. Based on the pixel coordinate table of the query image, at least one candidate reference image block is determined from the reference image for the target query image block.

[0244] Specifically, in one example, the implementation of step S22 above includes: Specification 20 / 35 Page 24 CN 120997535 A

[0245] S222, perform a reversal operation on the pixel coordinate table to obtain a reversed coordinate table;

[0246] S224, based on the reversed coordinate table, determine the coordinates of each pixel in the target query image block mapped to the corresponding coordinate positions in the reference image, and obtain a mapping result;

[0247] S226, based on the mapping result, determine at least one candidate reference image block from the reference image.

[0248] In a specific implementation, the above at least one candidate reference image block is determined from multiple reference image blocks corresponding to the reference image.

[0249] Let the target candidate reference image block be any one of the at least one candidate image block, then: in step 104 aboveThe included "calculating the contrast loss value between the target query image patch and the target candidate reference image patch based on the pixel coordinate table of the query image" may include the following steps:

[0250] S42. Determine the prior distribution probability of the target candidate reference image patch relative to the target query image patch based on the pixel coordinate table.

[0251] S44. Determine the recognition probability of correctly identifying the target candidate reference image patch from the set of noisy image patches according to the target query image patch and the associated set of noisy image patches; wherein, the set of noisy image patches is constructed based on all image patches in the reference image.

[0252] S46. Calculate the contrast loss value between the target query image patch and the target candidate reference image patch according to the prior distribution probability and the recognition probability.

[0253] In step S42 above, as shown in formula (4) given in the other embodiments above, the target query image block here can correspond to the query image block in formula (4) and the target candidate reference image block can correspond to the reference image block in formula (4). Based on formula (4), step S42 can be implemented by performing the following:

[0254] First, the pixel coordinate table of the query image is reversed to obtain the corresponding reverse coordinate table. This reverse coordinate table can represent the coordinate mapping relationship corresponding to the coordinate position of the pixel in the query image to the coordinate position in the reference image. Based on this reverse coordinate table, the number of pixels in the target query image block mapped to the target candidate reference image block after coordinate mapping can be determined. Further, the ratio of the number of pixels mapped to the target candidate reference image block to the total number of pixels in the target query image block is calculated, and this ratio is used as the initial prior distribution probability (also known as the prior estimated probability). Then, the initial prior distribution probability is normalized. Specifically, as shown in formula (5) in the other embodiments above, during the normalization process, it is also necessary to determine the initial prior distribution probability of each candidate reference image block other than the target candidate reference image block in at least one candidate reference image block relative to the target query image block, and according to the preset confidence adjustment parameter γ, to perform a power operation on the initial prior distribution probability corresponding to each candidate reference image block in at least one candidate reference image block, and to sum all the power operation results to obtain a sum value (that is, the sum value used in formula (5) to normalize the initial prior distribution probability corresponding to the target candidate reference image block; in specific implementation, the ratio of the γ-th power operation result of the initial prior distribution probability corresponding to the target candidate reference image block to the sum value is calculated, thereby obtaining the final prior distribution probability corresponding to the target candidate reference image block).

[0255] Based on the above content, in a specific implementable technical solution, the above S42 "based on the pixel coordinate table, determineThe "prior distribution probability of the target candidate reference image block relative to the target query image block" may include:

[0256] S422, determining the number of pixels in the target query image block mapped to the target candidate reference image block based on the pixel coordinate table; Specification 21 / 35 pages 25 CN 120997535 A

[0257] S424, calculating the ratio of the number of pixels to the total number of pixels in the target query image block;

[0258] S426, performing a power operation on the ratio according to a preset confidence adjustment parameter to obtain the power operation result;

[0259] S428, normalizing the power operation result to obtain the prior distribution probability.

[0260] Here, by introducing a confidence adjustment parameter, the flexibility of the distribution can be enhanced, the probability of the strongest match can be increased, and the situation of multiple candidate reference image blocks can be verified. In addition, by using pixel overlap information to determine the prior distribution probability corresponding to the candidate reference image block, this prior distribution probability can accurately quantify the real overlap ratio of the infringing area between the target query image block and the corresponding candidate reference image block, and can take into account the importance difference between different image blocks.

[0261] In the above S44, the recognition probability calculation can be understood as calculating the formula (6) given in the other embodiments above. Referring to formula (3), it is not difficult to know that:

[0262]

[0263] Wherein,

[0264]

[0265] and, the above z represents the image feature generated by encoding the corresponding image block by the visual coding model fθ. For example, it represents the image feature generated by encoding (such as the target query image block) by the visual coding model fθ; it represents the image feature generated by encoding (such as the target candidate reference image block) by the visual coding model fθ. And so on, the meaning can also be known.

[0266] From this content and combined with the above description of image patch size, it can be seen that when calculating the recognition probability, this scheme directly performs feature extraction and related calculations at the smallest unit image patch level of the visual coding model fθ (such as ViT or CNN). By operating on the smallest unit image patch, information loss can be avoided, and fine spatial details in the image can be guaranteed and utilized, which helps to improve the accuracy of image infringement area location using this recognition probability.

[0267] Further, based on the formula (6) given in the other embodiments above, in a specific implementable technical solution, the above step S46 "calculate the contrast loss value between the target query image patch and the target candidate reference image patch based on the prior probability and the recognition probability" may include:

[0268] S462, determining the probability difference measure between the prior distribution probability and the recognition probability;

[0269] S464, determining the contrast loss value based on the probability difference measure.

[0270] In the above, the probability difference measure can be, but is not limited to, KL divergence or JS divergence. In this embodiment, KL divergence is preferred. Further, the probability difference measure can be subjected to data expectation operation to obtain the corresponding contrast loss value.

[0271] As can be seen from the above description, this solution can directly perform comparison analysis at the image block level. Specifically, based on decomposing the image in the image pair into the smallest processed image blocks, the relevant formula (i.e., the relevant CopyNCE loss function) provided in this specification can be used to accurately locate the similarity (or matching, etc.) between image blocks of different images. The effect of this includes: enabling the image infringement detection model containing the relevant formula (i.e., the CopyNCE loss function) in the loss function to learn more detailed similarity correspondence between local regions than the whole image, thereby effectively identifying and locating complex local editing and tampering, such as image matting, local transformation, etc., rather than just judging the similarity between the whole image.

[0272] Furthermore, this application also visualizes the similar regions between the first image and the second image. This visualization is achieved by drawing a heatmap based on the similarity of corresponding image blocks between the first image and the second image. Based on this, the method provided in this embodiment may further include the following steps:

[0273] 108. Generate a heatmap reflecting the similar regions between the first image and the second image based on the similarity of corresponding image blocks between the first image and the second image;

[0274] 1010. Display the heatmap.

[0275] In step 108 above, specifically, a heatmap reflecting the similar regions between the first image and the second image is drawn based on the similarity of all image block pairs that have pixel correspondence between the first image and the second image. Specifically, this heatmap is drawn based on the similarity measure of the image block pairs, and the similarity measure of an image block pair is determined by performing a corresponding transformation operation on the similarity of the image block pair. The similarity mentioned here can refer to, but is not limited to, cosine similarity. The transformation operation is a non-linear transformation operation, which includes, but is not limited to, exponential operations (i.e., exp) with the base e of the natural logarithm.

[0276] For example, the similarity measure of image patch pairs can be calculated by the following formula:

[0277]

[0278] Wherein, represents the similarity measure between the i-th query image patch in query image q (e.g., the i-th image patch in the first image) and the j-th reference image patch in the corresponding reference image r (e.g., the j-th image patch in the second image). represents the image feature of the i-th query image patch, represents the image feature of the j-th reference image patch. represents the cosine similarity calculated based on the image features of the two image patches.

[0279] It should be noted here that, in conjunction with formulas (6) and (3) given in other embodiments of this specification, it can be seen that the similarity between corresponding image blocks between the first image and the second image, and the similarity measure obtained by performing corresponding transformation operations on the similarity, are actually obtained in the process of calculating the contrast loss value of corresponding image blocks between the first image and the second image (intermediate values ​​generated in the process of calculating the contrast loss value).

[0280] In addition, the heat map generated above includes at least one of a region affinity heat map and an affinity entropy heat map. The region affinity heat map is drawn directly based on the similarity measure of corresponding image blocks between the first image and the second image. The affinity entropy heat map is drawn based on the affinity entropy calculated using the similarity measure of corresponding image blocks between the first image and the second image. For example, the affinity entropy can be calculated by the following formula:

[0281] Wherein,

[0282] Wherein, ξi represents the affinity entropy, and the relevant parameters can be found in the aforementioned related content, which will not be elaborated here. This represents the sum of similarity metrics between all pairs of image blocks that have pixel correspondences between the query image q (e.g., the first image) and the reference image r (e.g., the second image). This represents the image feature of the k-th reference image block.

[0283] It should be noted here that: in conjunction with the aforementioned content related to step 104 or other embodiments, the specification page 23 / 35, 27 CN 120997535 A is provided. Equation (7) shows that the infringement analysis performed on the first image and the second image in this specification is a bidirectional block-level comparison analysis. Therefore, in the process of performing block-level comparison analysis on the two images with the first image as the query image q and the second image as the reference image r to calculate the first comparison loss value of the corresponding image blocks between the first image and the second image, a first similarity between the corresponding image blocks between the first image and the second image and a first similarity measure obtained by performing a corresponding transformation operation on the first similarity will be generated. Conversely, in the process of performing block-level comparison analysis on the two images with the second image as the query image q and the first image as the reference image r to calculate the second comparison loss value of the corresponding image blocks between the first image and the second image, a second similarity between the corresponding image blocks between the second image and the first image and a second similarity measure obtained by performing a corresponding transformation operation on the second similarity will be generated. Based on this, the heat map drawn in this embodiment can be generated based on the first similarity measure obtained in the process of calculating the first comparison loss and / or the second similarity measure obtained in the process of calculating the second comparison loss. When drawing a heatmap based on a first similarity metric and a second similarity metric, at least one of the following two methods is included.One approach is to directly draw a region affinity heatmap based on the average of the first similarity measure and the second similarity measure; another approach is to first calculate the first affinity entropy corresponding to the first similarity and the second affinity entropy corresponding to the second similarity measure, and then draw an affinity entropy heatmap based on the average of the first affinity entropy and the second affinity entropy.

[0284] Figure 6a shows two heatmaps generated during the image infringement detection process using different types of image infringement detection models (such as matchers or descriptors, whose loss functions include the CopyNCE loss provided in this specification) in this embodiment, namely, a region affinity heatmap and an affinity entropy heatmap. In Figure 6a, a squirrel's eye is selected as the infringed object. It can be clearly seen from the heatmap given in Figure 6a that the squirrel's eye area is clearly highlighted and the display is relatively concentrated, which can present a clear outline of the infringed object, which can effectively reveal the infringement correspondence between different images. In contrast, as shown in Figure 6b, the heatmap generated during image infringement detection using an image infringement detection model that does not integrate the CopyNCE loss provided in this specification appears messy and cannot effectively reveal the infringement correspondence between different images.

[0285] In the above 1010, the heatmap is displayed through the client interface. For example, as shown in Figure 7, if the image infringement detection model is used on the server side to complete the image infringement correspondence analysis between the first image and the second image, the server can send the generated heatmap to the client for display.

[0286] Therefore, the heatmap generated here can significantly improve the interpretability of the infringement correspondence and help improve the understanding.

[0287] Another embodiment of this specification also provides a data processing method. The data processing method includes the following steps:

[0288] 002, acquiring a first image and a second image;

[0289] 004, determining a first pixel coordinate table for the first image; the first pixel coordinate table is used to record the position information of pixels in the first image;

[0290] 006, based on the first pixel coordinate table, performing block-level comparison analysis on the first image and the second image, and calculating a first comparison loss value between image blocks in the first image and corresponding image blocks in the second image;

[0291] wherein, the first comparison loss value is used to optimize the parameters of the image infringement detection model, and the image infringement detection model can detect whether there is an infringement relationship between the first image and the second image.

[0292] For the specific implementation of each step given in this embodiment, please refer to the relevant content in other embodiments, which will not be elaborated here. In addition, the method provided in this embodiment may also include some specifications disclosed in other embodiments. 24 / 35 pages 28 CNStep 120997535 A can also be found in other embodiments, and will not be repeated here.

[0293] Figure 8 shows a flowchart of a data processing method provided in another exemplary embodiment of this specification. As shown in Figure 8, the method includes the following steps:

[0294] 202, acquiring a first image and a second image;

[0295] 204, inputting the first image and the second image into an image infringement detection model, executing the image infringement detection model to output detection results;

[0296] 206, displaying the detection results.

[0297] The detection result includes a heatmap reflecting similar regions between the first image and the second image. The heatmap is generated based on similarity information generated during the bidirectional block-level comparison analysis of the first image and the second image. The bidirectional block-level comparison analysis includes: using the first image as the query image and the second image as the reference image, and using the second image as the query image and the first image as the reference image, respectively, calculating the comparison loss value between image blocks in the query image and image blocks in the corresponding reference image based on the pixel coordinate table of the query image. The similarity information includes the similarity between image blocks in the query image and image blocks in the corresponding reference image obtained during the calculation of the comparison loss value.

[0298] The loss function of the above image infringement detection model includes the CopyNCE loss provided in other embodiments of this specification. In addition to the heatmap, the output detection result may also include an infringement probability value, which represents the probability that there is an infringement relationship between the first image and the second image. This probability is between 0 and 1.

[0299] For the specific implementation of each step given in the above embodiment, please refer to the relevant content in other embodiments. It will not be described in detail here. In addition, the method provided in this embodiment may also include some steps disclosed in other embodiments, and the relevant content in other embodiments can also be referred to, which will not be repeated here.

[0300] The solution provided in this embodiment makes the output detection result include a heat map, which can significantly improve the interpretability of the infringement correspondence and help improve the understanding. In addition, it can also improve the reliability of the image detection model, and at the same time provide users with more intuitive infringement area location information, which helps in subsequent content analysis and decision-making, such as determining whether there is an infringement relationship.

[0301] Figure 9 shows a flowchart of a data processing method provided in another exemplary embodiment of this specification. As shown in Figure 9, the method includes the following steps:

[0302] 302, acquiring a first image and a second image;

[0303] 304, inputting the first image and the second image into an image infringement detection model, executing the image infringement detection model to output the detection result; wherein, the loss function of the image infringement detection model includes a composite contrast loss function,The composite contrast loss function is constructed based on the first contrast loss function and the second contrast loss function. Using the first contrast loss function, based on the first pixel coordinate table of the first image, block-level contrast analysis can be performed on the first image and the second image to calculate the first contrast loss value between the image block in the first image and the corresponding image block in the second image. Using the second contrast loss function, based on the second pixel coordinate table of the second image, block-level contrast analysis can be performed on the second image and the first image to calculate the second contrast loss value between the image block in the second image and the corresponding image block in the first image.

[0304] The composite contrast loss function described above is the geometrically guided contrast loss function (CopyNCE loss) described in other embodiments of this specification. For a detailed description of the composite contrast loss function, please refer to the content related to CopyNCE loss provided in other embodiments, which will not be elaborated here.

[0305] For the specific implementation of each step given in the above embodiment, please refer to the relevant content in other embodiments, which will not be elaborated here at page 25 / 35, 29 CN 120997535 A. In addition, the method provided in this embodiment may also include some steps disclosed in other embodiments, and the relevant content in other embodiments can also be referred to, which will not be repeated here.

[0306] Figure 10 shows a schematic flowchart of a data processing method provided in another exemplary embodiment of this specification. As shown in Figure 10, the method includes the following steps:

[0307] 402. Obtain a training sample set, which contains multiple sample image pairs;

[0308] 404. Set the loss function used during the training of the contrastive learning model;

[0309] 405. Based on the multiple sample image pairs and the loss function, train the contrastive learning model to obtain an image infringement detection model;

[0310] Wherein, the loss function includes a composite contrastive loss function, which is constructed based on a first contrastive loss function and a second contrastive loss function; using the first contrastive loss function, based on the first pixel coordinate table of one sample image in the sample image pair, a block-level contrastive analysis can be performed on the two sample images contained in the sample image pair to calculate the first contrastive loss value of the corresponding image block between the two sample images; using the second contrastive loss function, based on the second pixel coordinate table of the other sample image in the sample image pair, a block-level contrastive analysis can be performed on the two sample images contained in the sample image pair to calculate the second contrastive loss value of the corresponding image block between the two sample images.

[0311] In the above, the training sample set may utilize publicly available image sample sets, such as ImageNet. And, the aforementioned...The contrastive learning model can be a matcher or a descriptor. The loss function used during the training of the contrastive learning model includes a baseline loss function and a composite contrastive function. The composite contrastive loss function is the geometrically guided contrastive loss function (CopyNCE loss) described in other embodiments of this specification.

[0312] For the specific implementation of each step given in the above embodiment, please refer to the relevant content in other embodiments, which will not be elaborated here. In addition, the method provided in this embodiment may also include some steps disclosed in other embodiments, which can also be referred to the relevant content in other embodiments, which will not be elaborated here.

[0313] Another exemplary embodiment of this specification also provides an image infringement detection model, which includes multiple network layers, including an input layer, an encoding layer, and a loss layer. Wherein,

[0314] the input layer is used to receive the input first image and the second image;

[0315] the encoding layer has two encoding modules, one of which is used to encode the first image block in the first image to output the image block features of the first image block, and the other encoding module is used to encode the second image block in the second image to output the image block features of the second image block;

[0316] the loss layer is provided with a loss function, the loss function including: a composite contrast loss function; the composite contrast loss function is constructed based on the first contrast loss function and the second contrast loss function; the first contrast loss function is used to perform block-level contrast analysis on the first image and the second image based on the first pixel coordinate table of the first image and the image block features output by the encoding layer, and calculate the first contrast loss value of the corresponding image block between the first image and the second image; the second contrast loss function is used to perform block-level contrast analysis on the second image and the first image based on the second pixel coordinate table of the second image and the image block features output by the encoding layer, and calculate the second contrast loss value of the corresponding image block between the second image and the first image;

[0317] The first contrast loss value and the second contrast loss value are used to optimize the parameters of the image infringement detection model. The image infringement detection model can detect whether there is an infringement relationship between the first image and the second image.

[0318] In the above, the encoding layer has two encoding modules, namely encoders. One encoding module is used to encode the image blocks in the first image (page 26 / 35, CN 120997535 A), and the other encoding module is used to encode the image blocks in the second image. Here, it can be understood that encoding processing refers to feature extraction of the corresponding image blocks, realizing the transformation of image block data into a series of image feature vectors. These image features retain the key information of the image blocks. The two encoding models can share weights.

[0319] And, the loss function deployed on the loss layer includes the baseline loss function and the composite contrast loss function. The composite contrast loss function is the geometrically guided contrast loss function (CopyNCE loss) described in other embodiments of this specification.

[0320] In addition to the above, the image infringement detection model may also include other network layers. For example, referring to Figure 4, when the image infringement detection model is a matcher, it may also include other network layers such as connection layer (Concat), fusion layer (Fusion), and head layer (Head). Connection layer is used to establish direct or indirect connection between the outputs of two encoding modules, and its form may be skip connection, dense connection, etc. Fusion layer is used to integrate feature information from different sources or branches. Head layer is usually located at the end of the entire network and is responsible for converting the learned feature mapping into the final task output.

[0321] In this embodiment, the image infringement detection model may be other types besides a matcher, such as a descriptor.

[0322] Using the image infringement detection model provided in this embodiment to perform image infringement detection analysis can achieve the following effects:

[0323] 1) High detection performance.

[0324] Test analysis was conducted using authoritative datasets in the field of image infringement detection (ICD), such as the DISC21 dataset. With the image infringement detection models being a matcher and a descriptor, both the matcher and descriptor achieved state-of-the-art (SOTA) performance. Specifically, the matcher achieved approximately 88.7% μAP and 83.9% RP90, while the descriptor achieved approximately 72.6% μAP and 68.4% RP90.

[0325] 2) Stronger generalization ability:

[0326] On datasets containing more difficult negative samples, such as the NDEC dataset, the composite alignment loss (CopyNCE loss) included in the image infringement detection model also performed well, with its μAP significantly higher than other strong baseline losses. For example, the μAP of the CopyNCE loss reached approximately 72.5%, while the μAP of other strong baseline losses (such as Strong ASL) reached approximately 64.1%. This indicates that the proposed scheme has better robustness and adaptability when dealing with more challenging real-world data.

[0327] On the AnyPattern dataset, the image detection model needs to deal with more diverse image data editing methods, and none of these image data editing methods have been used in the training phase. Under the same settings, the performance of the composite alignment loss (CopyNCE loss) can reach approximately 28.1% μAP and 34.4% R@1.

[0328] 3) Higher efficiency:

[0329] By integrating the composite alignment loss (CopyNCE loss) into the image infringement detection model, fine-grained learning can be internalized into the self-supervised training process, avoiding dependence on additional complex components (such as object detectors), makingThe overall system is more streamlined and efficient.

[0330] 4) The model has good robustness:

[0331] The composite alignment loss (CopyNCE loss) can show good robustness in model training. For example, in the descriptor, even if the CopyNCE loss is calculated in different layers (such as the 10th layer), similar results can be obtained, which further confirms the effectiveness of its design.

[0332] For a detailed description of the specific function of the image infringement detection model given in the above embodiment, please refer to the relevant content in other embodiments, which will not be elaborated here. Specification 27 / 35 pages 31 CN 120997535 A

[0333] Figure 11 shows a schematic flowchart of a data processing method provided by another exemplary embodiment of this specification. As shown in Figure 11, the method includes the following steps:

[0334] 502. Determine the first pixel coordinate table of the first image and the second pixel coordinate table of the second image;

[0335] 504. Perform a reversal operation on the first pixel coordinate table to obtain a reverse coordinate table;

[0336] 506. Based on the reverse coordinate table and the second pixel coordinate table, determine the pixel correspondence between the first image and the second image;

[0337] 508. Based on the pixel correspondence, perform block-level comparison analysis on the first image and the second image, and calculate the comparison loss value between the image block in the first image and the corresponding image block in the second image.

[0338] Steps 502, 504, and 506 above correspond to the reverse tracing related content supported by the PixTrace module in other embodiments of this specification.

[0339] For the specific implementation of each step given in this embodiment, please refer to the relevant content in other embodiments, which will not be elaborated here. In addition, the method provided in this embodiment may also include some steps disclosed in other embodiments, which can also be referred to the relevant content in other embodiments, which will not be elaborated here.

[0340] FIG12 shows a flowchart of a data processing method provided in another exemplary embodiment of this specification. As shown in FIG12, the method includes the following steps:

[0341] 602. Obtain a first pixel coordinate table of a first image and a second pixel coordinate table of a second image; wherein the first image and the second image are generated by editing a third image, and the first pixel coordinate table is generated by mapping the coordinates of pixels in the third image to the first image, and the second pixel coordinate table is generated by mapping the coordinates of pixels in the third image to the second image;

[0342] 604. Perform a reversal operation on the first pixel coordinate table to generate a reverse coordinate table;

[0343] 605. Based on the second pixel coordinate table and the reverse coordinate table, calculate the coordinates of pixels in the first image.Mapped to the second image, the pixel correspondence between the first image and the second image is obtained;

[0344] Wherein, the pixel correspondence is used to participate in the calculation of the contrast loss value between the image block in the first image and the corresponding image block in the second image.

[0345] In the above steps 604 to 605, after the first pixel coordinate table is reversed, the coordinates of each pixel in the first image will be mapped to the coordinate system of the third image, thereby generating a reverse coordinate table. Then, based on the second pixel coordinate table, the composite function used for coordinate transformation in the process of generating the second image by editing the third image can be determined. The coordinates in the reverse coordinate table can be updated again using the composite function, thereby realizing the mapping of the coordinates of the pixels in the first image to the second image, thereby obtaining the pixel correspondence between the first image and the second image.

[0346] The above steps 602, 604 and 606 specifically correspond to the image bridging tracking related content supported by the PixTrace module in other embodiments of this specification.

[0347] For the specific implementation of each step given in the above embodiment, please refer to the relevant content in other embodiments, which will not be elaborated here. In addition, the method provided in this embodiment may also include some steps disclosed in other embodiments, which can also be referred to the relevant content in other embodiments, which will not be elaborated here.

[0348] The above-mentioned method embodiments mainly describe the technical solution provided by this specification from the perspective of images. This technical solution can also be applied to video infringement detection and analysis scenarios. Assuming that the video duration of the first video is one minute and the video duration of the second video is 30 seconds, when performing video infringement detection and analysis on the first video and the second video, the corresponding detection process generally includes the following steps:

[0349] Step 1: Extract frames from the first video and the second video respectively. For example, extract 1 frame every 1 second. Then, 60 frames will be extracted from the video on page 28 / 35 of the specification, CN 120997535 A, and 30 frames will be extracted from the second video;

[0350] Step 3: Determine the image features of each extracted frame. Then: the first video will correspond to 60 image features, and the second video will correspond to 30 image features.

[0351] Step 5: Perform matching operations on the 60 image features corresponding to the first video and the 30 image features corresponding to the second video, and calculate the similarity of the image features between the first video and the second video, thus obtaining a 60*30 similarity matrix.

[0352] Step 7: Based on the similarity matrix, determine whether there is an infringement relationship between the first video and the second video.

[0353] Generally speaking, if a video segment in the first video and a video segment in the second video are related, the two videos will be considered as similar.If the segments infringe on each other, it will be found that the values ​​on the diagonal of the obtained similarity matrix will be higher, and the values ​​on the off-diagonal will be lower. After visualizing this similarity matrix, the brightness of the diagonal will also be higher (that is, there will be a brighter diagonal line in the similarity feature image). A target detector can be used to specifically find this diagonal line, thereby outputting where such a diagonal line that meets the brightness requirement was found on this similarity matrix. Then, based on the diagonal line found, the position is mapped to the first video and the second video, so it can be determined that a certain segment of the second video infringes on a certain segment of the first video.

[0354] The video infringement detection process described above is a standard detection process that can run on any image infringement detection model. Preferably, the image infringement detection model used in video infringement detection analysis is a descriptor, because the matcher will perform image reconstruction during analysis, and the analysis process is more complicated than that of the descriptor. Furthermore, if the image infringement detection model is obtained using the training method provided in this specification, that is, the loss function used during training includes the geometrically guided contrast loss function (CopyNCE loss) provided in this specification, the resulting video infringement detection analysis effect will also be relatively good.

[0355] Based on the above, another exemplary embodiment of this specification provides a data processing method. As shown in Figure 13, the method includes the following steps:

[0356] 702. Obtain a first video and a second video;

[0357] 704. Perform frame extraction processing on the first video to obtain a first number of frame images extracted from the first video;

[0358] 706. Perform frame extraction processing on the second video to obtain a second number of frame images extracted from the second video;

[0359] 708. Using the trained image infringement detection model, perform the following: analyze the similarity between the first number of frame images and the second number of frame images pairwise to obtain a similarity matrix; based on the similarity matrix, determine whether there is an infringement relationship between the first video and the second video.

[0360] Wherein, the loss function used by the image infringement detection model during training includes a composite contrast loss function; the geometrically guided contrast loss function is constructed based on a first contrast loss function and a second contrast loss function; using the first contrast loss function, based on the first pixel coordinate table of one sample image in the sample image pair, block-level contrast analysis can be performed on the two sample images contained in the sample image pair, and the first contrast loss value of the corresponding image block between the two sample images can be calculated; using the second contrast loss function, based on the second pixel coordinate table of the other sample image in the sample image pair, block-level contrast analysis can be performed on the two sample images contained in the sample image pair, and the second contrast loss value of the corresponding image block between the two sample images can be calculated.

[0361] For the specific implementation of each step given in the above embodiment, please refer to the relevant content in other embodiments, which will not be elaborated here. In addition, the method provided in this embodiment may also include some steps disclosed in other embodiments, such as those on pages 29 / 35 of the specification 33 CN 120997535 A, which will also be described in other embodiments, and will not be elaborated here.

[0362] Another embodiment of this specification provides a data processing system. Referring to the exemplary data processing system structure shown in Figure 7, the system includes: a server 200 and a client 100.

[0363] The server 200 is equipped with an image infringement detection model, used for: acquiring a first image and a second image; inputting the first image and the second image into the image infringement detection model, executing the image infringement detection model, and outputting detection results;

[0364] The client 100 is used to display the detection results;

[0365] Wherein, the detection results include a heatmap reflecting similar regions between the first image and the second image, the heatmap being generated based on similarity information generated during the bidirectional block-level comparison analysis of the first image and the second image; the bidirectional block-level comparison analysis includes: using the first image as the query image and the second image as the reference image, and using the second image as the query image and the first image as the reference image, respectively, calculating the comparison loss value between image blocks in the query image and image blocks in the corresponding reference image based on the pixel coordinate table of the query image; the similarity information includes the similarity between image blocks in the query image and image blocks in the corresponding reference image obtained during the calculation of the comparison loss value.

[0366] For the specific implementation of the terminal functions given in this embodiment, please refer to the relevant content in other embodiments, which will not be elaborated here. In addition, the terminal functions provided in this embodiment may also include some functions / steps disclosed in other embodiments, which can also be referred to the relevant content in other embodiments, which will not be elaborated here.

[0367] The device embodiments corresponding to the method embodiments provided in this specification will be described below.

[0368] FIG14 shows a schematic diagram of the structure of a data processing device provided in an exemplary embodiment of this specification. As shown in FIG14, the device includes: an acquisition module 82 and a comparison analysis module 84. The acquisition module 82 is used to acquire a first image and a second image. The comparison analysis module 84 is used to perform bidirectional block-level comparison analysis on the first image and the second image, respectively using the first image as the query image and the second image as the reference image, and using the second image as the query image and the first image as the reference image, and based on the pixel coordinate table of the query image, calculate the comparison loss value between the image block in the query image and the corresponding image block in the reference image; wherein, the pixel coordinate table is used to record the position of the pixel in the query image.The contrast loss value is used to optimize the parameters of the image infringement detection model, which can then be used to detect whether there is an infringement relationship between the first image and the second image.

[0369] In one embodiment, the target query image block is an image block in the query image, and the reference image has at least one candidate reference image block corresponding to the target query image block. The target query image block and the candidate reference image block have a pixel correspondence. If the target candidate reference image block is one of the at least one candidate image block, then the comparison analysis module 84, when calculating the comparison loss value between the target query image block and the target candidate reference image block based on the pixel coordinate table of the query image, is specifically used for: determining the prior distribution probability of the target candidate reference image block relative to the target query image block based on the first pixel coordinate table; determining the recognition probability of correctly identifying the target candidate reference image block from the set of noise image blocks according to the target query image block and the associated set of noise image blocks; wherein the set of noise image blocks is constructed based on all image blocks in the reference image; and calculating the comparison loss value between the target query image block and the target candidate reference image block according to the prior distribution probability and the recognition probability.

[0370] In one embodiment, the comparison analysis module 84, when used to determine the prior distribution probability of the target candidate reference image block relative to the target query image block based on the pixel coordinate table, is specifically used to: determine the number of pixels in the target query image block mapped to the target candidate reference image block based on the pixel coordinate table; calculate the ratio of the number of pixels to the total number of pixels in the target reference image block; perform a power operation on the ratio according to a preset confidence adjustment parameter to obtain the power operation result; normalize the power operation result to obtain the prior distribution probability.

[0371] In one embodiment, the comparison analysis module 84, when used to calculate the comparison loss value between the target query image block and the target candidate reference image block based on the prior probability and the recognition probability, is specifically used to: determine the probability difference measure between the prior distribution probability and the recognition probability; determine the comparison loss value based on the probability difference measure.

[0372] In one embodiment, the apparatus further includes: a reversal module, configured to reverse the pixel coordinate table to obtain a reversed coordinate table. The determining module 84 is further configured to, based on the reversed coordinate table, determine the coordinates of each pixel in the target query image block mapped to the corresponding coordinate position in the reference image, obtaining a mapping result; and, based on the mapping result, determine the at least one candidate reference image block from the reference image.

[0373] In one embodiment, the determining module 84 is further configured to determine a query region in the query image and a reference region in the reference image; the query region and the reference region are candidate region pairs with an infringement relationship. The apparatus further includes a partitioning module, configured to partition the query region according to a set image block size to obtain multiple query image blocks; and to partition the reference region according to the set image block size to obtain multiple reference image blocks. The target query image block is any one of the multiple query image blocks, and the at least one candidate reference image block is determined from the multiple second image blocks.

[0374] In one embodiment, the apparatus further includes a determining module, configured to determine the average contrast loss value of corresponding image blocks between the first image and the second image based on a first contrast loss value and a second contrast loss value. The average contrast loss value is used to optimize the parameters of the image infringement detection model; the first contrast loss value is calculated with the first image as the query image and the second image as the reference image, and the second contrast loss value is calculated with the second image as the query image and the first image as the reference image.

[0375] Another exemplary embodiment of this specification also provides a data processing apparatus, the structure of which can be seen in the data processing apparatus shown in FIG14. Specifically, the apparatus includes: an acquisition module, a determination module, and a comparison analysis module. The acquisition module is used to acquire a first image and a second image. The determination module is used to determine a first pixel coordinate table of the first image; the pixel coordinate table is used to record the position information of pixels in the first image; the comparison analysis module is used to perform block-level comparison analysis on the first image and the second image based on the first pixel coordinate table, and calculate a first comparison loss value between the image block in the first image and the corresponding image block in the second image. Wherein, the first comparison loss value is used to optimize the parameters of the image infringement detection model, and the image infringement detection model can detect whether there is an infringement relationship between the first image and the second image.

[0376] FIG15 shows a schematic diagram of the structure of a data processing apparatus provided in another exemplary embodiment of this specification. As shown in FIG15, the apparatus includes: an acquisition module 92, an execution module 94, and a display module 96. The acquisition module 92 is used to acquire a first image and a second image. Execution module 94 is used to input the first image and the second image into an image infringement detection model, execute the image infringement detection model, and output detection results. Display module 96 is used to display the inspection results. The detection results include a heatmap reflecting similar regions between the first image and the second image, the heatmap being generated based on similarity information generated during bidirectional block-level comparison analysis of the first and second images.The generated; the bidirectional block-level comparison analysis includes: using the first image as the query image and the second image as the reference image, and using the second image as the query image and the first image as the reference image, respectively, calculating the comparison loss value between the image block in the query image and the corresponding image block in the reference image based on the pixel coordinate table of the query image; the similarity information is included in the similarity between the image block in the query image and the corresponding image block in the reference image obtained in the process of calculating the comparison loss value.

[0377] Another exemplary embodiment of this specification also provides a data processing device, the structure of which can be seen from part of the structure of the data processing device shown in FIG15. Specifically, the device includes: an acquisition module and an execution module. The acquisition module is used to acquire the first image and the second image. The execution module is used to input the first image and the second image into the image infringement detection model, execute the image infringement detection model, and output the detection result. The loss function of the image infringement detection model includes a composite contrast loss function, which is constructed based on a first contrast loss function and a second contrast loss function. Using the first contrast loss function, a block-level contrast analysis can be performed on the first image and the second image based on the first pixel coordinate table of the first image, and a first contrast loss value between an image block in the first image and a corresponding image block in the second image can be calculated. Using the second contrast loss function, a block-level contrast analysis can be performed on the second image and the first image based on the second pixel coordinate table of the second image, and a second contrast loss value between an image block in the second image and a corresponding image block in the first image can be calculated.

[0378] Figure 16 shows a schematic diagram of a model training device provided in another exemplary embodiment of this specification. As shown in Figure 16, the device includes: an acquisition module 1012, a setting module 1014, and a training module 1016. The acquisition module 1012 is used to acquire a training sample set, which contains multiple sample image pairs. The setting module 1014 is used to set the loss function used during the training of the contrast learning model. The training module is used to train the contrastive learning model based on the multiple sample image pairs and the loss function to obtain an image infringement detection model. The loss function includes a composite contrastive loss function, which is constructed based on a first contrastive loss function and a second contrastive loss function. Using the first contrastive loss function, based on the first pixel coordinate table of one sample image in the sample image pair, block-level contrastive analysis can be performed on the two sample images contained in the sample image pair to calculate the first contrastive loss value of the corresponding image blocks between the two sample images. Using the second contrastive loss function, based on the sample...The second pixel coordinate table of another sample image in this image pair is used to perform block-level comparison analysis on the two sample images contained in the sample image pair, and calculate the second contrast loss value between corresponding image blocks between the two sample images.

[0379] FIG17 shows a schematic diagram of the structure of a data processing device provided in another exemplary embodiment of this specification. As shown in FIG17, the device includes: a determination module 1112, a reversal module 1114, and an analysis and calculation module 1116. The determination module 1112 is used to determine the first pixel coordinate table of the first image and the second pixel coordinate table of the second image. The reversal module 1114 is used to perform a reversal operation on the first pixel coordinate table to obtain a reversed coordinate table. The determination module 1112 is also used to determine the pixel correspondence between the first image and the second image based on the reversed coordinate table and the second pixel coordinate table. The analysis and calculation module 1116 is used to perform block-level comparison analysis on the first image and the second image based on the pixel correspondence, and calculate the contrast loss value between the image block in the first image and the corresponding image block in the second image.

[0380] FIG18 shows a schematic diagram of the structure of a data processing device provided in another exemplary embodiment of this specification. As shown in Figure 18, the device includes: an acquisition module 1212, an inversion module 1214, and a mapping acquisition module 1216. The acquisition module 1212 is used to acquire a first pixel coordinate table of a first image and a second pixel coordinate table of a second image; wherein the first image and the second image are generated by editing a third image, and the first pixel coordinate table is generated by mapping the coordinates of pixels in the third image to the first image, and the second pixel coordinate table is generated by mapping the coordinates of pixels in the third image to the second image. The inversion module 1214 is used to perform an inversion operation on the first pixel coordinate table to obtain an inverted coordinate table. The mapping acquisition module 1216 is used to map the coordinates of pixels in the first image to the second image based on the second pixel coordinate table and the inverted coordinate table, obtaining the pixel correspondence between the first image and the second image (see page 32 / 35 of specification, CN 120997535 A). The pixel correspondence is used to participate in calculating the contrast loss value between image blocks in the first image and corresponding image blocks in the second image.

[0381] FIG19 shows a schematic diagram of a data processing apparatus provided in another exemplary embodiment of this specification. As shown in FIG19, the apparatus includes: an acquisition module 1312, a frame extraction module 1314, and an execution module 1316. The acquisition module 1312 is used to acquire a first video and a second video. The frame extraction module 1314 is used to perform frame extraction processing on the first video to obtain a data processing device.The first number of frames are extracted from the first video; the second video is processed by frame extraction to obtain a second number of frames extracted from the second video. The execution module 1316 is used to perform the following using a trained image infringement detection model: analyzing the pairwise similarity between the first number of frames and the second number of frames to obtain a similarity matrix; and determining whether there is an infringement relationship between the first video and the second video based on the similarity matrix. The loss function used by the image infringement detection model during training includes a composite contrast loss function; the composite contrast loss function is constructed based on a first contrast loss function and a second contrast loss function; using the first contrast loss function, based on the first pixel coordinate table of one sample image in the sample image pair, block-level contrast analysis can be performed on the two sample images contained in the sample image pair to calculate the first contrast loss value of the corresponding image block between the two sample images; using the second contrast loss function, based on the second pixel coordinate table of the other sample image in the sample image pair, block-level contrast analysis can be performed on the two sample images contained in the sample image pair to calculate the second contrast loss value of the corresponding image block between the two sample images.

[0382] It should be noted that the above-mentioned devices can implement the technical solutions described in the corresponding method embodiments. The specific implementation principles of the above-mentioned modules or units can be found in the relevant content of the corresponding method embodiments, and will not be described in detail here.

[0383] In addition, this specification also provides an electronic device. As shown in FIG20, the electronic device includes: a memory 1411 and a processor 1412.

[0384] The memory 1411 stores executable code. When the processor 1412 executes the executable code, it can implement the steps in the method embodiments described in FIG5, FIG8-13.

[0385] Further, as shown in FIG20, the electronic device also includes: a communication component 1413, a display 1414, a power supply component 1415, an audio component 1416, and other components. FIG20 only schematically shows some components and does not mean that the electronic device only includes the components shown in FIG20. In addition, the components in the dashed box in FIG20 are optional components, not mandatory components, and the specific components depend on the product form of the electronic device. The electronic device in this embodiment can be a terminal device such as a desktop computer, laptop computer, smartphone, or IoT device; it can also be a server-side device such as a conventional server, cloud server, or server array; or it can be an integrated device combining a terminal device and a server-side device. If the electronic device in this embodiment is a terminal device such as a desktop computer, laptop computer, or smartphone, it may include the components within the dashed box in Figure 20; if the electronic device in this embodiment is a server-side device such as a conventional server, cloud server, or server array, it may not include the components within the dashed box in Figure 20.

[0386] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), Magnetic Storage, Flash Memory, Disk or Optical Disk.

[0387] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In one exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel.

[0388] The aforementioned display includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen may be implemented as a touchscreen to receive input signals from a user. The Touch Panel includes one or more touch sensors to sense touch, swipe, and gestures on the Touch Panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.

[0389] The aforementioned power supply component provides power to various components of the device in which the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component is located.

[0390] The aforementioned audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device in which the audio component is located is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal may be further stored in a memory or transmitted via a communication component. In some embodiments, the audio component further includes a speaker for outputting the audio signal.

[0391] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium may be volatile, non-volatile, or a combination thereof, and may be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, digital video disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transfer medium

[0392] . Furthermore, embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, it causes the computer to perform the methods described in Figures 5 to 7.

[0393] Embodiments of this specification also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the methods described in Figures 5 to 7.

[0394] Those skilled in the art should recognize that, in one or more of the above examples, the functions described in the various embodiments disclosed in this specification can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0395] In some cases, the actions or steps recorded in the claims can be performed in a different order than in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0396] It should also be noted that, in this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply a relationship between these entities or operations.There exists any such actual relationship or order. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0397] The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of the multiple embodiments disclosed in this specification. It should be understood that the above descriptions are merely specific implementations of the multiple embodiments disclosed in this specification and are not intended to limit the protection scope of the multiple embodiments disclosed in this specification. Any modifications, equivalent substitutions, improvements, etc., made based on the technical solutions of the multiple embodiments disclosed in this specification should be included within the protection scope of the multiple embodiments disclosed in this specification. Instruction manual, pages 35 / 35, 39, CN 120997535 A, Figure 1, Figure 2; Instruction manual, Figure 1 / 10, page 40, CN 120997535 A, Figure 3, Figure 4; Instruction manual, Figure 2 / 10, page 41, CN 120997535 A, Figure 5, Figure 6a; Instruction manual, Figure 3 / 10, page 42, CN 120997535 A, Figure 6b; Instruction manual, Figure 4 / 10, page 43, CN 120997535 A, Figure 7, Figure 8; Instruction manual, Figure 5 / 10, page 44, CN 120997535 A, Figure 9, Figure 10; Instruction manual, Figure 6 / 10, page 45, CN 120997535 A, Figure 11, Figure 12; Instruction manual, Figure 7 / 10, page 46, CN 120997535 A, Figure 13, Figure 14, Figure 15; Instruction manual, Figure 8 / 10, page 47, CN 120997535 A, Figure 16, Figure 17. Figure 18 Figure 19 Appendix to the Specification 9 / 10 Page 48 CN 120997535 A Figure 20 Appendix to the Specification 10 / 10 Page 49 CN 120997535 A Abstract The embodiment of the invention provides a data processing method, a model training method, an image infringement detection model and device.According to one scheme, after a first image and a second image are obtained, a bidirectional block-level contrastive analysis is executed on the first image and the second image. The bidirectional block-level contrastive analysis includes: by taking the first image as a query image and the second image as a reference image, based on a pixel coordinate table of the query image, calculating a comparison loss value between image blocks in the query image and image blocks in the corresponding reference image; and by taking the second image as a query image and the first image as a reference image, calculating a comparison loss value between the image blocks in the query image and the image blocks in the corresponding reference image based on the pixel coordinate table of the query image. The comparison loss value is used for optimizing parameters of an image infringement detection model, and the image infringement detection model can be used for detecting whether the first image and thesecond image have an infringement relationship or not. The pixel coordinate table is used for recording position information of pixels in the query image.

Claims

1. A data processing method, characterized by, The method comprises: acquiring a first image and a second image; performing bidirectional block-level contrast analysis on the first image and the second image, respectively taking the first image as a query image and the second image as a reference image, and taking the second image as a query image and the first image as a reference image, and calculating a contrast loss value between an image block in the query image and an image block in the corresponding reference image based on a pixel coordinate table of the query image; wherein the pixel coordinate table is used to record the position information of the pixels in the query image; and the contrast loss value is used to optimize the parameters of an image infringement detection model, and the image infringement detection model can detect whether there is an infringement relationship between the first image and the second image.

2. The method of claim 1, wherein, The target query image block is an image block in the query image, the reference image has at least one candidate reference image block corresponding to the target query image block, and the target image block and the candidate reference image block have a pixel correspondence relationship; The target candidate reference image block is one of the at least one candidate reference image block, and then: based on the pixel coordinate table of the query image, the contrast loss value between the target query image block and the target candidate reference image block is calculated, which comprises: based on the pixel coordinate table, determining the prior distribution probability of the target candidate reference image block relative to the target query image block; determining the identification probability of correctly identifying the target candidate reference image block from the noise image block set according to the target query image block and the related noise image block set; wherein the noise image block set is constructed based on all image blocks in the reference image; according to the prior distribution probability and the identification probability, the contrast loss value of the target query image block and the target candidate reference image block is calculated.

3. The method of claim 2, wherein, Based on the pixel coordinate table, the prior distribution probability of the target candidate reference image block relative to the target query image block is determined, which comprises: based on the pixel coordinate table, determining the number of pixels in the target query image block that map to the pixels inside the target candidate reference image block; calculate the ratio of the number of pixels to the total number of pixels in the target query image block; according to the preset confidence adjustment parameter, the ratio is subjected to power operation to obtain a power operation result; the power operation result is normalized to obtain the prior distribution probability.

4. The method of claim 2, wherein, Based on the prior probability and the identification probability, the contrast loss value of the target query image block and the target candidate reference image block is calculated, which comprises: determining the probability difference measure between the prior distribution probability and the identification probability; based on the probability difference measure, the contrast loss value is determined.

5. The method of claim 2, wherein, Further comprising: performing an inverse operation on the pixel coordinate table to obtain an inverse coordinate table; based on the inverse coordinate table, the coordinate mapping of each pixel in the target query image block to the corresponding coordinate position in the reference image is determined to obtain a mapping result; according to the mapping result, the at least one candidate reference image block is determined from the reference image.

6. The method of claim 2, wherein, Further comprising: determining a query region in the query image and a reference region in the reference image; The query region and the reference region are a candidate region pair in an infringement relationship; According to the set image block size, the query region is divided to obtain a plurality of query image blocks; According to the set image block size, the reference region is divided to obtain a plurality of reference image blocks; The target query image block is one of the plurality of query image blocks, and the at least one candidate reference image block is determined from the plurality of reference image blocks.

7. The method according to any one of claims 1 to 6, characterized in that, Also includes: According to the first contrast loss value and the second contrast loss value, the average contrast loss value of the corresponding image blocks between the first image and the second image is determined; Wherein, the average contrast loss value is used to optimize the parameters of the image infringement detection model; the first contrast loss value is calculated in the case of taking the first image as the query image and the second image as the reference image, and the second contrast loss value is calculated in the case of taking the second image as the query image and the first image as the reference image.

8. A model training method, comprising: Including: Obtain a training sample set, which contains a plurality of sample image pairs; Set the loss function used when training the contrast learning model; Based on the plurality of sample image pairs and the loss function, the contrast learning model is trained to obtain an image infringement detection model; Wherein, the loss function includes a composite contrast loss function, and the composite contrast loss function is constructed based on a first contrast loss function and a second contrast loss function; using the first contrast loss function, the first pixel coordinate table of one of the sample image pairs can be used to perform block-level contrast analysis on the two sample images contained in the sample image pair, and calculate the first contrast loss value between the corresponding image blocks of the two sample images. Using the second contrast loss function, the second pixel coordinate table of the other sample image in the sample image pair can be used to perform block-level contrast analysis on the two sample images contained in the sample image pair, and calculate the second contrast loss value between the corresponding image blocks of the two sample images.

9. An image infringement detection model, characterized in that, Including: A plurality of network layers; The plurality of network layers include an input layer, an encoding layer, and a loss layer; wherein The input layer is used to receive the input first image and the second image The encoding layer has two encoding modules, one of which is used to encode the first image block in the first image to output the image block feature of the first image block, and the other is used to encode the second image block in the second image to output the image block feature of the second image block; The loss layer is provided with a loss function, and the loss function comprises a composite contrast loss function; the composite contrast loss function is constructed based on a first contrast loss function and a second contrast loss function; the first contrast loss function is used for performing block-level contrast analysis on the first image and the second image based on a first pixel coordinate table of the first image and image block features output by the encoding layer, and calculating a first contrast loss value of corresponding image blocks between the first image and the second image; the second contrast loss function is used for performing block-level contrast analysis on the second image and the first image based on a second pixel coordinate table of the second image and the image block features output by the encoding layer, and calculating a second contrast loss value of corresponding image blocks between the second image and the first image; The first contrast loss value and the second contrast loss value are used for optimizing parameters of the image infringement detection model, and the image infringement detection model can be used to detect whether there is an infringement relationship between the first image and the second image.

10. An electronic device, comprising: A device comprising a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method of any one of claims 1-8.

Images