Data processing method, model training method, image infringement detection model and equipment

Abstract

The embodiment of the invention provides a data processing method, a model training method, an image infringement detection model and equipment. According to one scheme, after a first image and a second image are obtained, bidirectional block-level contrastive analysis is executed on the first image and the second image, and the bidirectional block-level contrastive analysis comprises the steps that the first image serves as a query image, the second image serves as a reference image, and the reference image is obtained based on a pixel coordinate table of the query image; calculating a comparison loss value between the image blocks in the query image and the image blocks in the corresponding reference image; and by taking the second image as a query image and the first image as a reference image, calculating a comparison loss value between the image blocks in the query image and the image blocks in the corresponding reference image based on the pixel coordinate table of the query image. The comparison loss value is used for optimizing parameters of an image infringement detection model, and the image infringement detection model can be used for detecting whether the first image and the second image have an infringement relationship or not. The pixel coordinate table is used for recording position information of pixels in the query image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a data processing method, a model training method, an image infringement detection model and equipment. BACKGROUND

[0002] With the rapid popularization of digital media, the dissemination of image and video content on the network becomes extremely convenient, but this also leads to an increase in image and video content infringement, editing and tampering. In the fields of multimedia anti-piracy and image / video retrieval, it is particularly important to accurately identify these manipulated content. For this purpose, image infringement detection (ICD) technology is applied. In recent years, although with the progress of deep learning methods, especially self-supervised learning methods, image infringement detection technology has made remarkable achievements, and can meet the needs of practical applications to a certain extent, but in the face of complex image editing conversion, the performance is poor and the detection accuracy is low.

[0003] Therefore, it is necessary to provide an infringement detection technical scheme that can better face the situation of complex image editing conversion. SUMMARY

[0004] The embodiments in the specification provide a data processing method, a model training method, an image infringement detection model, a system, equipment, a storage medium and a program product, to improve the robustness and performance of image infringement detection, so as to better cope with the infringement detection needs in the case of increasingly complex image editing. Among them,

[0005] In a first embodiment, the specification provides a data processing method, which comprises:

[0006] obtaining a first image and a second image;

[0007] performing bidirectional block-level contrast analysis on the first image and the second image, respectively taking the first image as a query image and the second image as a reference image, and taking the second image as a query image and the first image as a reference image, and based on a pixel coordinate table of the query image, calculating a contrast loss value between image blocks in the query image and corresponding image blocks in the reference image;

[0008] The pixel coordinate table is used to record the position information of the pixels in the query image, and the contrast loss value is used to optimize the parameters of the image infringement detection model, and the image infringement detection model can be used to detect whether there is an infringement relationship between the first image and the second image.

[0009] In a second embodiment, the specification also provides a data processing method, which comprises:

[0010] obtaining a first image and a second image;

[0011] determining a first pixel coordinate table of the first image; the first pixel coordinate table is used to record the position information of the pixels in the first image;

[0012] performing block-level comparative analysis on the first image and the second image based on the first pixel coordinate table, and calculating a first comparative loss value between image blocks in the first image and corresponding image blocks in the second image;

[0013] The first comparative loss value is used to optimize the parameters of an image infringement detection model, and the image infringement detection model can be used to detect whether there is an infringement relationship between the first image and the second image.

[0014] In a third embodiment, the present specification also provides a data processing method, which comprises:

[0015] obtaining a first image and a second image;

[0016] inputting the first image and the second image into an image infringement detection model, and executing the image infringement detection model to output a detection result;

[0017] The detection result contains a heat map reflecting the similar areas between the first image and the second image, and the heat map is generated according to the similarity information generated in the process of performing bidirectional block-level comparative analysis on the first image and the second image. The bidirectional block-level comparative analysis comprises: taking the first image as a query image and the second image as a reference image, and taking the second image as a query image and the first image as a reference image, respectively, calculating the comparative loss value between the image blocks in the query image and the corresponding image blocks in the reference image based on the pixel coordinate table of the query image; the similarity information includes the similarity between the image blocks in the query image and the corresponding image blocks in the reference image obtained in the process of calculating the comparative loss value.

[0018] In a fourth embodiment, the present specification also provides a data processing method, which comprises:

[0019] obtaining a first image and a second image;

[0020] inputting the first image and the second image into an image infringement detection model, and executing the image infringement detection model to output a detection result;

[0021] The loss function of the image infringement detection model comprises a composite contrast loss function, and the composite contrast loss function is constructed based on a first contrast loss function and a second contrast loss function; the first contrast loss function is used to perform block-level contrast analysis on the first image and the second image based on a first pixel coordinate table of the first image, and a first contrast loss value between image blocks in the first image and corresponding image blocks in the second image is calculated; the second contrast loss function is used to perform block-level contrast analysis on the second image and the first image based on a second pixel coordinate table of the second image, and a second contrast loss value between image blocks in the second image and corresponding image blocks in the first image is calculated.

[0022] In a fifth embodiment, the specification provides a model training method, which comprises:

[0023] obtaining a training sample set comprising a plurality of sample image pairs;

[0024] setting a loss function used in training of a contrast learning model;

[0025] training the contrast learning model based on the plurality of sample image pairs and the loss function to obtain an image infringement detection model;

[0026] The loss function comprises a composite contrast loss function, and the composite contrast loss function is constructed based on a first contrast loss function and a second contrast loss function; the first contrast loss function is used to perform block-level contrast analysis on two sample images contained in the sample image pair based on a first pixel coordinate table of one sample image in the sample image pair, and a first contrast loss value of corresponding image blocks between the two sample images is calculated; the second contrast loss function is used to perform block-level contrast analysis on the two sample images contained in the sample image pair based on a second pixel coordinate table of the other sample image in the sample image pair, and a second contrast loss value of corresponding image blocks between the two sample images is calculated.

[0027] In a sixth embodiment, the specification provides an image infringement detection model, which comprises a plurality of network layers; the plurality of network layers comprise an input layer, an encoding layer, and a loss layer; wherein,

[0028] The input layer is configured to receive an input first image and a second image.

[0029] The encoding layer comprises two encoding modules, one of which is configured to perform encoding processing on a first image block in the first image to output image block features of the first image block, and the other of which is configured to perform encoding processing on a second image block in the second image to output image block features of the second image block.

[0030] The loss layer is provided with a loss function, and the loss function comprises a composite contrast loss function; the composite contrast loss function is constructed based on a first contrast loss function and a second contrast loss function; the first contrast loss function is used for performing block-level contrast analysis on the first image and the second image based on first pixel coordinates of the first image and image block features output by the encoding layer, and calculating a first contrast loss value of corresponding image blocks between the first image and the second image; the second contrast loss function is used for performing block-level contrast analysis on the second image and the first image based on second pixel coordinates of the second image and the image block features output by the encoding layer, and calculating a second contrast loss value of corresponding image blocks between the second image and the first image;

[0031] The first contrast loss value and the second contrast loss value are used for optimizing parameters of the image infringement detection model, and the image infringement detection model can be used to detect whether there is an infringement relationship between the first image and the second image.

[0032] In a seventh embodiment, the specification provides a data processing method, which comprises:

[0033] determining a first pixel coordinate table of a first image and a second pixel coordinate table of a second image;

[0034] performing an inversion operation on the first pixel coordinate table to obtain an inverse coordinate table;

[0035] determining a pixel correspondence relationship between the first image and the second image based on the inverse coordinate table and the second pixel coordinate table;

[0036] performing block-level contrast analysis on the first image and the second image based on the pixel correspondence relationship, and calculating a contrast loss value between image blocks in the first image and corresponding image blocks in the second image.

[0037] In an eighth embodiment, the specification provides a data processing method, which comprises:

[0038] obtaining a first pixel coordinate table of a first image and a second pixel coordinate table of a second image; wherein the first image and the second image are generated by editing a third image, and the first pixel coordinate table is generated by mapping coordinates of pixels in the third image to the first image, and the second pixel coordinate table is generated by mapping coordinates of pixels in the third image to the second image;

[0039] performing an inversion operation on the first pixel coordinate table to obtain an inverse coordinate table;

[0040] mapping, based on the second pixel coordinate table and the reverse coordinate table, a pixel in the first image to a pixel in a second image, to obtain a pixel correspondence relationship between the first image and the second image;

[0041] The pixel correspondence relationship is used to participate in calculation of a contrast loss value between an image block in the first image and a corresponding image block in the second image.

[0042] In a ninth embodiment, the present specification provides a data processing method, which comprises:

[0043] obtaining a first video and a second video;

[0044] performing frame extraction processing on the first video to obtain a first number of frame images extracted from the first video;

[0045] performing frame extraction processing on the second video to obtain a second number of frame images extracted from the second video;

[0046] using the trained image infringement detection model to perform: analyzing the similarity of each pair of images between the first number of frame images and the second number of frame images to obtain a similarity matrix; and determining whether there is an infringement relationship between the first video and the second video based on the similarity matrix.

[0047] The loss function used by the image infringement detection model in the training process comprises a composite contrast loss function; the composite contrast loss function is constructed based on a first contrast loss function and a second contrast loss function; the first contrast loss function is used to perform block-level contrast analysis on two sample images contained in a sample image pair based on a first pixel coordinate table of one sample image in the sample image pair, and calculate a first contrast loss value of corresponding image blocks between the two sample images; the second contrast loss function is used to perform block-level contrast analysis on the two sample images contained in the sample image pair based on a second pixel coordinate table of the other sample image in the sample image pair, and calculate a second contrast loss value of corresponding image blocks between the two sample images.

[0048] In a tenth embodiment, the present specification provides a data processing system, which comprises:

[0049] a server, which is deployed with an image infringement detection model, and is configured to: obtain a first image and a second image; input the first image and the second image into the image infringement detection model, and execute the image infringement detection model to output a detection result;

[0050] a client, configured to display the detection result;

[0051] The detection result includes a heat map reflecting a similar region between the first image and the second image, and the heat map is generated according to a contrast loss value between an image block in the first image and an image block in the second image calculated by performing block-level contrast analysis on the first image and the second image.

[0052] In a eleventh embodiment, the present specification provides a data processing apparatus, comprising:

[0053] The acquisition module is configured to acquire a first image and a second image.

[0054] The contrast analysis module is configured to perform bidirectional block-level contrast analysis on the first image and the second image, respectively taking the first image as a query image and the second image as a reference image, and taking the second image as a query image and the first image as a reference image, and calculating a contrast loss value between an image block in the query image and an image block in the corresponding reference image based on a pixel coordinate table of the query image.

[0055] The pixel coordinate table is used to record the position information of the pixels in the query image, and the contrast loss value is used to optimize the parameters of the image infringement detection model, and the image infringement detection model can be used to detect whether there is an infringement relationship between the first image and the second image.

[0056] In a twelfth embodiment, the present specification further provides a data processing apparatus, comprising:

[0057] The determination module is configured to determine a first pixel coordinate table of the first image, and the pixel coordinate table is used to record the position information of the pixels in the first image.

[0058] The contrast analysis module is configured to perform block-level contrast analysis on the first image and the second image based on the first pixel coordinate table, and calculate a first contrast loss value between an image block in the first image and an image block in the corresponding second image.

[0059] The first contrast loss value is used to optimize the parameters of the image infringement detection model, and the image infringement detection model can be used to detect whether there is an infringement relationship between the first image and the second image.

[0060] In a thirteenth embodiment, the present specification further provides a data processing apparatus, comprising:

[0061] The acquisition module is configured to acquire a first image and a second image.

[0062] The execution module is configured to input the first image and the second image into an image infringement detection model, and execute the image infringement detection model to output a detection result.

[0063] a display module configured to display the detection result;

[0064] The detection result comprises a heat map reflecting a similar region between the first image and the second image, and the heat map is generated according to similarity information generated in a bidirectional block-level comparison analysis process performed on the first image and the second image; the bidirectional block-level comparison analysis comprises: taking the first image as a query image and the second image as a reference image, and taking the second image as a query image and the first image as a reference image, respectively, and calculating a comparison loss value between an image block in the query image and an image block in the corresponding reference image based on a pixel coordinate table of the query image; the similarity information comprises the similarity between the image block in the query image and the image block in the corresponding reference image obtained in the process of calculating the comparison loss value.

[0065] In a fourteenth embodiment, the present specification also provides a data processing apparatus, which comprises:

[0066] an acquisition module configured to acquire a first image and a second image;

[0067] an execution module configured to input the first image and the second image into an image infringement detection model, and execute the image infringement detection model to output a detection result;

[0068] The loss function of the image infringement detection model comprises a composite comparison loss function, and the composite comparison loss function is constructed based on a first comparison loss function and a second comparison loss function; the first comparison loss function is used to perform block-level comparison analysis on the first image and the second image based on a first pixel coordinate table of the first image, and calculate a first comparison loss value between an image block in the first image and a corresponding image block in the second image; the second comparison loss function is used to perform block-level comparison analysis on the second image and the first image based on a second pixel coordinate table of the second image, and calculate a second comparison loss value between an image block in the second image and a corresponding image block in the first image.

[0069] In a fifteenth embodiment, the present specification also provides a data processing apparatus, which comprises:

[0070] an acquisition module configured to acquire a training sample set, wherein the training sample set comprises a plurality of sample image pairs;

[0071] a setting module configured to set a loss function used in training of a contrast learning model;

[0072] a training module configured to perform model training on the contrast learning model based on the plurality of sample image pairs and the loss function, and obtain an image infringement detection model;

[0073] The loss function comprises a composite contrast loss function constructed based on a first contrast loss function and a second contrast loss function; the first contrast loss function is used to perform block-level contrast analysis on two sample images contained in the sample image pair based on a first pixel coordinate table of one sample image in the sample image pair, and calculate a first contrast loss value of corresponding image blocks between the two sample images; the second contrast loss function is used to perform block-level contrast analysis on the two sample images contained in the sample image pair based on a second pixel coordinate table of the other sample image in the sample image pair, and calculate a second contrast loss value of corresponding image blocks between the two sample images.

[0074] In a sixteenth embodiment, the present specification also provides a data processing apparatus, which comprises:

[0075] A determination module is configured to determine a first pixel coordinate table of a first image and a second pixel coordinate table of a second image.

[0076] An inversion module is configured to perform an inversion operation on the first pixel coordinate table to obtain an inverse coordinate table.

[0077] The determination module is further configured to determine a pixel correspondence relationship between the first image and the second image based on the inverse coordinate table and the second pixel coordinate table.

[0078] An analysis and calculation module is configured to perform block-level contrast analysis on the first image and the second image based on the pixel correspondence relationship, and calculate a contrast loss value between image blocks in the first image and corresponding image blocks in the second image.

[0079] In a seventeenth embodiment, the present specification also provides a data processing apparatus, which comprises:

[0080] An acquisition module is configured to acquire a first pixel coordinate table of a first image and a second pixel coordinate table of a second image; wherein the first image and the second image are generated by performing an editing operation on a third image, and the first pixel coordinate table is generated by mapping coordinates of pixels in the third image to the first image, and the second pixel coordinate table is generated by mapping coordinates of pixels in the third image to the second image.

[0081] An inversion module is configured to perform an inversion operation on the first pixel coordinate table to obtain an inverse coordinate table.

[0082] The mapping obtaining module is configured to map the coordinates of the pixels in the first image to the second image based on the second pixel coordinate table and the reverse coordinate table, and obtain a pixel correspondence relationship between the first image and the second image.

[0083] The pixel correspondence relationship is used to participate in calculation of a contrast loss value between image blocks in the first image and corresponding image blocks in the second image.

[0084] In an eighteenth embodiment, the present specification also provides a data processing apparatus, which comprises:

[0085] The obtaining module is configured to obtain a first video and a second video.

[0086] The frame extracting module is configured to perform frame extraction processing on the first video to obtain a first number of frame images extracted from the first video, and perform frame extraction processing on the second video to obtain a second number of frame images extracted from the second video.

[0087] The executing module is configured to use the trained image infringement detection model to perform: analyzing the similarity of each pair of images between the first number of frame images and the second number of frame images to obtain a similarity matrix; and determining whether there is an infringement relationship between the first video and the second video based on the similarity matrix.

[0088] The loss function used in the training process of the image infringement detection model comprises a composite contrast loss function; the composite contrast loss function is constructed based on a first contrast loss function and a second contrast loss function; the first contrast loss function is used to perform block-level contrast analysis on two sample images contained in a sample image pair based on a first pixel coordinate table of one sample image in the sample image pair, and calculate a first contrast loss value of corresponding image blocks between the two sample images; the second contrast loss function is used to perform block-level contrast analysis on the two sample images contained in the sample image pair based on a second pixel coordinate table of the other sample image in the sample image pair, and calculate a second contrast loss value of corresponding image blocks between the two sample images.

[0089] In a nineteenth embodiment, the present specification provides an electronic device comprising a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the methods provided in the present specification.

[0090] In a twentieth embodiment, the present specification provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, the computer is caused to execute the methods provided in the present specification.

[0091] The twenty-first embodiment provides a computer program product comprising computer programs / instructions which, when executed by a processor, implement the methods provided in the specification.

[0092] The scheme provided by the above embodiments of the specification is that after the first image and the second image are obtained, the first image and the second image are subjected to bidirectional block-level contrast analysis. In the bidirectional block-level contrast analysis, the first image is taken as the query image and the second image is taken as the reference image, and the second image is taken as the query image and the first image is taken as the reference image, the pixel coordinate table based on the query image is used to perform block-level contrast analysis on the query image and the reference image, so as to calculate the contrast loss value between the image blocks in the query image and the corresponding image blocks in the reference image. In the above, the pixel coordinate table based on the query image is used to perform block-level contrast analysis on the query image and the reference image, which can accurately obtain the pixel-level geometric correspondence between different images, capture more fine spatial details in the image, avoid information loss, and provide a high-quality basis for subsequent calculation of the contrast loss value between the corresponding image blocks of different images. In addition, the contrast analysis is directly performed at the image block level, which can learn the contrast relationship between more detailed local regions than the whole image. As can be seen from the above, the present scheme can provide pixel-level more fine-grained analysis when performing contrast analysis on different images, which can ensure effective processing of complex image editing scenes and improve the matching accuracy between the corresponding image blocks of different images. The above contrast analysis is performed by using the corresponding image infringement detection model, and the result output by the image infringement detection model contains a heat map reflecting the similar regions between the first image and the second image. It can be seen that the present case also realizes the visualization of the infringement region, which can provide more intuitive infringement region positioning information for the user and can also improve the reliability of the model. In addition, the patrol function of the above image infringement detection model includes a composite contrast loss function, which is constructed based on the first contrast loss function and the second contrast loss function. The first contrast loss function is used to perform block-level contrast analysis on the first image and the second image based on the first pixel coordinate table of the first image, and calculate the first contrast loss between the image blocks in the first image and the corresponding image blocks in the second image. The second contrast loss function is used to perform block-level contrast analysis on the second image and the first image based on the second pixel coordinate table of the second image, and calculate the second contrast loss between the image blocks in the second image and the corresponding image blocks in the first image. By using the composite contrast loss function, the affinity between the image blocks can be established on the basis of the pixel overlap ratio, and the importance difference of different image blocks can also be considered, which can avoid the conflict gradient caused by false matching, which can greatly improve the accuracy of model learning and the ability to robustly correspond. Specifically, the model can learn an image representation that is highly robust to complex editing, thereby effectively improving the detection performance in the image infringement detection scene. Attached Figure Description

[0093] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0094] Figure 1 This specification provides an example of matching different images;

[0095] Figure 2 This is a schematic diagram illustrating the principle of pixel tracking for image infringement simulation using the pixel coordinate tracking module (PixTrace) as an exemplary embodiment of this specification.

[0096] Figure 3 This specification provides an exemplary embodiment of the diagram illustrating the calculation of similarity between corresponding image patches in different images using geometrically guided contrast loss (CopyNCE).

[0097] Figure 4 A schematic diagram of the network structure of the matcher provided in an exemplary embodiment of this specification;

[0098] Figure 5 A flowchart illustrating a data processing method provided for an exemplary embodiment of this specification;

[0099] Figure 6a and Figure 6b A schematic diagram of a heat map provided for an exemplary embodiment of this specification;

[0100] Figure 7 This specification provides a schematic diagram of the structure of a data processing system as an exemplary embodiment.

[0101] Figures 8 to 13 A flowchart illustrating the data processing method provided in the embodiments of this specification;

[0102] Figures 14 to 19 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this specification;

[0103] Figure 20 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this specification. Detailed Implementation

[0104] As mentioned before, with the popularity of digital media, the infringement, editing and tampering of image and video content have become increasingly common, and identifying these manipulated content is crucial for multiple fields, such as image / video retrieval and multimedia anti-piracy. To this end, image infringement detection (ICD) techniques have emerged to address this challenge. Currently, the mainstream method for image infringement detection is to adopt self-supervised learning (SSL), which has made significant progress, especially in contrastive learning. However, traditional self-supervised learning methods mainly focus on view-level learning (i.e., the similarity of the entire image), often ignoring the fine-grained correspondence at the region or patch level, which makes them perform poorly when dealing with complex editing transformations and difficult to effectively handle various advanced tampering. Specifically, this view-level self-supervised learning method constructs image pairs by applying various transformations to the entire image and conducts contrastive learning, which has the problems of lacking fine-grained correspondence learning, relying only on image-level labels, and being not robust to complex editing. The above lack of fine-grained correspondence learning refers to focusing on the similarity of the entire image without deeply learning the precise correspondence between regions or patches, which makes it ineffective in identifying complex or local editing (such as image cutout, local transformation). The above reliance on image-level labels refers to the model based on self-supervised learning needs to infer the infringement region from the entire image-level label, which is insufficient for precise positioning of infringement behavior. And the above not robust to complex editing refers to in the face of large-scale geometric transformation, content filling and other complex editing, the view-level features are slow to capture the subtle traces of manipulation.

[0105] To make up for the shortcomings of the above view-level self-supervised learning, some other methods have also been proposed, such as heuristic local visual self-supervised learning method, region feature extraction method based on ROI pooling (ROIPooling), method of using target detection model or heuristic rules to extract local regions, local feature matching (LMF) method, etc. The following will introduce and explain some of the above proposed methods respectively.

[0106] 1) Heuristic local visual self-supervised learning method,

[0107] In self-supervised learning, most local modeling methods are heuristic, and therefore are often referred to as heuristic local visual self-supervised learning methods. Among such methods, there are attempts to explore fine-grained correspondences through nearest neighbor (NN) matching strategies to improve the recognition ability of tampered images. For example, a feature-based nearest neighbor matching method (Feature-based Nearest Neighbor, FeatNN) retrieves the nearest neighbor in the feature space as the positive sample. For example, a location-based nearest neighbor matching method (Location-based Nearest Neighbor, LocNN) compares the corresponding positions of image block center coordinates on the original image to identify positive samples. For example, a method combining FeatNN and LoaNN attempts to take advantage of both FeatNN and LoaNN. Although these heuristic local visual self-supervised learning methods based on nearest neighbor (NN) matching have improved detection results to some extent, they still have inherent inaccuracies that can lead to false positive matches, partial matches, ignore important differences, supervision noise, and poor performance on complex editing effects. For example: FeatNN is susceptible to interference from semantically similar but non-infringing regions (different objects but similar semantics), and LocNN cannot determine whether there is actual overlap between image blocks (image blocks in one image have no corresponding part in another image, but LocNN can still give false matches). Both of these cases can lead to false matches. In addition, FeatNN and LocNN often cannot determine the exact number of positive sample image blocks, which can lead to partial matches, and completely ignore the importance differences of different positive sample image blocks. These problems are crucial to image infringement detection and directly affect detection efficiency.

[0108] Figure 1 The matching cases in the heuristic local visual self-supervised learning method are exemplarily shown in the following table. Figure 1 In the table, Match A: Although there is a nearest neighbor block on the query image (which is infringing Figure 1 ), when LocNN is used for matching, no overlap error matching will occur because there is no corresponding image in the original block in the query image; Match B: neither LocNN nor FeatNN can retrieve all positive sample image blocks, only partial matching can be achieved; Match C: when FeatNN is used for matching, it may be misled by semantically similar objects, resulting in false matching.

[0109] Wherein, the false positive match, also known as inaccurate match, refers to the error of identifying irrelevant regions as infringement parts. Partial match refers to the failure to identify all real infringement regions (also known as infringement image blocks), resulting in insufficient recall rate (failure to retrieve all positive samples). Ignoring the importance difference refers to that all matched image blocks (positive samples) are usually treated equally, and the difference in the degree of overlap or contribution between different image blocks is ignored. And the above-mentioned supervised noise, that is, the introduction of noise supervision, refers to: due to inaccurate matching, a large number of conflicting gradient signals (i.e. a large number of false supervision signals) will be generated in the training process, which will have a serious negative impact on the convergence and performance of the model. Poor effect on complex editing: especially LocNN, when facing complex editing such as rotation, stretching, and matting, which leads to deformation or irregular infringement regions of image blocks, its detection effect will be greatly reduced.

[0110] 2) Region feature extraction method based on ROI pooling

[0111] In many detection or segmentation tasks, in order to extract fixed-length features from regions of arbitrary size, ROI pooling will be used. This scheme sacrifices fine-grained spatial details because: ROI pooling is essentially a downsampling operation that loses fine spatial information within the original region, resulting in features that are too rough and unsuitable for image infringement detection (ICD) tasks that require high-precision spatial correspondence.

[0112] 3) Method of using object detection model or heuristic rule to extract local region

[0113] Such methods, such as using an object detection model to crop suspicious infringement regions or applying heuristic rules to extract multiple local regions for integration, have the problems of detection error propagation and high computational overhead. Among them, detection error propagation: for example, if the object detection model fails to accurately identify the infringement region, its error will be propagated downstream, affecting the final detection performance. High computational overhead: multi-stage pipeline operations (such as first detection and then comparison) will increase the overall computational complexity and time overhead.

[0114] 4) Local Feature Matching (LMF) method

[0115] Such methods are usually used to estimate camera pose, depth map, or keypoint correspondence established from different views of the same object, and have problems such as inapplicability to complex editing, data dependence and error, and difficulty in direct migration. Among them, the reasons for inapplicability to complex editing include: complex editing involved in the image infringement detection (ICD) task, such as image matting, content synthesis, etc., which cannot be described by a simple homography matrix, while LMF mainly handles set transformation, but it is powerless to content-level editing. Data dependence and error: when auxiliary information such as depth map is needed, the collection cost is high and there may be estimation error. Difficult to migrate directly: the keypoint-oriented loss function is difficult to directly apply to the image infringement detection (ICD) task, because ICD focuses on the controversial infringement of regions or content, rather than the matching of specific key points.

[0116] From the above analysis, although some existing methods have certain effect in image infringement detection, they have obvious limitations in solving the core problems of inaccurate fine-grained correspondence in negative infringement scenarios, noisy supervision signal, and low computational efficiency. Therefore, there is an urgent need for a method that can establish reliable pixel-level correspondence and effectively integrate this geometric perception into a self-supervised learning framework to significantly improve the robustness and performance of image infringement detection (ICD) and to cope with increasingly complex image editing challenges.

[0117] To this end, a new image infringement detection technical solution is provided in this specification, which aims to use the inherent geometric "traceability" of image content to solve the problems of insufficient fine-grained correspondence learning and supervision operation in image infringement detection. Specifically, the main basic idea is to provide accurate pixel-level supervision and effectively handle complex image editing by introducing a pixel coordinate tracking module (PixTrace) and a geometry-guided contrast loss (CopyNCE) to overcome the problems of false matching and noise of traditional methods.

[0118] In order for those skilled in the art to better understand the technical solutions in the specification, the technical solutions in the embodiments of the specification will be described clearly and completely below in conjunction with the drawings in the embodiments. Obviously, the described embodiments are only a part of the embodiments of the specification, not all. Based on the embodiments in the specification, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the specification.

[0119] It should be noted that, for the convenience of description, only parts related to the technical solutions are shown in the drawings. The embodiments in the specification and the features in the embodiments can be combined with each other without conflict. In addition, the words "first", "second", "third" and the like in the embodiments of the specification are only for information differentiation, and do not have any limiting effect.

[0120] Before introducing the embodiments in this specification, some technical terms appearing in this specification are briefly introduced.

[0121] Image Copyright Detection (ICD): Aims to identify manipulated content between pairs of images.

[0122] Self-Supervised Learning (SSL): A machine learning paradigm that trains models by generating supervisory signals from the data itself.

[0123] Pixel Coordinate Tracking Module (PixTrace): Used to maintain an explicit spatial mapping of pixels after editing transformations.

[0124] Geometric Guided Contrastive Loss Function (CopyNCE): Used to regularize image patch affinity with the mapping verified by PixTrace.

[0125] Patch Affinity: The degree of similarity or correlation between different regions or patches in an image.

[0126] Mean Average Precision (μAP): A commonly used evaluation metric in image copyright detection and retrieval tasks, used to measure the ability of a model to identify image similarity.

[0127] Recall at 90% Precision (RP90): A commonly used evaluation metric in image copyright detection, used to represent the proportion of true copyright samples that the model can recall when the precision reaches 90%.

[0128] Descriptor: A form of model used to extract feature representations of images, which are used for image copyright identification through feature similarity.

[0129] Matcher: A form of model that directly takes pairs of images as input and performs a binary classification task (judges whether it is a copyright image pair).

[0130] Vision Transformer (ViT): A Transformer architecture-based image processing model mainly used for computer vision tasks.

[0131] Information Noise-Contrastive Estimation (InfoNCE): A commonly used contrastive learning loss function.

[0132] Hard Negatives: In contrastive learning, refers to those examples that are similar to positive samples but are actually negative samples, or negative samples that are difficult to distinguish.

[0133] Geometric Traceability: refers to the ability to trace the original correspondence of pixels or regions after the image has undergone editing operations.

[0134] Query Image and Reference Image: In the field of image infringement detection, the query image usually refers to the image to be detected, i.e., the image that wants to check whether there is infringement. The reference image refers to the benchmark image for comparison. The benchmark image can be the original image or other images, such as a benchmark image derived from the original image through editing operations.

[0135] In order to facilitate the understanding of the technical solution, the pixel coordinate tracking module (PixTrace) and the geometric guided contrast loss (CopyNCE) introduced in the technical solution will be described in detail. At the same time, the related loss function composition is also described for the image infringement detection model (such as matcher or descriptor, etc.).

[0136] I. Pixel Coordinate Tracking Module (PixTrace)

[0137] The pixel coordinate tracking module can also be understood as a pixel tracking pipeline, which is used to model the coordinate mapping at the pixel level. In this pixel coordinate tracking module, a coordinate table T is defined. The coordinate table T is a dictionary-based data structure that contains a series of coordinate pairs in the corresponding image as keys and values.

[0138] In specific implementation, the above coordinate table T supports the following operations:

[0139] 1) Query operation, such as T[c] represents querying the corresponding value with coordinate c as the key;

[0140] 2) Update operation, such as F(T) represents updating all values in T using function F;

[0141] 3) Inversion operation (also known as inverse transformation), such as T -1 represents the inversion of key-value pairs, i.e., mapping the value back to the key.

[0142] Based on the above coordinate table T, all edited pixels can be tracked, and thus the coordinates of the pixels in the infringing image can be mapped to the original image through the coordinate table T.

[0143] For example, see Figure 2As shown, by modifying the original image I o Taking the simulation of infringement by performing various image editing operations as an example, the steps performed in constructing the corresponding infringing image and the corresponding coordinate table T include:

[0144] Step 1: Initialization

[0145] Select the original image I o (e.g., an initial image that has not undergone any transformation), and based on the original image I o The coordinate table T is initialized using the coordinates of the pixels in the middle. After initialization, the coordinate table T = T0. o , where the original image I o The coordinates of each pixel in the image are mapped to itself, i.e., T(m,n) = (m,n). In other words, a coordinate table T corresponding to the original image is created. o , for recording the original image T o The coordinate position of the middle pixel changes without modifying the original image I. o Before any actual editing is performed, the coordinate position of each pixel remains unchanged by default, which means that for the original image I o Each pixel in the image still has its coordinates (m,n) mapped to its own position (m,n).

[0146] Step 2: Editing and Coordinate Update

[0147] In the original image I o A series of editing processes (such as affine transformation, perspective transformation, image matting, etc.) are performed to generate the first infringing image I. a During the process, each editing operation causes a change in pixel coordinates, and a corresponding function f can be used to update the coordinate table to obtain the first infringing image I. a Corresponding coordinate table T ao .

[0148] For example, suppose the original image I o If the coordinates of a certain pixel in the image are (m, n), then in the original image I... o During an affine transformation, the corresponding function f1 for coordinate transformation can be used to transform the original image I. o The coordinates of a certain pixel in the image are (m,n), which are mapped to the first infringing image I. a The coordinate position (i, j) in the middle, that is: f1(T0(m,n))=(i,j).

[0149] For simplicity, in this case, it refers to a specific image (such as the original image I). oAll editing operations performed on an image (I) and the corresponding coordinate transformation functions are sequentially combined to form a new function, which can be called a composite function F. For example, in the original image I... o The first infringing image I is generated by performing a series of editing operations. a At this time, the functions for coordinate transformation corresponding to this series of editing operations can be combined sequentially to form a composite function F. a The composite function Thus, through this composite function F a It can realize the coordinate table T o Update and generate the first infringing image I a Coordinate table T ao T ao =F a (T o ).

[0150] Among them, in the composite function F a In the middle, f1, f2, ..., f N It is a series of functions for coordinate transformations, each function performing a corresponding coordinate transformation for a specific image editing operation; and, circles symbolize... This represents the composition of functions, for example, in This represents a composite operation between the first function f1 and the second function f1, where the input to the first function f1 is the original image I. o Corresponding coordinate table T o The output of the first function f1 is used as the input of the second function f2.

[0151] Step 3, Reverse Tracking

[0152] By analyzing the first infringing image I a Corresponding coordinate table T ao By performing a reversal operation, the corresponding reversed coordinate table T can be obtained. oa .

[0153] in,

[0154] It should be noted here that: Figure 2 The reverse operation shown on the far right can be considered as F -1 In pixel-level discrete representation, and the reverse mapping can be directly passed through T -1 Obtainable without specific F -1 Among them, T, as shown, -1 (i,j)=(m,n) represents an image (such as the first infringing image I). a The coordinates (i,j) of a pixel in one image correspond to another image (such as the original image I). oThe coordinate position (m,n) in the image means that the coordinate position of a pixel in one image is (i,j), while the corresponding coordinate position in another image is (m,n).

[0155] Step 4: Image-to-Image Bridge Tracking

[0156] By using the reverse operation, if there is another image that is also in the original image I... o Generated by editing, for example Figure 2 The second infringing image I shown in the image b Also in the original image I o The second infringing image I was generated through editing. b Corresponding coordinate table T bo =F b (T o ), then the original image I o It can serve as a bridge to realize the first infringing image I a Second infringing image I b Pixel tracking between them.

[0157] For example, if you want to find the first infringing image I a Each pixel in the second infringing image I b The corresponding coordinates in the image are then used to determine the first infringing image I. a Corresponding coordinate table T ao First use the reverse operation The first infringing image I a Mapping the coordinates of each pixel back to the original image T o In the coordinate system, and then using the second infringing image I b The corresponding composite function F b Transfer the coordinates of these pixels from the original image T o The coordinate system is mapped to the second infringing image I b In the coordinate system. Therefore: the first infringing image I a The coordinates of each pixel in the image are mapped to the second infringing image I. b In the coordinate system, the corresponding generated mapping coordinate table Similarly, the second infringing image I b The coordinates of each pixel in the image are mapped to the first infringing image I. a In the coordinate system, the corresponding generated mapping coordinate table

[0158] From the above, through the pixel coordinate tracking module (PixTrace) provided above, the pixel correspondence relationship between the infringing image and the original image, or even between different images derived from the same original image, can be tracked. Specifically, the PixTrace module can determine and accurately obtain the combined correspondence relationship of the similarity level through an explicit pixel coordinate tracking mechanism. In addition, the PixTrace module is designed to track the coordinates of pixels after complex and sequential editing operations (including affine, perspective, image cutout, color jitter, etc.) by maintaining an iteratively updated coordinate table, so that it can process various nonlinear transformations and content-level tampering of images, thereby enabling the present case to exhibit stronger robustness when using the PixTrace module to implement image infringement detection in various actual scenarios. Complex image editing and tampering behaviors can be performed. Among them, through the pixel-level information provided by the PixTrace module during image infringement detection, high-quality basis can be provided for subsequent fine-grained learning.

[0159] II. Geometric guidance contrast loss (CopyNCE)

[0160] The purpose of introducing the geometric guidance contrast loss function (CopyNCE) in the specification is to use the accurate pixel-level correspondence information provided by the pixel coordinate tracking module (PixTrace) for any two images to regularize the affinity of image patches at the image patch level, thereby providing pixel-level guidance for image infringement detection and infringement region positioning.

[0161] The above geometric contrast loss function is derived on the basis of a contrast learning loss function, such as the information noise contrast estimation (InfoNCE) function. The following describes the derivation process by taking the derivation of the geometric guidance contrast loss function introduced in the specification on the basis of the InfoNCE function as an example.

[0162] The InfoNCE function aims to maximize the mutual information I(x; x p ) between the sample x and its positive sample x p , which satisfies the following constraint condition:

[0163] I(x; x p ) ≥ log |X| + E X [log p(x p |X, x)]

[0164] where E X [log p(x p |X, x)] represents the correct selection of the positive sample x pThe expected logarithmic probability; |X| represents the cardinality of the noise set X, and correspondingly, log|X| is a constant term. Furthermore, the noise set X is a set of samples, typically containing one positive sample (correlated with x) and the remaining samples are negative samples (unrelated to x). p Let X be a positive sample contained in X (i.e., a positive sample related to X).

[0165] To maximize I(x; x p InfoNCE is typically formulated as given by the following formula (1):

[0166]

[0167] In the above formula (1), g θ The function is used to calculate the similarity measure between two samples, x. j Let x represent the j-th sample (which could be either negative or positive) in the noise set X. Let the given sample x be denoted as xj. h and x p or x j denoted as x v In general, g θ (x h ,x v )=exp(cos(z h ,z v ) / τ), where z h ,z v They represent x respectively h ,x v Features, cos(z) h ,z v ) represents z h and z v The cosine similarity is given by τ, where τ is a temperature parameter (used to control the smoothness of the similarity distribution).

[0168] As above, formula (1) can be understood as: calculate the similarity score of positive sample pairs, normalize it to the similarity score relative to all sample pairs, and then take the negative logarithm as the loss value.

[0169] In the context of image infringement detection (ICD), assuming the query image q is an infringing edited version of the reference image r, their corresponding manipulated image regions (i.e., the corresponding infringing and infringed image regions) represent R respectively. q and R r In the following content, for ease of distinction, R will be used. q A query region is called a query region in the query image q (a local region of interest in the current query image q), and R is...r one of the reference regions in the reference image r (for matching or comparing with the query region).

[0170] To enable the corresponding infringement detection model (such as a matcher or a descriptor) to locate the infringement forgery region of the query image q relative to the reference image, the mutual information I(R q ; R r ) is proposed as an auxiliary learning goal. Similar to InfoNCE, a set of contrastive noise wherein, represents the interference region (also referred to as the negative sample region). Thus, in combination with the aforementioned formula (1), the geometric guidance contrastive loss function (CopyNCE) can be derived as the following formula (2):

[0171]

[0172] wherein, f θ represents a visual encoding model with parameters θ, used to embed regions or image blocks in the corresponding image. For example, f θ may be a visual encoding model based on a visual architecture such as ViT or CNN (convolutional neural network).

[0173] In mainstream visual architectures (such as ViT or CNN), the common method for extracting features from any region relies on ROI pooling, but ROI pooling essentially sacrifices fine-grained spatial details, resulting in features that are too rough for the image infringement detection (ICD) task. To overcome this limitation, the present solution is to divide larger image regions (containing the query region R q and the reference region R r ) into a number of minimum unit image blocks, respectively, in order to apply CopyNCE at a finer granularity level. Specifically, the query region R q and the reference region R r are divided into and respectively, wherein, is the i-th query image block in the query region R q , is the j-th reference image block in the reference region R r , and in specific implementation, or is the minimum unit image block (also referred to as the minimum processing image block) defined by the corresponding visual architecture (for example, in ViT, the size of the minimum unit image block is usually 16x16 pixels); and the symbol ∪ i or ∪ j represents the union operation. That is, the larger image regions Rq and R r Perform decomposition. If querying image patches Corresponding to the reference image block Indicates the reference region R r The i-th reference image block in the query image block. For positive sample image patches, the contrast noise set at the image patch level can be reformulated as: Accordingly, the formula for CopyNCE can be converted into the following formula (3):

[0174]

[0175] In this formula (3), E represents the mathematical expectation operation. For ease of description of g... θ The decomposition will Noted as R i and will or R x Noted as R j Then g θ It can be decomposed into: g θ (R i ,R j )=exp(cos(z i ,z j ) / τ). Where, z i =f θ (R i ) represents the visual encoding model f θ For image block R i The image features (tokens) generated by encoding, R i For querying region R q The i-th query image block And, z j =f θ (R j ) represents the visual model f θ For image block R j The image features generated by encoding, R j For reference region R r The i-th image patch Or it could be the contrast noise set R X An image block R x .

[0176] Here, feature extraction and loss calculation are performed directly at the smallest unit image patch (e.g., 16×16 pixels) level of the visual encoder (e.g., ViT). By operating on the smallest unit image patch, information loss that may be caused by ROI pooling can be avoided, thus enabling better capture and utilization of fine spatial details in the image. This is crucial for image infringement detection (ICD) tasks that require high-precision localization of infringement areas.

[0177] It is worth noting here that: for each queried image block, such as It is possible to find multiple candidate positive sample image patches (as reference image patches) in the reference image r. In specific implementations, this can be based on querying image patches. When the coordinates of each pixel in the query image are mapped to the corresponding coordinates in the reference image r, it determines which reference image blocks in the reference image r are related to the query image block. Correspondingly, this method finds multiple candidate positive sample image patches. For example, if querying an image patch... If the coordinates of one or more pixels in the image are mapped to fall within the j-th reference image block in the reference image r, then the j-th reference image block can be considered as such. This is a positive sample image patch. The above candidate positive sample image patches and the query image patch... The degree of correlation between two image patches can be quantified by their spatial overlap; that is, the greater the overlap, the higher the correlation (i.e., the more similar) between the two image patches. In InfoNCE, the goal is to make the probability of identifying a unique positive sample from a noise set as close to 1 as possible. Based on this, when multiple positive image patches exist, the probability of identifying different positive image patches can be regularized using a prior distribution. Assume... If it is one of multiple positive sample image patches, then it is identified. The probability of this positive sample image patch can be obtained from the prior distribution. Perform regularization. Prior distribution. It is possible and This is defined by the area ratio of the shared region between the two image patches. In practice, this area ratio can be approximated by pixel count. Based on the coordinate table T provided by PixTrace (used to represent the correspondence between pixels before and after editing), we can first... The model is as follows:

[0178]

[0179] In formula (4) above, T[c] represents the query image patch. The coordinates c of a certain pixel in the image are mapped to the corresponding coordinate position in the reference image r. Indicates querying image blocks all pixels in the query image block | | denotes the number of elements in the respective set, e.g. denotes the total number of pixels in the query image block denotes the total number of pixels in the query image block denotes the number of pixels in the query image block denotes the total number of pixels in the query image block

[0180] Further, in order to enhance the flexibility of the prior distribution, a confidence adjustment parameter γ (also referred to as a confidence sharpening parameter) is introduced to adjust the certainty of the prior, and the

[0181]

[0182]

[0183] In practice, the value of γ can be, but is not limited to, 1, 2, 3, etc. Preferably, γ can be 3.

[0184] For example, assume that the query image block corresponds to three reference image blocks and in the reference image r, i.e. and These three reference image blocks are the three positive sample image blocks found in the reference image r for the query image block Then, the initial prior estimate is as follows:

[0185]

[0186] Assume γ = 2, then after the power transformation, we have:

[0187] ​​​​​​​​​​​​​

[0188] Further, after normalization:

[0189]

[0190] As can be seen from the above, after γ = 2 sharpening, the probability of the originally strongest matching item is higher, and other items are further weakened. That is, by sharpening the original initial prior, the model's ability to recognize the true matching region can be enhanced.

[0191] Here, the prior distribution By using the pixel-level traceability provided by the aforementioned PixTrace module, the regularization of the image block affinity (i.e., similarity) is established on the basis of the accurate image block pixel overlap ratio, rather than a simple binary positive and negative sample judgment.

[0192] By introducing the confidence adjustment parameter γ and subsequently based on this prior distribution When further converting the CopyNCE loss, the loss form based on the corresponding divergence (such as KL divergence) is involved, which enables the CopyNCE loss to handle multiple positive samples and assign different importance weights to different positive samples according to the actual region overlap degree, thereby effectively overcoming the noise gradient and performance degradation caused by false matching, partial matching, or ignoring important differences in traditional contrast learning.

[0193] Based on the above, further, the formula of CopyNCE can be converted to the following formula (6):

[0194]

[0195] wherein, is the KL[·‖·] represents the KL divergence. The KL divergence, also known as the Relative Entropy, is used to measure the difference between two probability distributions. According to the tuning experience of InfoNCE, the negative samples in InfoNCE should be as difficult as possible for the model to distinguish from positive samples. Under this guidance, the present case believes that the regions in the same image naturally have similar visual features, especially the regions near the infringing and infringed parts. Therefore, in CopyNCE, a contrast noise set R X is constructed by collecting all image blocks in the reference image. X = R R , R Rdenotes all image patches in the reference image r. With this contrastive noise set design, hard-negative samples within the reference image are preserved, which are image patches in the reference image that are visually similar to positive image patches Spatially neighboring image patches, both of which usually exhibit high visual similarity due to local texture continuity (e.g., repeating patterns), pose challenging interference factors for contrast judgment.

[0196] As can be seen from the above, in the CopyNCE loss, the accurate pixel overlap information provided by the PixTrace module is used to construct a clean, noise-free prior distribution q. Through this prior distribution q, the true overlap ratio of the infringement area between the query image patch and the reference image patch can be accurately quantified, and the importance difference of different image patches is considered. In addition, the following effects are also achieved: the model (image infringement detection model) containing the loss function of the CopyNCE loss can receive high-quality, low-noise supervision signals during the training process, and can avoid conflicting gradients caused by false matching, which can greatly improve the model's ability to learn accurate and robust corresponding relationships.

[0197] Figure 3 An area ratio calculation example of the shared area of the corresponding image patches between the query image and the reference image is shown in FIG. 6, and the area ratio is calculated by using the formula (6) above. As shown in FIG. 6, for one query image patch Figure 3 in the query image, four reference image patches #1, #2, #3 and #4 are found in the reference image r as corresponding candidate positive image patches, and it is calculated that the area ratios of the shared areas of the query image patch and the four reference image patches #1, #2, #3 and #4 are 12%, 20%, 20% and 48%, respectively.

[0198] Further, when infringement detection is performed on two images, since the corresponding relationship between infringement and infringement is bidirectional, that is, one image may be an infringement edited version of another image, but conversely, the other image may also be an infringement edited version of the corresponding one image. Through image infringement detection, it is intended to analyze whether there is an infringement relationship between the two images, but it is impossible to determine which image is the infringement edited version. Based on this, the expression form of CopyNCE is finally derived to be a symmetric form, as shown in the following formula (7):

[0199]

[0200] III. Loss function composition in image infringement detection model

[0201] ​To integrate the CopyNCE into the training pipeline of the corresponding image infringement detection model, and make the corresponding image infringement detection model (such as descriptor or matcher) endowed with basic infringement detection capability, the CopyNCE is taken as an auxiliary loss function in the present case, and is combined with the baseline loss function of the image infringement detection model such as descriptor or matcher to form the final loss function of the model.

[0202] That is, in the present case, the total loss function of the image infringement detection model is composed of two parts, one of which is the baseline loss function (used to evaluate the basic performance of the model), and the other part is the CopyNCE loss (used to enhance the image infringement detection capability of the model).

[0203] Exemplarily, as shown below:

[0204] For the matcher, the expression of the total loss function L mat of the matcher is shown in the following formula (8):

[0205]

[0206] For the descriptor, the expression of the total loss function L des of the descriptor is shown in the following formula (9):

[0207]

[0208] Wherein, w NCE is the weight of the CopyNCE loss. In the matcher and the descriptor, the values of w NCE are generally different, of course, they can also be the same in some other embodiments. Here, the value of w NCE is not specifically limited, and generally the value range of w NCE may be between 1 and 5. For example, in the matcher, the value of w NCE may be 1; and in the descriptor, the value of w NCE may be 5.

[0209] The above represents the baseline loss function of the matcher, which is defined as a simple binary classification function, and can be used to evaluate the basic performance of the matcher. represents the baseline loss function of the descriptor, which can be defined as the InfoNCE loss between the KoLeo loss and the image global feature, and can improve the robustness and generalization ability of the descriptor.

[0210] In summary, the scheme realizes the direct integration of fine-grained correspondence learning into the self-supervised training framework through the aforementioned PixTrace module and CopyNCE loss, forming a relatively unified and end-to-end image infringement detection method. Among them, the PixTrace module generates a supervision signal in the data preparation stage, and the CopyNCE loss is used as an auxiliary loss to integrate the network training, rather than requiring an additional independent target detection model or complex post-processing steps. This can produce beneficial effects including: reducing the error accumulation and propagation that may exist between multi-stage tasks, while the present case can achieve higher detection performance in a more compact and efficient manner, reducing the overall computational complexity and deployment cost.

[0211] Based on the above, the technical solutions provided by the present specification will be described in the form of method embodiments.

[0212] The method embodiments provided by the present specification are executed by an electronic device with logical operation function. The electronic device is deployed with a corresponding image infringement detection model. The electronic device can be a server. The server can be a commonly used server, a cloud or a virtual server, etc., and the present application embodiments do not make specific limitations. Of course, in other embodiments, the electronic device can also be a client, in which case, for example, the client has high-performance computing capability. The client can be any terminal device such as a mobile phone, a tablet computer, a smart wearable device, etc. The present specification mainly describes the server as the execution subject of the method embodiments.

[0213] Figure 5 The flowchart of the data processing method provided by an embodiment of the present specification is shown. As shown in Figure 5 The data processing method includes the following steps:

[0214] 102, obtaining a first image and a second image;

[0215] 104, performing bidirectional block-level contrast analysis on the first image and the second image, respectively taking the first image as the query image and the second image as the reference image, and taking the second image as the query image and the first image as the reference image, based on the pixel coordinate table of the query image, calculating the contrast loss value between the image blocks in the query image and the corresponding reference image;

[0216] Wherein, the pixel coordinate table is used to record the position information of the pixels in the query image; the contrast loss value optimizes the parameters of the image infringement detection model, and the image infringement detection model can be used to detect whether the first image and the second image have an infringement relationship.

[0217] In the above 102, the first image and the second image can be input by the user through the interactive interface provided by the client, or can also be obtained by crawling from the corresponding data source platform (such as a website, etc.). The embodiment does not make specific limitation on the acquisition mode of the first image and the second image.

[0218] Wherein, if the first image and the second image have an infringement relationship, one of the first image and the second image can be an original image, and the other can be generated by editing the original image, or the first image and the second image are derived from the same image (i.e. both are generated by editing the same image).

[0219] Further, in the process of continuing to compare and analyze the first image and the second image to analyze whether there is an infringement relationship between the first image and the second image, one of the images can be used as a query image and the other as a reference image for comparison and analysis to analyze the comparison loss value between the image blocks in the query image and the corresponding reference image. Then, the other image is used as a query image and the other image is used as a reference image for comparison and analysis to analyze the comparison loss value between the image blocks in the query image and the corresponding reference image. Then, the comparison loss values between the image blocks in the first image and the corresponding image blocks in the second image analyzed by the bidirectional analysis are averaged, and according to the average comparison loss value, it can be determined whether there is an infringement relationship between the first image and the second image.

[0220] Therefore, the bidirectional block-level comparison analysis of the first image and the second image described in the above 104 can specifically include:

[0221] 1042, taking the first image as a query image and the second image as a reference image, based on the pixel coordinates of the query image, performing block-level comparison analysis on the query image and the reference image, and calculating the first comparison loss value between the image blocks in the query image and the corresponding image blocks in the reference image;

[0222] 1044, taking the second image as a query image and the first image as a reference image, based on the pixel coordinates of the query image, performing block-level comparison analysis on the query image and the reference image, and calculating the second comparison loss value between the image blocks in the query image and the corresponding image blocks in the reference image.

[0223] In step 1042 above, the pixel coordinates of the queried image are actually the first pixel coordinates of the first image (used to record the position information of pixels in the first image). And the first contrast loss value is actually: based on the first pixel coordinate table of the first image, a block-level contrast analysis is performed on the first image and the second image, and the contrast loss value between the image blocks in the first image and the corresponding image blocks in the second image is calculated. The specific implementation of this step 10426 can be found in formula (6) or formula (7) provided in the other embodiments above.

[0224] In step 1044 above, the pixel coordinates of the queried image are actually the second pixel coordinates of the second image (used to record the position information of pixels in the second image). And the second contrast loss value is actually: based on the second pixel coordinate table of the second image, a block-level contrast analysis is performed on the second image and the first image, and the contrast loss value between the image blocks in the second image and the corresponding image blocks in the first image is calculated. For the specific implementation of this step 1044, refer to the formula (7) provided in the other embodiments above. and For a detailed calculation, please refer to the formula (6) above. The specific expression and calculation.

[0225] From the above, it can be understood that the contrast loss value mentioned in 106 above includes a first contrast loss value and a second contrast loss value. The first contrast loss value is calculated when the first image is the query image and the second image is the reference image, and the second contrast loss value is calculated when the second image is the query image and the first image is the reference image.

[0226] Furthermore, based on formula (7) given in the other embodiments above, the average of the first and second contrast loss values ​​calculated above can be used to optimize the parameters of the corresponding image infringement detection model. Therefore, the method provided in this embodiment may further include the following steps:

[0227] 106. Based on the second contrast loss value and the first contrast loss value, determine the average contrast loss value of corresponding image blocks between the first image and the second image;

[0228] The average contrast loss value is used to optimize the parameters of the image infringement detection model in order to improve its performance.

[0229] When the image infringement detection model is used to analyze the infringement between the first image and the second image to determine whether there is an infringement relationship between them, the inspection result output by the image infringement detection model for the first image and the second image will include a numerical value, which can be between 0 and 1, for example, to represent the probability of an infringement relationship between the first image and the second image. That is, the output value is an infringement probability. Of course, the inspection result output by the image infringement detection model can also include other content, such as a heat map. The description of the heat map will be described in detail in the relevant content below, and will not be described in detail here.

[0230] The following will be described in detail in combination with the formula (6) given in the foregoing other embodiments and the formula (4)-(5) related to the formula (6) to describe the specific implementation of the step 104 of calculating the contrast loss value between the image block in the query image and the image block in the corresponding reference image based on the pixel coordinates of the query image.

[0231] In the step 104, the coordinate table can be initialized according to the coordinates of the pixels in the query image to determine the pixel coordinate table corresponding to the query image, which records the coordinate position information of each pixel in the query image in the first image.

[0232] In order to realize the block-level comparison analysis of the query image and the reference image, the query image and the reference image need to be divided into a plurality of image blocks according to a certain image block size based on the pixel coordinates of the query image. The image infringement detection model used in this embodiment to realize the block-level comparison analysis of the query image and the reference image can be trained based on a corresponding preset model (such as a matcher or a descriptor) and the loss function used in the training of the matcher or the descriptor includes the geometric guided contrast loss function (CopyNCE) given by the formula (7) in the foregoing other embodiments. That is, the loss function of the image infringement detection model includes the geometric guided contrast loss function (CopyNCE).

[0233] In one division manner, the query image and the reference image can be directly divided according to a certain image block size. That is, in one implementation scheme, the method provided by the embodiment can further include the following steps:

[0234] S162, divide the query image according to the set image block size to obtain a plurality of query image blocks corresponding to the query image;

[0235] S164, divide the reference image according to a set image block size to obtain a plurality of reference image blocks corresponding to the reference image.

[0236] Alternatively, in order to simplify the subsequent comparative analysis of image blocks, the query image and the reference image can also be subjected to region of interest extraction first, so as to segment a corresponding query region from the query image and a corresponding reference image region from the reference image, the query region and the reference region being a candidate region pair that is identified as possibly having an infringement relationship. Further, the query region and the reference region can be divided respectively. Based on this, in another implementable technical solution, the method provided by the embodiment can further include the following steps:

[0237] S162', determine a query region in the query image and a reference region in the reference image; the query region and the reference region being a candidate region pair having an infringement relationship;

[0238] S164', divide the query region according to a set image block size to obtain a plurality of query image blocks;

[0239] S166', divide the reference region according to a set image block size to obtain a plurality of reference image blocks.

[0240] The set image block size is related to the visual encoding model contained in the image infringement detection model used, and specifically, the set image block size is the size of the smallest unit image block (also referred to as the smallest processing image block) of the visual encoding model. For example, if the visual architecture of the visual encoder is ViT, the set image block size can be 16x16 pixels.

[0241] Suppose that the target query image block is any one of the plurality of query image blocks corresponding to the query image, and the reference image has at least one candidate reference image block corresponding to the target query image block. The at least one candidate reference image block can be filtered out from the plurality of reference image blocks corresponding to the reference image according to the first pixel coordinate table of the query image. For example, the pixel coordinate table of the query image can be subjected to an inversion operation, so as to determine the coordinate positions in the reference image to which the pixel coordinates in the target query image are mapped when the pixel coordinates in the target query image are mapped to the reference image, so as to determine in which reference image blocks the pixel coordinates in the target query image are mapped to in the reference image, and the determined reference image blocks are taken as the candidate reference image blocks. Thus, the target query image and the candidate reference image block have a pixel correspondence relationship, which means that one or more pixels in the target query image block are mapped to the candidate reference image block when the pixel coordinates in the target query image block are mapped.

[0242] With the above content, the method provided by the embodiment further includes the following steps.

[0243] S22, determining at least one candidate reference image block from the reference image for the target query image block based on the pixel coordinate table of the query image.

[0244] Specifically, in an example, the implementation of the above step S22 includes:

[0245] S222, performing an inverse operation on the pixel coordinate table to obtain an inverse coordinate table;

[0246] S224, determining the coordinate mapping of each pixel in the target query image block to the corresponding coordinate position of the reference image based on the inverse coordinate table to obtain a mapping result;

[0247] S226, determining at least one candidate reference image block from the reference image according to the mapping result.

[0248] Specifically, in implementation, the at least one candidate reference image block is determined from a plurality of reference image blocks corresponding to the reference image.

[0249] Let the target candidate reference image block be any one of the at least one candidate image block, then the "calculating the contrast loss value between the target query image block and the target candidate reference image block based on the pixel coordinate table of the query image" included in the above step 104 can include the following steps:

[0250] S42, determining the prior distribution probability of the target candidate reference image block relative to the target query image block based on the pixel coordinate table.

[0251] S44, determining an identification probability of correctly identifying the target candidate reference image block from a set of noise image blocks based on the target query image block and the set of related noise image blocks, wherein the set of noise image blocks is constructed based on all image blocks in the reference image.

[0252] S46, calculating the contrast loss value between the target query image block and the target candidate reference image block according to the prior distribution probability and the identification probability.

[0253] In the above step S42, as described in the formula (4) given in the foregoing other embodiments, the target query image block here can correspond to the query image block in the formula (4) and the target candidate reference image block can correspond to the reference image block in the formula (4) Then based on the formula (4), this step S42 can be implemented by performing the following:

[0254] The pixel coordinate table of the query image is first inverted to obtain a corresponding reverse coordinate table. The reverse coordinate table can represent the coordinate mapping relationship between the coordinates of the pixels in the query image and the corresponding coordinate positions in the reference image. Thus, based on the reverse coordinate table, the number of pixels in the target query image block that are mapped into the target candidate reference image block after coordinate mapping can be determined. Further, the ratio of the number of pixels mapped into the target candidate reference image block to the total number of pixels in the target query image block is calculated, and the ratio is taken as the initial prior distribution probability (also referred to as the prior estimation probability). Then, the initial prior distribution probability is normalized. Specifically, as shown in the formula (5) given in the foregoing embodiments, when the normalization is performed, the initial prior distribution probability of each candidate reference image block other than the target candidate reference image block relative to the target query image block is determined, and the initial prior distribution probability corresponding to each candidate reference image block in the at least one candidate reference image block is subjected to power operation according to a preset confidence adjustment parameter γ, and the sum of all power operation results is obtained to obtain a total sum value (i.e., the value of the sum in the formula (5)). The total sum value is used to normalize the initial prior distribution probability corresponding to the target candidate reference image block. Specifically, in implementation, the ratio of the γth power operation result of the initial prior distribution probability corresponding to the target candidate reference image block to the total sum value is calculated to obtain the final prior distribution probability corresponding to the target candidate reference image block.

[0255] Based on the foregoing, in a specific implementable technical solution, the S42 “determining the prior distribution probability of the target candidate reference image block relative to the target query image block based on the pixel coordinate table” can include:

[0256] S422, determining the number of pixels in the target query image block that are mapped into the target candidate reference image block based on the pixel coordinate table;

[0257] S424, calculating the ratio of the number of pixels to the total number of pixels in the target query image block;

[0258] S426, performing power operation on the ratio according to a preset confidence adjustment parameter to obtain a power operation result;

[0259] S428, normalizing the power operation result to obtain the prior distribution probability.

[0260] Here, by introducing a confidence adjustment parameter, the flexibility of the first distribution can be enhanced, the probability of the strongest matching item can be improved, and multiple candidate reference image blocks can be checked. In addition, by using pixel overlap information to determine the prior distribution probability corresponding to the candidate reference image block, the prior distribution probability can accurately quantify the real overlap ratio of the infringement area between the target query image block and the corresponding candidate reference image block, and can consider the importance difference between different image blocks.

[0261] In the above S44, the identification probability calculation can be understood as calculating According to formula (3), it is not difficult to know that:

[0262]

[0263] Among them,

[0264]

[0265] And, the above z represents the image feature generated by encoding the image block by the visual coding model f θ For example, The image feature generated by encoding the image block by the visual coding model f θ For example, The image feature generated by encoding the image block by the visual coding model f The image feature generated by encoding the image block by the visual coding model f θ For example, The image feature generated by encoding the image block by the visual coding model f The meaning of

[0266] From this content and in combination with the above description of the image block size, it can be seen that when calculating the identification probability, the present scheme directly performs feature extraction and related calculations at the level of the smallest unit image block of the visual coding model f θ Such as ViT or CNN, here by operating the smallest unit image block, information loss can be avoided, and fine spatial details in the image can be ensured and utilized, which helps to improve the accuracy of subsequent identification probability for image infringement area positioning.

[0267] Further, based on the formula (6) given in the foregoing embodiments, in a specific implementable technical solution, the step S46 "calculating the comparison loss value of the target query image block and the target candidate reference image block based on the prior probability and the identification probability" can include:

[0268] S462, determining the probability difference measure between the prior distribution probability and the identification probability;

[0269] S464, determine the contrast loss value based on the probability difference measure.

[0270] In the above, the probability difference measure can be, but is not limited to, KL divergence or JS divergence, etc. In the present embodiment, the KL divergence is preferred. Further, the probability difference measure can be subjected to data expectation operation to obtain the corresponding contrast loss value.

[0271] As can be seen from the above description, the present scheme can directly perform comparison analysis at the image block level. Specifically, based on decomposing the images in the image pair into the minimum processing image blocks, the similarity (or matching, etc.) between the image blocks of different images can be accurately located by using the related formula (i.e. the related CopyNCE loss function) provided in the present specification, which has effects including: enabling the image infringement detection model containing the related formula (i.e. the CopyNCE loss function) in the loss function to learn more detailed similarity corresponding relationship between local regions than the image as a whole, thereby being able to effectively identify and locate complex local editing and tampering such as image cutout, local transformation, etc., rather than merely judging the similarity between the entire images.

[0272] In addition, the present application also visualizes the similar regions between the first image and the second image, which is achieved by drawing a heat map based on the similarity of the corresponding image blocks between the first image and the second image. Based on this, the method provided in the present embodiment can further include the following steps:

[0273] 108, generate a heat map reflecting the similar regions between the first image and the second image based on the similarity of the corresponding image blocks between the first image and the second image;

[0274] 1010, display the heat map.

[0275] In the above 108, the heat map reflecting the similar regions between the first image and the second image is drawn according to the similarity of all image block pairs having pixel correspondence relationship between the first image and the second image. Specifically, the heat map is drawn according to the similarity measure of the image block pairs, and the similarity measure of an image block pair is determined by subjecting the similarity of the image block pair to a corresponding transformation operation. The similarity referred to herein can be, but is not limited to, cosine similarity, and the transformation operation is a nonlinear transformation operation, which includes but is not limited to exponential operation (i.e. exp) with the base number e of natural logarithm.

[0276] Exemplarily, the similarity measure of the image block pair can be calculated by the following formula:

[0277]

[0278] wherein, denotes a similarity measure between the i-th query image block (e.g., the i-th image block in the first image) in the query image q and the j-th reference image block (e.g., the j-th image block in the second image) in the corresponding reference image r. denotes the image feature of the i-th query image block, denotes the image feature of the j-th reference image block. denotes the cosine similarity computed from the image features of the two image blocks.

[0279] It is to be noted that, according to the formula (6) and the formula (3) given in other embodiments of the present disclosure, the similarity between the corresponding image blocks of the first image and the second image, and the similarity measure obtained by corresponding transformation operation on the similarity, are actually obtained in the process of calculating the contrast loss value between the corresponding image blocks of the first image and the second image (as intermediate values in the process of calculating the contrast loss value).

[0280] Further, the generated heat map comprises at least one of a region affinity heat map and an affinity entropy heat map. The region affinity heat map is directly drawn according to the similarity measure between the corresponding image blocks of the first image and the second image. The affinity entropy heat map is drawn based on the affinity entropy calculated using the similarity measure between the corresponding image blocks of the first image and the second image. For example, the affinity entropy can be calculated by the following formula:

[0281] wherein,

[0282] wherein, ξ i denotes the affinity entropy, and the related The parameters in the above formula can refer to the related content described above, and will not be described in detail here. denotes the sum value of the similarity measures between all image block pairs having pixel correspondence relationship between the query image q (e.g., the first image) and the reference image r (e.g., the second image). denotes the image feature of the k-th reference image block.

[0283] It should be noted here that, in conjunction with the aforementioned content related to step 104 or formula (7) given in other embodiments, it can be seen that when this specification performs infringement analysis on the first image and the second image, it performs bidirectional block-level comparison analysis. Therefore, in the process of using the first image as the query image q and the second image as the reference image r to perform block-level comparison analysis on these two images to calculate the first comparison loss value of the corresponding image blocks between the first image and the second image, a first similarity between the corresponding image blocks between the first image and the second image will be generated, as well as a first similarity measure obtained by performing corresponding transformation operations on the first similarity. Conversely, in the process of performing block-level comparative analysis on the second image as the query image q and the first image as the reference image r to calculate the second contrast loss value of the corresponding image blocks between the first and second images, a second similarity between the corresponding image blocks between the second and first images is generated, as well as a second similarity measure obtained by performing corresponding transformation operations on the second similarity. Based on this, the heatmap drawn in this embodiment can be generated based on the first similarity measure obtained in the calculation of the first contrast loss and / or the second similarity measure obtained in the calculation of the second contrast loss. Specifically, when drawing a heatmap based on the first and second similarity measures, at least one of the following two methods is used: one method is to directly draw a region affinity heatmap based on the average of the first and second similarity measures; the other method is to first calculate the first affinity entropy corresponding to the first similarity and the second affinity entropy corresponding to the second similarity measure, and then draw an affinity entropy heatmap based on the average of the first and second affinity entropies.

[0284] Figure 6a This embodiment illustrates two types of heatmaps generated during image infringement detection using different types of image infringement detection models (such as matchers or descriptors, whose loss functions include the CopyNCE loss provided in this specification): a region affinity heatmap and an affinity entropy heatmap. Figure 6a In this case, squirrel eyes were chosen as the object of infringement, through which... Figure 6a The heatmap provided clearly shows that the squirrel's eye area is clearly highlighted and concentrated, clearly outlining the infringed object. This effectively reveals the infringement relationship between different images. In contrast, see [link to heatmap]. Figure 6b If the heatmap generated during the image infringement detection process is generated using an image infringement detection model that does not integrate the CopyNCE loss provided in this manual, it will appear messy and cannot effectively reveal the infringement correspondence between different images.

[0285] In 1010, when the heat map is displayed, it is displayed through a client interface. For example, see Figure 7 As shown, if the image infringement correspondence relationship analysis between the first image and the second image is completed by using the image infringement detection model on the server side, the server can send the generated heat map to the client for display.

[0286] With the above, the heat map generated can significantly improve the explainability of the infringement correspondence relationship, which helps to improve the understanding.

[0287] Another embodiment of the present specification also provides a data processing method. The data processing method comprises the following steps:

[0288] 002、obtain a first image and a second image;

[0289] 004、determine a first pixel coordinate table of the first image; the first pixel coordinate table is used to record the position information of the pixels in the first image;

[0290] 006、based on the first pixel coordinate table, performing block-level comparative analysis on the first image and the second image, and calculating a first comparison loss value between the image blocks in the first image and the corresponding image blocks in the second image;

[0291] The first comparison loss value is used to optimize the parameters of the image infringement detection model, and the image infringement detection model can be used to detect whether the first image and the second image have an infringement relationship.

[0292] For the specific implementation of each step of the above embodiment, please refer to the related content in other embodiments, which will not be described in detail here. In addition, the method provided by the present embodiment can also include some steps disclosed in other embodiments, and the related content in other embodiments can also be referred to, which will not be described here.

[0293] Figure 8 The flowchart of the data processing method provided by another exemplary embodiment of the present specification is shown. As shown in Figure 8 The method comprises the following steps:

[0294] 202、obtain a first image and a second image;

[0295] 204、input the first image and the second image into an image infringement detection model, and execute the image infringement detection model to output a detection result;

[0296] 206、display the detection result.

[0297] The detection result includes a heat map reflecting a similar region between the first image and the second image, and the heat map is generated according to similarity information generated in a bidirectional block-level comparison analysis process on the first image and the second image. The bidirectional block-level comparison analysis includes: taking the first image as a query image and the second image as a reference image, and taking the second image as a query image and the first image as a reference image, respectively, and calculating a comparison loss value between an image block in the query image and an image block in the corresponding reference image based on a pixel coordinate table of the query image. The similarity information includes the similarity between the image block in the query image and the image block in the corresponding reference image obtained in the process of calculating the comparison loss value.

[0298] The loss function of the image infringement detection model includes the CopyNCE loss provided in other embodiments of the present specification. In addition, the output detection result can include an infringement probability value in addition to the heat map, and the infringement probability value is used to represent the probability of the existence of an infringement relationship between the first image and the second image, and the probability is between 0 and 1.

[0299] For specific implementation of each step of the present embodiment, please refer to the related content in other embodiments, which will not be described in detail here. In addition, the method provided by the present embodiment can also include some steps disclosed in other embodiments, and the related content in other embodiments can also be referred to, which will not be described here.

[0300] The scheme provided by the present embodiment makes the output detection result include a heat map, which can significantly improve the explainability of the infringement corresponding relationship, and helps to improve the understanding. In addition, it can also improve the reliability of the image detection model, and at the same time provides more intuitive infringement region positioning information for the user, which is helpful for subsequent content analysis and decision making, such as whether there is an infringement relationship.

[0301] Figure 9 A flowchart of a data processing method provided by another example embodiment of the present specification is shown. As shown in Figure 9 The method includes the following steps:

[0302] 302, obtaining a first image and a second image;

[0303] 304. Input the first image and the second image into the image infringement detection model, execute the image infringement detection model and output the detection result; wherein, the loss function of the image infringement detection model includes a composite contrast loss function, which is constructed based on the first contrast loss function and the second contrast loss function; using the first contrast loss function, based on the first pixel coordinate table of the first image, block-level contrast analysis can be performed on the first image and the second image to calculate the first contrast loss value between the image block in the first image and the corresponding image block in the second image; using the second contrast loss function, based on the second pixel coordinate table of the second image, block-level contrast analysis can be performed on the second image and the first image to calculate the second contrast loss value between the image block in the second image and the corresponding image block in the first image.

[0304] The composite contrast loss function described above is the same as the geometrically guided contrast loss function (CopyNCE loss) described in other embodiments of this specification. For a detailed description of the composite contrast loss function, please refer to the content related to CopyNCE loss provided in other embodiments; it will not be elaborated upon here.

[0305] For specific implementation details of the steps given in this embodiment, please refer to the relevant content in other embodiments, which will not be elaborated here. Furthermore, the method provided in this embodiment may also include some steps disclosed in other embodiments, which can also be referred to the relevant content in other embodiments, which will not be elaborated here.

[0306] Figure 10 A flowchart illustrating a data processing method provided in yet another exemplary embodiment of this specification is shown. For example... Figure 10 As shown, the method includes the following steps:

[0307] 402. Obtain a training sample set, wherein the training sample set contains multiple sample image pairs;

[0308] 404. Set the loss function used during the training of the contrastive learning model;

[0309] 405. Based on the multiple sample image pairs and the loss function, train the contrastive learning model to obtain an image infringement detection model;

[0310] The loss function includes a composite contrast loss function constructed based on a first contrast loss function and a second contrast loss function. The first contrast loss function is used to perform block-level contrast analysis on two sample images contained in the sample image pair based on a first pixel coordinate table of one sample image in the sample image pair, and calculate a first contrast loss value of corresponding image blocks between the two sample images. The second contrast loss function is used to perform block-level contrast analysis on the two sample images contained in the sample image pair based on a second pixel coordinate table of the other sample image in the sample image pair, and calculate a second contrast loss value of corresponding image blocks between the two sample images.

[0311] In the above, the training sample set can be a public image sample set such as ImageNet, etc. The contrast learning model can be a matcher or a descriptor. The loss function used when the contrast learning model is trained includes a baseline loss function and a composite contrast function. The composite contrast loss function is the geometric guided contrast loss function (CopyNCE loss) described in other embodiments of the present specification.

[0312] For specific implementation of each step given in the above embodiment, refer to the related content in other embodiments, which will not be repeated here. In addition, the method provided in the present embodiment can also include some steps disclosed in other embodiments, and the related content in other embodiments can also be referred to, which will not be repeated here.

[0313] Another exemplary embodiment of the present specification also provides an image infringement detection model, which includes a plurality of network layers, and the plurality of network layers include an input layer, an encoding layer and a loss layer. Wherein,

[0314] The input layer is configured to receive an input first image and a second image

[0315] The encoding layer has two encoding modules, one of which is configured to encode the first image block in the first image to output the image block feature of the first image block, and the other of which is configured to encode the second image block in the second image to output the image block feature of the second image block.

[0316] The loss layer is provided with a loss function, and the loss function comprises a composite contrast loss function; the composite contrast loss function is constructed based on a first contrast loss function and a second contrast loss function; the first contrast loss function is used for performing block-level contrast analysis on the first image and the second image based on a first pixel coordinate table of the first image and image block features output by the encoding layer, and calculating a first contrast loss value of corresponding image blocks between the first image and the second image; the second contrast loss function is used for performing block-level contrast analysis on the second image and the first image based on a second pixel coordinate table of the second image and the image block features output by the encoding layer, and calculating a second contrast loss value of corresponding image blocks between the second image and the first image.

[0317] The first contrast loss value and the second contrast loss value are used for optimizing parameters of the image infringement detection model, and the image infringement detection model can be used to detect whether there is an infringement relationship between the first image and the second image.

[0318] In the above, the encoding layer has two encoding modules, that is, an encoder, one of which is used for encoding processing of image blocks in the first image, and the other is used for encoding processing of image blocks in the second image. Understandably, the encoding processing refers to feature extraction of the corresponding image blocks, realizing the conversion of image block data into a series of image feature vectors, and these image features retain the key information of the image blocks. The above two encoding models can share weights.

[0319] In addition, the loss function deployed on the loss layer comprises a baseline loss function and a composite contrast loss function. The composite contrast loss function is the geometric guided contrast loss function (CopyNCE loss) described in other embodiments of the present specification.

[0320] In addition to the above, the image infringement detection model can further comprise other network layers. For example, referring to Figure 4 In the case of the image infringement detection model being a matcher, other network layers such as a connection layer (Concat), a fusion layer (Fusion), and a head layer (Head) can also be included. The connection layer is used to establish direct or indirect contact between the outputs of the two encoding modules, and its form can be, for example, a skip connection, a dense connection, etc. The fusion layer is used to integrate feature information from different sources or branches. The head layer is usually located at the end of the entire network and is responsible for mapping the learned features into the final task output.

[0321] In the present embodiment, in addition to being a matcher, the image infringement detection model can also be of other types, such as a descriptor.

[0322] The image infringement detection model provided in the embodiment can be used for image infringement detection analysis, and the following effects can be achieved.

[0323] 1) High detection performance.

[0324] Using a relatively authoritative data set in the image infringement detection (ICD) field, such as the DISC21 data set, for test analysis, the matcher and the descriptor both achieve SOTA performance (i.e., relatively optimal performance) in the case of the image infringement detection model being a matcher and a descriptor. Specifically, the matcher can achieve about 88.7% μAP and 83.9% RP90, and the descriptor can achieve about 72.6% μAP and 68.4% RP90.

[0325] 2) Stronger generalization ability:

[0326] On the NDEC data set containing more difficult negative samples, the composite comparison loss (CopyNCE loss) included in the image infringement detection model also performs well, with a μAP significantly higher than other strong baseline losses. For example, the μAP of the CopyNCE loss can reach about 72.5%, while the μAP of other strong baseline losses (such as Strong ASL) can reach about 64.1%. This indicates that the present scheme has better robustness and adaptability when dealing with more challenging real-world data.

[0327] On the AnyPattern data set, the image detection model needs to face more diverse image data editing methods, and these image data editing methods have not been used in the training stage. Under the same setting, the performance of the composite comparison loss (CopyNCE loss) can reach about 28.1% μAP and 34.4% R@1.

[0328] 3) Higher efficiency:

[0329] By integrating the composite comparison loss (CopyNCE loss) into the image infringement detection model, fine-grained learning can be internalized into the self-supervised training process, avoiding the dependence on additional complex components (such as target detectors), making the overall system more streamlined and efficient.

[0330] 4) Good model robustness:

[0331] The composite comparison loss (CopyNCE loss) can exhibit good robustness in model training. For example, in the descriptor, even if the CopyNCE loss is calculated at different layers (such as the 10th layer), similar results can be obtained, further confirming the effectiveness of its design.

[0332] For details of the specific role of the image infringement detection model given in the above embodiment, please refer to the relevant content in other embodiments, which will not be described here.

[0333] Figure 11 A flowchart of a data processing method provided by another example embodiment of the present specification is shown. As shown in Figure 11 The method includes the following steps:

[0334] 502, determine the first pixel coordinate table of the first image and the second pixel coordinate table of the second image;

[0335] 504, perform an inverse operation on the first pixel coordinate table to obtain an inverse coordinate table;

[0336] 506, based on the inverse coordinate table and the second pixel coordinate table, determine the pixel correspondence relationship between the first image and the second image;

[0337] 508, based on the pixel correspondence relationship, perform block-level comparison analysis on the first image and the second image, and calculate the contrast loss value between the image blocks in the first image and the corresponding image blocks in the second image.

[0338] The above steps 502, 504 and 506 correspond to the inverse tracking related content supported by the PixTrace module described in other embodiments of the present specification.

[0339] For details of the implementation of each step given in the above embodiment, please refer to the relevant content in other embodiments, which will not be described here. In addition, the method provided by the present embodiment can also include some steps disclosed in other embodiments, and the relevant content in other embodiments can also be referred to, which will not be described here.

[0340] Figure 12 A flowchart of a data processing method provided by another example embodiment of the present specification is shown. As shown in Figure 12 The method includes the following steps:

[0341] 602, obtain the first pixel coordinate table of the first image and the second pixel coordinate table of the second image; wherein the first image and the second image are generated by editing the third image, and the first pixel coordinate table is generated by mapping the coordinates of the pixels in the third image to the first image, and the second pixel coordinate table is generated by mapping the coordinates of the pixels in the third image to the second image;

[0342] 604, perform an inverse operation on the first pixel coordinate table to generate an inverse coordinate table;

[0343] 605、based on the second pixel coordinate table and the reverse coordinate table, mapping the coordinates of the pixels in the first image to the second image to obtain the pixel correspondence between the first image and the second image;

[0344] The pixel correspondence is used to participate in calculating a contrast loss value between an image block in the first image and a corresponding image block in the second image.

[0345] In the steps 604-605, after the first pixel coordinate table is inverted, the coordinates of the pixels in the first image are mapped to the coordinate system of the third image to generate the reverse coordinate table. Then, based on the second pixel coordinate table, the composite function used for coordinate transformation in the process of generating the second image by editing the third image can be determined, and the coordinates in the reverse coordinate table can be updated using the composite function, so as to map the coordinates of the pixels in the first image to the second image, thereby obtaining the pixel correspondence between the first image and the second image.

[0346] The steps 602, 604 and 606 are specifically related to the image bridging tracking supported by the PixTrace module described in other embodiments of the present specification.

[0347] For specific implementation of each step of the present embodiment, refer to the related content in other embodiments, which will not be described in detail here. In addition, the method provided by the present embodiment can also include some steps disclosed in other embodiments, and the related content in other embodiments can also be referred to, which will not be described here.

[0348] The method embodiments provided above mainly describe the technical solutions provided by the present specification from the perspective of images. The technical solutions can also be applied to video infringement detection and analysis scenarios. Assuming that the video duration of the first video obtained is one minute and the video duration of the second video is 30 seconds, when video infringement detection and analysis is performed on the first video and the second video, the corresponding detection process generally includes the following steps:

[0349] Step 1: frame extraction is performed on the first video and the second video respectively, for example, one frame is extracted every one second, so that 60 frames of images are extracted from the first video and 30 frames of images are extracted from the second video;

[0350] Step 3: image features of each frame of image extracted are determined. Then, the first video will correspond to 60 image features, and the second video will correspond to 30 image features.

[0351] Step 5, the 60 image features corresponding to the first video and the 30 image features corresponding to the second video are matched to calculate the similarity of each two image features between the first video and the second video, so as to obtain a 60*30 similarity matrix.

[0352] Step 7, based on the similarity matrix, it is determined whether there is an infringement relationship between the first video and the second video.

[0353] Generally speaking, if a video segment in the first video and a video segment in the second video are mutually infringing, the values on the diagonal of the obtained similarity matrix will be relatively high, and the values off the diagonal will be relatively low. After visualizing the similarity matrix, the brightness of the diagonal will also be relatively high (i.e. the similarity feature image will have a relatively bright diagonal line). A target detector can be used to find this diagonal line, so as to output from which to which in the similarity matrix a diagonal line meeting the brightness requirement is found. Then, based on the found diagonal line, the positions corresponding to the first video and the second video are found, so as to determine that a video segment in the second video infringes a video segment in the first video.

[0354] The video infringement detection process described above is a standard detection process that can be run on any image infringement detection model. Preferably, the image infringement detection model used when performing video infringement detection analysis is a descriptor, because the matcher will perform image reorganization when analyzing, and the analysis process is relatively cumbersome compared to the descriptor. Further, if the image infringement detection model is obtained by using the training method provided in the specification, i.e. the loss function used during training includes the geometric guidance contrast loss function (CopyNCE loss) provided in the specification, the video infringement detection analysis effect will also be relatively good.

[0355] Based on the above, another exemplary embodiment of the specification provides a data processing method. As shown in the figure, the method comprises the following steps: Figure 13

[0356] 702, obtain a first video and a second video;

[0357] 704, frame extraction processing is performed on the first video to obtain a first number of frame images extracted from the first video;

[0358] 706, frame extraction processing is performed on the second video to obtain a second number of frame images extracted from the second video;

[0359] ​708、using the trained image infringement detection model, performing: analyzing the similarity between each pair of images between the first number of frames of images and the second number of frames of images to obtain a similarity matrix; determining whether there is an infringement relationship between the first video and the second video based on the similarity matrix.

[0360] In the training process, the loss function used by the image infringement detection model includes a composite contrast loss function; the geometric guidance contrast loss function is constructed based on a first contrast loss function and a second contrast loss function; using the first contrast loss function, block-level contrast analysis can be performed on two sample images contained in a sample image pair based on a first pixel coordinate table of one sample image in the sample image pair, and a first contrast loss value of corresponding image blocks between the two sample images is calculated; using the second contrast loss function, block-level contrast analysis can be performed on two sample images contained in a sample image pair based on a second pixel coordinate table of another sample image in the sample image pair, and a second contrast loss value of corresponding image blocks between the two sample images is calculated.

[0361] For the specific implementation of each step given in the above embodiment, refer to the related content in other embodiments, which will not be repeated here. In addition, the method provided in the embodiment can also include some steps disclosed in other embodiments, and the related content in other embodiments can also be referred to, which will not be repeated here.

[0362] The present specification also provides another embodiment of a data processing system. As described above, Figure 7 The exemplary data processing system structure is shown in the figure, which includes a server 200 and a client 100

[0363] The server 200 is deployed with an image infringement detection model, which is used to: obtain a first image and a second image; input the first image and the second image into the image infringement detection model, and execute the image infringement detection model to output a detection result;

[0364] The client 100 is used to display the detection result.

[0365] The detection result includes a heatmap reflecting similar regions between the first image and the second image. The heatmap is generated based on similarity information generated during bidirectional block-level comparison analysis of the first and second images. The bidirectional block-level comparison analysis includes: using the first image as the query image and the second image as the reference image, and using the second image as the query image and the first image as the reference image, respectively, calculating the contrast loss value between image blocks in the query image and image blocks in the corresponding reference image based on the pixel coordinate table of the query image. The similarity information includes the similarity between image blocks in the query image and image blocks in the corresponding reference image obtained during the calculation of the contrast loss value.

[0366] For specific implementation details of the terminal functions given in this embodiment, please refer to the relevant content in other embodiments, which will not be elaborated here. Furthermore, the terminal functions provided in this embodiment may also include some functions / steps disclosed in other embodiments, which can also be referred to the relevant content in other embodiments, which will not be elaborated here.

[0367] The apparatus embodiments corresponding to the various method embodiments provided in this specification are described below.

[0368] Figure 14 A schematic diagram of the structure of a data processing apparatus provided in an exemplary embodiment of this specification is shown. For example... Figure 14 As shown, the device includes an acquisition module 82 and a comparison analysis module 84. The acquisition module 82 is used to acquire a first image and a second image. The comparison analysis module 84 is used to perform bidirectional block-level comparison analysis on the first image and the second image, respectively using the first image as the query image and the second image as the reference image, and using the second image as the query image and the first image as the reference image, based on the pixel coordinate table of the query image, to calculate the comparison loss value between image blocks in the query image and corresponding image blocks in the reference image; wherein the pixel coordinate table is used to record the position information of pixels in the query image. The comparison loss value is used to optimize the parameters of the image infringement detection model, which can detect whether there is an infringement relationship between the first image and the second image.

[0369] In an implementation, the target query image block is one image block in the query image, the reference image has at least one candidate reference image block corresponding to the target query image block, the target query image block has a pixel correspondence relationship with the candidate reference image block; the target candidate reference image block is one of the at least one candidate image block, and the contrast analysis module 84 is specifically configured to: determine the prior distribution probability of the target candidate reference image block relative to the target query image block based on the first pixel coordinate table; determine the identification probability of correctly identifying the target candidate reference image block from the set of noise image blocks according to the target query image block and the set of related noise image blocks; wherein the set of noise image blocks is constructed based on all image blocks in the reference image; and calculate the contrast loss value of the target query image block and the target candidate reference image block according to the prior distribution probability and the identification probability.

[0370] In an implementation, the contrast analysis module 84 is specifically configured to: determine the number of pixels in the target query image block that are mapped to pixels within the target candidate reference image block based on the pixel coordinate table; calculate the ratio of the number of pixels to the total number of pixels in the target reference image block; perform power operation on the ratio according to a preset confidence adjustment parameter to obtain a power operation result; and normalize the power operation result to obtain the prior distribution probability, when determining the prior distribution probability of the target candidate reference image block relative to the target query image block based on the pixel coordinate table.

[0371] In an implementation, the contrast analysis module 84 is specifically configured to: determine the probability difference measure between the prior distribution probability and the identification probability; and determine the contrast loss value based on the probability difference measure, when calculating the contrast loss value of the target query image block and the target candidate reference image block based on the prior probability and the identification probability.

[0372] In an implementation, the apparatus further includes an inversion module configured to perform an inversion operation on the pixel coordinate table to obtain an inverse coordinate table, and the determination module 84 is further configured to: determine the coordinate mapping of each pixel in the target query image block to the corresponding coordinate position in the reference image based on the inverse coordinate table to obtain a mapping result; and determine the at least one candidate reference image block from the reference image according to the mapping result.

[0373] In an embodiment, the determining module 84 is further configured to determine a query region in the query image and a reference region in the reference image, and the query region and the reference region are a candidate region pair having an infringement relationship. The apparatus further includes a dividing module configured to divide the query region according to a set image block size to obtain a plurality of query image blocks, and divide the reference region according to the set image block size to obtain a plurality of reference image blocks. The target query image block is any one of the plurality of query image blocks, and the at least one candidate reference image block is determined from the plurality of second image blocks.

[0374] In an embodiment, the apparatus further includes a determining module configured to determine an average contrast loss value of corresponding image blocks between the first image and the second image according to a first contrast loss value and a second contrast loss value. The average contrast loss value is used to optimize parameters of the image infringement detection model. The first contrast loss value is calculated in a case where the first image is a query image and the second image is a reference image, and the second contrast loss value is calculated in a case where the second image is a query image and the first image is a reference image.

[0375] Another exemplary embodiment provided in the specification also provides a data processing apparatus. The structure of the apparatus can be seen from Figure 14 a data processing apparatus shown in the specification. Specifically, the apparatus includes an obtaining module, a determining module, and a contrast analysis module. The obtaining module is configured to obtain a first image and a second image. The determining module is configured to determine a first pixel coordinate table of the first image, and the pixel coordinate table is used to record position information of pixels in the first image. The contrast analysis module is configured to perform block-level contrast analysis on the first image and the second image based on the first pixel coordinate table, and calculate a first contrast loss value between an image block in the first image and a corresponding image block in the second image. The first contrast loss value is used to optimize parameters of an image infringement detection model, and the image infringement detection model can be used to detect whether the first image and the second image have an infringement relationship.

[0376] Figure 15 A structural schematic diagram of a data processing apparatus provided by another exemplary embodiment in the specification is shown. As Figure 15As shown, the apparatus includes: an acquisition module 92, an execution module 94, and a display module 96. The acquisition module 92 is configured to acquire a first image and a second image. The execution module 94 is configured to input the first image and the second image into an image infringement detection model, and execute the image infringement detection model to output a detection result. The display module 96 is configured to display the detection result. The detection result includes a heat map reflecting a similar region between the first image and the second image, and the heat map is generated according to similarity information generated in a bidirectional block-level comparison analysis process of the first image and the second image. The bidirectional block-level comparison analysis includes: taking the first image as a query image and the second image as a reference image, and taking the second image as a query image and the first image as a reference image, respectively; and calculating a comparison loss value between an image block in the query image and a corresponding image block in the reference image based on a pixel coordinate table of the query image. The similarity information includes a similarity between the image block in the query image and the corresponding image block in the reference image obtained in the process of calculating the comparison loss value.

[0377] Another example embodiment in the specification also provides a data processing apparatus. The structure of the apparatus can be seen from Figure 15 The structure of the data processing apparatus. Specifically, the apparatus includes: an acquisition module, an execution module. The acquisition module is configured to acquire a first image and a second image. The execution module is configured to input the first image and the second image into an image infringement detection model, and execute the image infringement detection model to output a detection result. The loss function of the image infringement detection model includes a composite comparison loss function, and the composite comparison loss function is constructed based on a first comparison loss function and a second comparison loss function. The first comparison loss function can be used to perform block-level comparison analysis on the first image and the second image based on a first pixel coordinate table of the first image, and calculate a first comparison loss value between an image block in the first image and a corresponding image block in the second image. The second comparison loss function can be used to perform block-level comparison analysis on the second image and the first image based on a second pixel coordinate table of the second image, and calculate a second comparison loss value between an image block in the second image and a corresponding image block in the first image.

[0378] Figure 16 The structure of the model training apparatus provided by another example embodiment in the specification is shown. As Figure 16As shown, the apparatus includes: an acquisition module 1012, a setting module 1014, and a training module 1016. The acquisition module 1012 is configured to acquire a training sample set, wherein the training sample set includes a plurality of sample image pairs. The setting module 1014 is configured to set a loss function used in training of a contrast learning model. The training module is configured to perform model training on the contrast learning model based on the plurality of sample image pairs and the loss function, to obtain an image infringement detection model. The loss function includes a composite contrast loss function, and the composite contrast loss function is constructed based on a first contrast loss function and a second contrast loss function. The first contrast loss function is used to perform block-level contrast analysis on two sample images included in the sample image pair based on a first pixel coordinate table of one sample image in the sample image pair, to calculate a first contrast loss value of corresponding image blocks between the two sample images. The second contrast loss function is used to perform block-level contrast analysis on the two sample images included in the sample image pair based on a second pixel coordinate table of another sample image in the sample image pair, to calculate a second contrast loss value of corresponding image blocks between the two sample images.

[0379] Figure 17 A structural schematic diagram of a data processing apparatus provided by still another example embodiment of the present specification is shown. As shown in the figure, Figure 17 As shown, the apparatus includes: a determination module 1112, an inversion module 1114, and an analysis and calculation module 1116. The determination module 1112 is configured to determine a first pixel coordinate table of a first image and a second pixel coordinate table of a second image. The inversion module 1114 is configured to perform an inversion operation on the first pixel coordinate table to obtain an inverse coordinate table. The determination module 1112 is further configured to determine a pixel correspondence relationship between the first image and the second image based on the inverse coordinate table and the second pixel coordinate table. The analysis and calculation module 1116 is configured to perform block-level contrast analysis on the first image and the second image based on the pixel correspondence relationship, to calculate a contrast loss value between an image block in the first image and a corresponding image block in the second image.

[0380] Figure 18 A structural schematic diagram of a data processing apparatus provided by still another example embodiment of the present specification is shown. As shown in the figure, Figure 18As shown, the apparatus includes: an obtaining module 1212, an reversing module 1214, a mapping obtaining module 1216. The obtaining module 1212 is configured to obtain a first pixel coordinate table of a first image and a second pixel coordinate table of a second image; wherein the first image and the second image are generated by performing an editing operation on a third image, and the first pixel coordinate table is generated by mapping the coordinates of pixels in the third image to the first image, and the second pixel coordinate table is generated by mapping the coordinates of pixels in the third image to the second image. The reversing module 1214 is configured to perform a reverse operation on the first pixel coordinate table to obtain a reverse coordinate table. The mapping obtaining module 1216 is configured to map the coordinates of pixels in the first image to the second image based on the second pixel coordinate table and the reverse coordinate table to obtain a pixel correspondence relationship between the first image and the second image. The pixel correspondence relationship is used to participate in calculating a contrast loss value between image blocks in the first image and corresponding image blocks in the second image.

[0381] Figure 19 A structural schematic diagram of a data processing apparatus provided by still another example embodiment in the specification is shown. As shown in the figure, Figure 19 As shown, the apparatus includes: an obtaining module 1312, an extracting module 1314, and an executing module 1316. The obtaining module 1312 is configured to obtain a first video and a second video. The extracting module 1314 is configured to perform frame extraction processing on the first video to obtain a first number of frame images extracted from the first video, and perform frame extraction processing on the second video to obtain a second number of frame images extracted from the second video. The executing module 1316 is configured to use a trained image infringement detection model to perform: analyzing the similarity of two images between the first number of frame images and the second number of frame images to obtain a similarity matrix; and determining whether there is an infringement relationship between the first video and the second video based on the similarity matrix. The loss function used by the image infringement detection model in the training process includes a composite contrast loss function; the composite contrast loss function is constructed based on a first contrast loss function and a second contrast loss function; the first contrast loss function is used to perform block-level contrast analysis on two sample images included in a sample image pair based on a first pixel coordinate table of one sample image in the sample image pair to calculate a first contrast loss value of corresponding image blocks between the two sample images; and the second contrast loss function is used to perform block-level contrast analysis on the two sample images included in the sample image pair based on a second pixel coordinate table of the other sample image in the sample image pair to calculate a second contrast loss value of corresponding image blocks between the two sample images.

[0382] It should be noted that the above devices need to be described here: the above provided devices can realize the technical solutions described in the above corresponding method embodiments, and the principles of the implementation of the above modules or units can be referred to the related content in the above corresponding method embodiments, which will not be described in detail here.

[0383] In addition, the embodiments of the present specification also provide an electronic device. As shown in Figure 20 , the electronic device includes a memory 1411 and a processor 1412.

[0384] The memory 1411 stores executable code. When the processor 1412 executes the executable code, the steps in the method embodiments described as Figure 5 , Figures 8 to 13 .

[0385] Further, as shown in Figure 20 , the electronic device further includes a communication component 1413, a display 1414, a power component 1415, an audio component 1416 and other components. Figure 20 Some components are only schematically shown in the above Figure 20 , which does not mean that the electronic device only includes the components shown in the above. In addition, Figure 20 , the components in the dashed box are optional components, not mandatory components, which can be determined according to the product form of the electronic device. The electronic device of the present embodiment can be implemented as a desktop computer, a notebook computer, a smart phone or an IOT device, etc. terminal device, or a conventional server, a cloud server or a server array, etc. server device, or it can also be an integrated device of terminal device and server device, etc. If the electronic device of the present embodiment is implemented as a desktop computer, a notebook computer, a smart phone, etc. terminal device, it can contain the components in the dashed box in the above Figure 20 ; if the electronic device of the present embodiment is implemented as a conventional server, a cloud server or a server array, etc. server device, it can not contain the components in the dashed box in the above Figure 20 .

[0386] The above-described memory can be implemented by any type of volatile or nonvolatile memory devices or a combination thereof, such as a Static Random-Access Memory (SRAM), an Electrically Erasable Programmable Read Only Memory (EEPROM), an Erasable Programmable Read Only Memory (EPROM), a Programmable Read-Only Memory (PROM), a Read-Only Memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or a compact disk.

[0387] The above-described communication component is configured to facilitate communication between the device in which the communication component is located and other devices in a wired or wireless manner. The device in which the communication component is located can access a wireless network based on a communication standard, such as a 2G, 3G, 4G / LTE, 5G, or the like mobile communication network, or a combination thereof. In an example embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast managing system via a broadcast channel.

[0388] The above-described display includes a screen, which can include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes the touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect a duration and a pressure related to the touching or the sliding action.

[0389] The above-described power component provides power to various components of the device in which the power component is located. The power component can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which the power component is located.

[0390] The above-described audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device in which the audio component is located is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory or transmitted via the communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0391] Accordingly, the embodiments of the present application also provide a computer readable storage medium storing a computer program, when the computer program is executed by a processor, the processor is enabled to implement each step in the above-mentioned method embodiments. The computer readable storage medium includes volatile or non-volatile or their combination, and can be removable or non-removable. Examples of the computer readable storage medium include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital video disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium

[0392] In addition, the embodiments of the present application also provide a computer readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, the computer is enabled to perform the method as described above. Figures 5 to 7

[0393] The embodiments of the present application also provide a computer program product, including computer programs / instructions, which, when executed by a processor, implement the method as described above. Figures 5 to 7

[0394] Those skilled in the art should be aware that the functions described in the above one or more examples can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium.

[0395] In some cases, the actions or steps recited in the claims can be performed in a different order than the order described in the embodiments and still achieve the desired results. In addition, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In some embodiments, multi-task processing and parallel processing are also possible or advantageous.

[0396] ​​It is also need to note that, in the present document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0397] The specific implementation described above is for the purpose of further detailing the purposes, technical solutions and beneficial effects of the embodiments disclosed in the present specification. It should be understood that the above description is only a specific implementation of the embodiments disclosed in the present specification and is not intended to limit the protection scope of the embodiments disclosed in the present specification. Any modification, equivalent replacement, improvement and the like made on the basis of the technical solutions of the embodiments disclosed in the present specification shall be included in the protection scope of the embodiments disclosed in the present specification.

Claims

1. A data processing method, characterized by, The method comprises: acquiring a first image and a second image; performing bidirectional block-level contrast analysis on the first image and the second image, respectively taking the first image as a query image and the second image as a reference image, and taking the second image as a query image and the first image as a reference image, and calculating a contrast loss value between an image block in the query image and an image block in the corresponding reference image based on a pixel coordinate table of the query image; wherein the pixel coordinate table is used to record the position information of the pixels in the query image; and the contrast loss value is used to optimize the parameters of an image infringement detection model, and the image infringement detection model can detect whether there is an infringement relationship between the first image and the second image.

2. The method of claim 1, wherein, The target query image block is an image block in the query image, the reference image has at least one candidate reference image block corresponding to the target query image block, and the target image block and the candidate reference image block have a pixel correspondence relationship; The target candidate reference image block is one of the at least one candidate reference image block, and then: based on the pixel coordinate table of the query image, the contrast loss value between the target query image block and the target candidate reference image block is calculated, which comprises: based on the pixel coordinate table, determining the prior distribution probability of the target candidate reference image block relative to the target query image block; determining the identification probability of correctly identifying the target candidate reference image block from the noise image block set according to the target query image block and the related noise image block set; wherein the noise image block set is constructed based on all image blocks in the reference image; according to the prior distribution probability and the identification probability, the contrast loss value of the target query image block and the target candidate reference image block is calculated.

3. The method of claim 2, wherein, Based on the pixel coordinate table, the prior distribution probability of the target candidate reference image block relative to the target query image block is determined, which comprises: based on the pixel coordinate table, determining the number of pixels in the target query image block that map to the pixels inside the target candidate reference image block; calculate the ratio of the number of pixels to the total number of pixels in the target query image block; according to the preset confidence adjustment parameter, the ratio is subjected to power operation to obtain a power operation result; the power operation result is normalized to obtain the prior distribution probability.

4. The method of claim 2, wherein, Based on the prior probability and the identification probability, the contrast loss value of the target query image block and the target candidate reference image block is calculated, which comprises: determining the probability difference measure between the prior distribution probability and the identification probability; based on the probability difference measure, the contrast loss value is determined.

5. The method of claim 2, wherein, Further comprising: performing an inverse operation on the pixel coordinate table to obtain an inverse coordinate table; based on the inverse coordinate table, the coordinate mapping of each pixel in the target query image block to the corresponding coordinate position in the reference image is determined to obtain a mapping result; according to the mapping result, the at least one candidate reference image block is determined from the reference image.

6. The method of claim 2, wherein, Further comprising: determining a query region in the query image and a reference region in the reference image; The query region and the reference region are a candidate region pair in an infringement relationship; According to the set image block size, the query region is divided to obtain a plurality of query image blocks; According to the set image block size, the reference region is divided to obtain a plurality of reference image blocks; The target query image block is one of the plurality of query image blocks, and the at least one candidate reference image block is determined from the plurality of reference image blocks.

7. The method according to any one of claims 1 to 6, characterized in that, Also includes: According to the first contrast loss value and the second contrast loss value, the average contrast loss value of the corresponding image blocks between the first image and the second image is determined; Wherein, the average contrast loss value is used to optimize the parameters of the image infringement detection model; the first contrast loss value is calculated in the case of taking the first image as the query image and the second image as the reference image, and the second contrast loss value is calculated in the case of taking the second image as the query image and the first image as the reference image.

8. A model training method, comprising: Including: Obtain a training sample set, which contains a plurality of sample image pairs; Set the loss function used when training the contrast learning model; Based on the plurality of sample image pairs and the loss function, the contrast learning model is trained to obtain an image infringement detection model; Wherein, the loss function includes a composite contrast loss function, and the composite contrast loss function is constructed based on a first contrast loss function and a second contrast loss function; using the first contrast loss function, the first pixel coordinate table of one of the sample image pairs can be used to perform block-level contrast analysis on the two sample images contained in the sample image pair, and calculate the first contrast loss value between the corresponding image blocks of the two sample images. Using the second contrast loss function, the second pixel coordinate table of the other sample image in the sample image pair can be used to perform block-level contrast analysis on the two sample images contained in the sample image pair, and calculate the second contrast loss value between the corresponding image blocks of the two sample images.

9. An image infringement detection model, characterized in that, Including: A plurality of network layers; The plurality of network layers include an input layer, an encoding layer, and a loss layer; wherein The input layer is used to receive the input first image and the second image The encoding layer has two encoding modules, one of which is used to encode the first image block in the first image to output the image block feature of the first image block, and the other is used to encode the second image block in the second image to output the image block feature of the second image block; The loss layer is provided with a loss function, and the loss function comprises a composite contrast loss function; the composite contrast loss function is constructed based on a first contrast loss function and a second contrast loss function; the first contrast loss function is used for performing block-level contrast analysis on the first image and the second image based on a first pixel coordinate table of the first image and image block features output by the encoding layer, and calculating a first contrast loss value of corresponding image blocks between the first image and the second image; the second contrast loss function is used for performing block-level contrast analysis on the second image and the first image based on a second pixel coordinate table of the second image and the image block features output by the encoding layer, and calculating a second contrast loss value of corresponding image blocks between the second image and the first image; The first contrast loss value and the second contrast loss value are used for optimizing parameters of the image infringement detection model, and the image infringement detection model can be used to detect whether there is an infringement relationship between the first image and the second image.

10. An electronic device, comprising: A device comprising a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method of any one of claims 1-8.

Images