Image data infringement determination method and device, storage medium and electronic equipment

Abstract

The invention provides an image data infringement determination method and device, a storage medium and electronic equipment. Comprising the steps of obtaining a to-be-detected image, performing segmentation processing on the to-be-detected image to obtain a plurality of to-be-detected sub-images, extracting features of each to-be-detected sub-image to obtain feature information of each to-be-detected sub-image, the feature information comprising graphic feature information and text feature information; performing feature mask processing on each to-be-detected sub-image to obtain mask information of each to-be-detected sub-image, the mask information including graphic feature mask information and text feature mask information; and constructing an image database, determining the similarity between each to-be-detected sub-image and the sample sub-image according to the sample image data, the feature information of each to-be-detected sub-image and the mask information of each to-be-detected sub-image, and determining whether the to-be-detected image is infringement according to the similarity. The problem that in an infringement detection method in the prior art, the detail retrieval precision is poor due to the fact that similarity calculation is conducted after image global features are extracted is solved.

Description

Technical Field

[0001] The present application relates to the technical field of determining image data infringement, and more specifically, to an image data infringement determination method, device, computer-readable storage medium, and electronic device. Background Art

[0002] Enterprises inevitably use image promotional materials in cultural promotion and other activities. In the design process of many promotional materials, some of the images may be borrowed from the Internet. Many images with copyright notices have been circulated many times on the Internet, and the materials we see are just pictures. If some image materials are used without the authorization of the original author, it will constitute infringement. In order to avoid copyright risks in a timely manner and ensure the compliance of enterprises, the importance of copyright protection for images has become increasingly prominent.

[0003] At present, the conventional infringement detection method is to search for images and verify whether there is infringement through the search results. The usual search is to extract the global features of the image and then perform similarity calculation, and the accuracy of detail retrieval is poor. Summary of the invention

[0004] The main purpose of the present application is to provide a method, device, computer-readable storage medium and electronic device for determining infringement of image data, so as to at least solve the problem that the prior art infringement detection method extracts global features of the image and then performs similarity calculation, resulting in poor detail retrieval accuracy.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for determining infringement of image data is provided, comprising: obtaining an image to be detected, and using image segmentation technology to segment the image to be detected to obtain multiple sub-images to be detected, and extracting features of each of the sub-images to be detected to obtain feature information of each of the sub-images to be detected, wherein the feature information includes graphic feature information and text feature information; performing feature masking processing on each of the sub-images to be detected to obtain mask information of each of the sub-images to be detected, wherein the mask information includes graphic feature mask information and text feature mask information; constructing an image database, wherein the image database includes multiple sample images and sample image data corresponding to each of the sample images, the sample image data includes sample sub-images, feature information of the sample sub-images, and the mask information of the sample sub-images; determining the similarity between each of the sub-images to be detected and the sample sub-images according to the sample image data, the feature information of each of the sub-images to be detected, and the mask information of each of the sub-images to be detected, and determining whether the image to be detected is infringing according to the similarity.

[0006] Optionally, determining the similarity between each sub-image to be detected and the sample sub-image according to the sample image data, the feature information of each sub-image to be detected and the mask information of each sub-image to be detected includes: determining the similarity between each sub-image to be detected and the sample sub-image according to the Euclidean distance formula: Determine the similarity between the sub-image to be detected and the sample sub-image, where D ij is the similarity, q j is the feature information of the sub-image to be detected, is the graphic feature mask information of the sub-image to be detected, is the text feature mask information of the sub-image to be detected, k i is the characteristic information of the sample sub-image, is the graphic feature mask information of the sample sub-image, is the text feature mask information of the sample sub-image.

[0007] Optionally, the similarity between each sub-image to be detected and the sample sub-image is determined according to the sample image data, the feature information of each sub-image to be detected and the mask information of each sub-image to be detected, and whether the image to be detected is infringing is determined according to the similarity, including: using Euclidean distance to determine the similarity between each sub-image to be detected and each sample sub-image; sorting the similarities between each sub-image to be detected and each sample sub-image in descending order, selecting a preset number of sample sub-images before the similarity of each sub-image to be detected to perform original image voting calculation, and obtaining a similarity voting score between the image to be detected and the multiple sample images; selecting a target sample image whose similarity voting score is greater than a score threshold, and determining whether the image to be detected is infringing according to the target sample image.

[0008] Optionally, determining whether the image to be detected is infringing according to the target sample image includes: determining whether the target sample image contains copyright information; and determining that the image to be detected is infringing if it is determined that the target sample image contains the copyright information.

[0009] Optionally, determining the similarity between each sub-image to be detected and the sample sub-image according to the sample image data, the feature information of each sub-image to be detected and the mask information of each sub-image to be detected, and determining whether the image to be detected is infringing according to the similarity, including: when it is determined that the text feature mask information of the sub-image to be detected and the text feature mask information of the sample sub-image are both 1, obtaining the structural coding information of the sub-image to be detected and the sample sub-image, and determining the edit distance between the sub-image to be detected and the sample sub-image according to the structural coding information of the sub-image to be detected and the sample sub-image; Determine the text structure similarity between the sub-image to be detected and the sample sub-image, where r ij is the text structure similarity, sum is the structural coding information of the sub-image to be detected and the structural coding information of the sample sub-image, and ldist is the edit distance; according to Ar ij +BD ij =F ij , determine the final similarity between the sub-image to be detected and the sample sub-image, and determine whether the image to be detected is infringing according to the final similarity, where A and B are weight coefficients, A+B=1, F ij is the final similarity.

[0010] Optionally, extracting features of each of the sub-images to be detected to obtain feature information of each of the sub-images to be detected includes: identifying the text content of each of the sub-images to be detected to obtain text information of each of the sub-images to be detected; encoding keywords of the text content of each of the sub-images to be detected according to a word meaning dictionary to obtain word meaning coding information of each of the sub-images to be detected, wherein the word meaning dictionary includes multiple groups of data pairs of keywords and word meaning coding; determining the text feature information of each of the sub-images to be detected according to the text information and the word meaning coding information of each of the sub-images to be detected, and using a graphic feature extraction method to obtain the graphic feature information of each of the sub-images to be detected.

[0011] Optionally, performing feature mask processing on each of the sub-images to be detected to obtain mask information of each of the sub-images to be detected includes: according to a text feature mask formula: Determine the text feature mask information of each sub-image to be detected, wherein: is the text feature mask information, L is the feature vector length, and i is the current position subscript; according to the graphic feature mask formula: Determine the text feature mask information of each sub-image to be detected, wherein: is the graphic feature mask information.

[0012] According to another aspect of the present application, an image data infringement determination device is provided, comprising: an acquisition unit, used to acquire an image to be detected, and use image segmentation technology to segment the image to be detected to obtain multiple sub-images to be detected, and extract features of each of the sub-images to be detected to obtain feature information of each of the sub-images to be detected, wherein the feature information includes graphic feature information and text feature information; a processing unit, used to perform feature mask processing on each of the sub-images to be detected to obtain mask information of each of the sub-images to be detected, wherein the mask information includes graphic feature mask information and text feature mask information; a construction unit, used to construct an image database, wherein the image database includes multiple sample images and sample image data corresponding to each of the sample images, and the sample image data includes sample sub-images, feature information of the sample sub-images, and the mask information of the sample sub-images; a determination unit, used to determine the similarity between each of the sub-images to be detected and the sample sub-images according to the sample image data, the feature information of each of the sub-images to be detected, and the mask information of each of the sub-images to be detected, and determine whether the image to be detected is infringing according to the similarity.

[0013] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods for determining infringement of image data.

[0014] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the image data infringement determination methods.

[0015] Applying the technical solution of the present application, an image to be detected is obtained, and the image to be detected is segmented by using image segmentation technology to obtain multiple sub-images to be detected, and the features of each sub-image to be detected are extracted to obtain feature information of each sub-image to be detected, wherein the feature information includes graphic feature information and text feature information; feature masking is performed on each sub-image to be detected to obtain mask information of each sub-image to be detected, wherein the mask information includes graphic feature mask information and text feature mask information; an image database is constructed, wherein the image database includes multiple sample images and sample image data corresponding to each sample image, and the sample image data includes sample sub-images, feature information of the sample sub-images, and mask information of the sample sub-images; the similarity between each sub-image to be detected and the sample sub-image is determined based on the sample image data, the feature information of each sub-image to be detected, and the mask information of each sub-image to be detected, and whether the image to be detected is infringing is determined based on the similarity. By segmenting the image, extracting sub-image information and corresponding graphic feature information and text feature information, and calculating the similarity between the image to be detected and the sample image based on the extracted features, refined image infringement review is achieved, solving the problem of poor detail retrieval accuracy caused by the existing infringement detection methods that extract global features of the image and then perform similarity calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for executing an image data infringement determination method provided in an embodiment of the present application is shown;

[0018] Figure 2 A schematic diagram of a process of determining image data infringement according to an embodiment of the present application is shown;

[0019] Figure 3 A schematic diagram of a process of determining a specific image data infringement according to an embodiment of the present application is shown;

[0020] Figure 4 A schematic diagram of image segmentation using a panoramic segmentation method provided in accordance with an embodiment of the present application is shown;

[0021] Figure 5 A structural block diagram of an image data infringement determination device provided according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0022] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] As introduced in the background technology, the prior art infringement detection method extracts the global features of the image and then performs similarity calculation, which results in poor detail retrieval accuracy. In order to solve the problem that the prior art infringement detection method extracts the global features of the image and then performs similarity calculation, which results in poor detail retrieval accuracy, the embodiments of the present application provide an image data infringement determination method, device, computer-readable storage medium and electronic device.

[0026] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0027] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of an image data infringement determination method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0028] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the image data infringement determination method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. The transmission device 106 is used to receive or send data via a network. The specific examples of the above networks may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0029] In this embodiment, a method for determining infringement of image data running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] Figure 2 FIG. 1 is a flow chart of a method for determining image data infringement according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0031] Step S201, obtaining an image to be detected, and using image segmentation technology to segment the image to be detected to obtain multiple sub-images to be detected, and extracting features of each sub-image to be detected to obtain feature information of each sub-image to be detected, wherein the feature information includes graphic feature information and text feature information;

[0032] Specifically, the image to be detected can come from the infringement detection requirements proposed by the user, or from the automatic infringement detection request of the process; image segmentation refers to obtaining the segmentation results of independent instances in the image by using image segmentation. Image segmentation technology can adopt panoramic segmentation based on deep learning, or it can use traditional segmentation methods based on edges, thresholds, graph theory, etc.

[0033] Panoptic segmentation refers to a unified global segmentation view. Given a predetermined set of S semantic classes encoded as S = {0,...,S-1}, the task requires a panoptic segmentation algorithm to map each pixel i of the image to a pair (l i ,z i ), where l i represents the semantic class of pixel i, z i Indicates its instance id. i Pixels of the same class are grouped into different fragments. Ambiguous or out-of-class pixels can be assigned a special null label; that is, not all pixels must have a semantic label.

[0034] Step S202, performing feature mask processing on each sub-image to be detected to obtain mask information of each sub-image to be detected, wherein the mask information includes graphic feature mask information and text feature mask information;

[0035] Among them, masking is a method for extracting and describing information about specific areas in an image. By defining a mask, the areas of interest in the image can be marked and feature information can be extracted from these areas. The mask can be a binary image, where white represents the area of ​​interest and black represents the area of ​​no interest. Using the mask, the image can be locally analyzed and the features of the specific area can be extracted, thereby achieving more accurate image analysis and processing.

[0036] Step S203, constructing an image database, wherein the image database includes a plurality of sample images and sample image data corresponding to each sample image, and the sample image data includes a sample sub-image, feature information of the sample sub-image, and mask information of the sample sub-image;

[0037] Specifically, an image database is constructed, which includes a plurality of sample images, a sample sub-image corresponding to each sample image, feature information of the sample sub-image corresponding to each sample image, and mask information of the sample sub-image corresponding to each sample image.

[0038] The image database can be constructed by grabbing a large amount of sample image data from the Internet, or it can be a customized database within the enterprise. The database is constructed as an image database to be protected, and infringement review will be carried out through searching the database later.

[0039] Step S204, determining the similarity between each sub-image to be detected and the sample sub-image based on the sample image data, the feature information of each sub-image to be detected and the mask information of each sub-image to be detected, and determining whether the image to be detected is infringing based on the similarity.

[0040] Specifically, by calculating the similarity between each sub-image to be detected and each sample sub-image in the image database, it is determined whether the image to be detected is infringing, and the similar area between the image to be detected and the image that meets the infringement conditions is determined as the infringing area and marked in the result image.

[0041] Through this embodiment, an image to be detected is obtained, and the image to be detected is segmented by using image segmentation technology to obtain multiple sub-images to be detected, and the features of each sub-image to be detected are extracted to obtain feature information of each sub-image to be detected, wherein the feature information includes graphic feature information and text feature information; feature mask processing is performed on each sub-image to be detected to obtain mask information of each sub-image to be detected, wherein the mask information includes graphic feature mask information and text feature mask information; an image database is constructed, wherein the image database includes multiple sample images and sample image data corresponding to each sample image, and the sample image data includes sample sub-images, feature information of the sample sub-images, and mask information of the sample sub-images; the similarity between each sub-image to be detected and the sample sub-image is determined based on the sample image data, the feature information of each sub-image to be detected, and the mask information of each sub-image to be detected, and whether the image to be detected is infringing is determined based on the similarity. By segmenting the image, extracting sub-image information and corresponding graphic feature information and text feature information, and calculating the similarity between the image to be detected and the sample image based on the extracted features, refined image infringement review is achieved, solving the problem of poor detail retrieval accuracy caused by the existing infringement detection methods that extract global features of the image and then perform similarity calculations.

[0042] In the specific implementation process, the similarity between each sub-image to be detected and the sample sub-image is determined according to the sample image data, the feature information of each sub-image to be detected and the mask information of each sub-image to be detected, including: according to the Euclidean distance formula: Determine the similarity between the sub-image to be detected and the sample sub-image, where Dij is the similarity, q j is the feature information of the sub-image to be detected, is the graphic feature mask information of the sub-image to be detected, is the text feature mask information of the sub-image to be detected, k i is the feature information of the sample sub-image, is the graphic feature mask information of the sample sub-image, is the text feature mask information of the sample sub-image.

[0043] This method uses Euclidean distance to calculate the similarity between the sub-image to be detected and the sample sub-image. Euclidean distance is an intuitive distance calculation method with a simple calculation process and high calculation efficiency. It is suitable for large-scale similarity calculation. Therefore, it is efficient to use Euclidean distance to calculate the similarity between the sub-image to be detected and the sample sub-image.

[0044] Specifically, the similarity between each sub-image to be detected and the sample sub-image is determined based on the sample image data, the feature information of each sub-image to be detected and the mask information of each sub-image to be detected, and whether the image to be detected is infringing is determined based on the similarity, including: using Euclidean distance to determine the similarity between each sub-image to be detected and each sample sub-image; sorting the similarities between each sub-image to be detected and each sample sub-image in descending order, selecting a preset number of sample sub-images before the similarity of each sub-image to be detected to perform original image voting calculation, and obtaining a similarity voting score between the image to be detected and multiple sample images; selecting a target sample image whose similarity voting score is greater than a score threshold, and determining whether the image to be detected is infringing based on the target sample image.

[0045] After using Euclidean distance to determine the similarity between each sub-image to be detected and each sample sub-image, the method sorts the similarities between each sub-image to be detected and each sample sub-image in descending order, selects the first k sample sub-images similar to each sub-image to be detected, and uses voting calculation to calculate the score of the sample image to which the selected sample sub-image belongs. When the score of the sample image is greater than a set score threshold, it is considered that the sample image and the image to be detected are similar; wherein the number of k selected and the score threshold are specifically set according to the needs of image infringement review, for example, k can be set to 5, 8, 10, etc., and the score threshold can be set to 50, 60, etc.

[0046] More specifically, determining whether the image to be detected is infringing based on the target sample image includes: determining whether the target sample image contains copyright information; and determining that the image to be detected is infringing if it is determined that the target sample image contains copyright information.

[0047] The method determines whether the sample image has copyright information when it is determined that the sample image is similar to the image to be detected. If copyright information exists, the image to be detected is considered to be infringing; if copyright information does not exist, the image to be detected is considered to have no infringement problem.

[0048] Further, the similarity between each sub-image to be detected and the sample sub-image is determined according to the sample image data, the feature information of each sub-image to be detected and the mask information of each sub-image to be detected, and whether the image to be detected is infringing is determined according to the similarity, including: when it is determined that the text feature mask information of the sub-image to be detected and the text feature mask information of the sample sub-image are both 1, the structural coding information of the sub-image to be detected and the sample sub-image is obtained, and the editing distance between the sub-image to be detected and the sample sub-image is determined according to the structural coding information of the sub-image to be detected and the sample sub-image; Determine the text structure similarity between the sub-image to be detected and the sample sub-image, where r ij is the text structure similarity, sum is the structural coding information of the sub-image to be detected and the structural coding information of the sample sub-image, and ldist is the edit distance; according to Ar ij +BD ij =F ij , determine the final similarity between the sub-image to be detected and the sample sub-image, and determine whether the image to be detected is infringing based on the final similarity, where A and B are weight coefficients, A+B=1, F ij is the final similarity.

[0049] This method can also combine Euclidean distance and text structure similarity to determine the final similarity; the Euclidean distance is used to determine the similarity between the sub-image to be detected and each sample sub-image, and when the text mask information of the sub-image to be detected and the sample sub-image is determined to be a full 1 vector, the text structure similarity of each sample sub-image of the sub-image to be detected is calculated, and the final similarity is determined by the Euclidean distance similarity and the text structure similarity. When it is determined that the text mask information of the sub-image to be detected and the sample sub-image is not a full 1 vector, the Euclidean distance similarity is used as the final similarity. The similarity calculated in this way is more accurate, which improves the accuracy of image infringement review.

[0050] Furthermore, the features of each sub-image to be detected are extracted to obtain feature information of each sub-image to be detected, including: identifying the text content of each sub-image to be detected to obtain text information of each sub-image to be detected; encoding the keywords of the text content of each sub-image to be detected according to a word meaning dictionary to obtain word meaning coding information of each sub-image to be detected, the word meaning dictionary including multiple groups of data pairs of keywords and word meaning coding; determining the text feature information of each sub-image to be detected according to the text information and word meaning coding information of each sub-image to be detected, and using a graphic feature extraction method to obtain the graphic feature information of each sub-image to be detected.

[0051] When the semantic category of the sub-image segmentation result to be detected is text, the text processing module is used to extract the text sub-feature T of the current segmentation block. i The sub-characteristic T i It is obtained through the text processing module. The text processing module includes the text recognition module, the word meaning dictionary encoding module and the word structure encoding module. Text recognition is based on text detection, and the text content in the image is recognized and converted into text information. The main problem solved by text recognition is what each word is. Here, the recognized text content is marked as C i The word meaning dictionary encoding module extracts keywords from the text content and finds the word meaning dictionary definition corresponding to the keyword. The word meaning dictionary refers to the data pair of words and word definitions. Since the word meaning dictionary has a one-to-many situation for word interpretation, it can realize the acquisition of synonym information. This encoding module outputs the word meaning encoding information D of the text information i In addition, based on the above text information and word meaning encoding information, the semantic information of the sample image is formed. Among them, the semantic information T i' =[C i ,D i ]; Semantic features are extracted based on semantic information. Here, the text feature vector (text feature information) T is obtained by converting the semantic information into a word vector. i In addition, according to the structural coding module, the similarity between structures is obtained by structurally coding the keywords. The structural coding mainly identifies the text content C i Structural information obtained by encoding each word according to the strokes and glyph structure of the official dictionary

[0052] The sub-image to be detected in this method is extracted through the visual processing module to extract the graphic sub-feature V i, each sub-image is subjected to a feature extraction method to obtain graphic sub-feature information, and the feature is represented in the form of a matrix or vector. The method for obtaining graphic features here includes but is not limited to the method of using a convolution kernel to convolve the financial image to obtain the feature points of the sub-image, and can also be extracted by traditional digital image algorithms, such as SIFT (Scale Invariant Feature Transform), HOG (Histogram of Oriented Gradients), etc.

[0053] Specifically, feature mask processing is performed on each sub-image to be detected to obtain mask information of each sub-image to be detected, including: according to the text feature mask formula: Determine the text feature mask information of each sub-image to be detected, where: is the text feature mask information, L is the feature vector length, and i is the current position subscript; according to the graphic feature mask formula: Determine the text feature mask information of each sub-image to be detected, where: It is the graphic feature mask information.

[0054] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the image data infringement determination method of the present application will be described in detail below in conjunction with specific embodiments.

[0055] The existing technology for image infringement review first extracts the global features of the image, determines the suspected homologous images, and then segments the similar areas in the image to be detected and each suspected homologous image, and determines whether there is infringement by calculating the similar areas. This solution can quickly detect infringement, but at the same time, it first performs a preliminary search based on the overall features, which may ignore images with infringing materials in local areas. Secondly, the similar area calculation does not take into account the regional semantic information, and cannot detect text infringement.

[0056] This embodiment achieves refined review by segmenting the image, extracting sub-image information and corresponding semantic class information, and then encoding and extracting features of different semantic classes. This includes: calculating feature information of the image to be detected, determining the suspected sub-image corresponding to the image to be detected; performing voting calculations on each suspected sub-image; arranging suspected homologous images in descending order according to voting scores, and judging that the image to be detected is infringing if it meets the specified conditions, and determining the similar area between the image to be detected and the image that meets the infringement conditions as the infringing area, and marking it in the result map.

[0057] Since the above-mentioned features contain multiple modal information, have small feature granularity, faster calculation, precise positioning, and high accuracy, they can detect regional image material infringement and text infringement, and can mark suspected infringing areas and corresponding infringing materials in the review results, providing support for subsequent staff to further judge and protect their rights.

[0058] This embodiment relates to a specific method for determining infringement of image data, which is applicable to the scenario of reviewing whether corporate promotional materials are infringing, and can also be used for refined scene image retrieval, such as Figure 3 As shown, it specifically includes the following contents:

[0059] This implementation includes the establishment of an offline image database. The image database can be constructed by crawling a large amount of sample image data from the Internet, or it can be a customized database within the enterprise. The construction of this database is to serve as an image database to be protected, and infringement review will be carried out later by searching this database.

[0060] The offline image database construction phase specifically includes the following contents:

[0061] 1) Build an offline image database to be protected and calculate the features of the image to be protected. The image database pictures are called sample images in this article. The target image to be detected is called the target image.

[0062] 2) Construct an offline image database to be protected, that is, maintain the feature data of the sample images to be protected, that is, use the model processing module to obtain the sample feature K for the sample image. The model processing module includes an image segmentation module and a sub-feature extraction module.

[0063] 3) Image segmentation module refers to obtaining the segmentation results of independent instances in the image by using image segmentation. Here, panoramic segmentation based on deep learning can be used, or traditional segmentation based on edge, threshold, graph theory, etc. can be used.

[0064] 4) This embodiment takes panoptic segmentation as an example. The panorama of panoptic segmentation refers to a unified global segmentation view. Given a predetermined set of S semantic classes encoded as S = {0,...,S-1}, this task requires the panoptic segmentation algorithm to map each pixel i of the image to a pair (l i ,z i ), where l i represents the semantic class of pixel i, z i Indicates its instance id. i Pixels of the same class are grouped into different fragments. Ambiguous or out-of-class pixels can be assigned a special null label; that is, not all pixels must have a semantic label.

[0065] 5) Here, sub-images are extracted based on different semantic classes and instance IDs. Sub-image extraction can be based on the sub-images belonging to (l i ,z i ) pixels belonging to z i Instances and semantic classes i The pixels of the sub-image are 1 and the other pixels are 0 to generate a mask, and the mask is multiplied by the original image to obtain a sub-image. The edges of the sub-image are cropped until the row and column contain at least one non-zero element, as shown in the following example. Figure 4 shown.

[0066] 6) After the image is segmented, I sample sub-images are obtained, each of which has a corresponding semantic class and instance id. i ,z i ).

[0067] 7) Sample image features Get sub-feature k for sub-image i i =[V i ,T i ], that is, obtaining the visual sub-image information V i and text sub-image information T i .

[0068] 8) When the semantic category of the sub-image segmentation result is text, the text processing module extracts sub-features T from the current segmentation block. i The sub-characteristic T i Obtained through the text processing module.

[0069] a) The text processing module includes a text recognition module, a word meaning dictionary encoding module and a character structure encoding module.

[0070] b) Text recognition is to identify the text content based on text detection and convert the text content in the image into text information. The main problem to be solved by text recognition is what each text is. Here, the recognized text content is marked as C i .

[0071] c) Word meaning dictionary encoding module, after extracting keywords from the text content, finds the word meaning dictionary definition corresponding to the keyword. The word meaning dictionary refers to the data pair of words and word definitions. Since the word meaning dictionary has a one-to-many situation for word interpretation, it can realize the acquisition of synonym information. This encoding module outputs the word meaning encoding information D of the text information i .

[0072] d) Based on the above text information and word meaning encoding information, the semantic information of the sample image is formed. i' =[C i ,D i ]; semantic features are extracted based on semantic information. Here, the semantic feature vector T is obtained by converting the semantic information into word vectors. i .

[0073] e) Structural coding module, which obtains the similarity between structures by structural coding of keywords. Structural coding mainly identifies the content of text C i Structural information obtained by encoding each word according to the strokes and glyph structure of the official dictionary

[0074] 9) Among them, sub-feature V i Obtained through the visual processing module.

[0075] a) All sub-images are processed by the visual processing module to extract sub-features V i ,Each sub-image is subjected to feature extraction method to obtain visual sub-feature information, which is represented in matrix or vector form.

[0076] b) The method for obtaining visual features here includes but is not limited to the method of obtaining feature points of sub-images by convolving the financial image with a convolution kernel. Feature extraction can also be performed through traditional digital image algorithms, such as SIFT (Scale Invariant Feature Transform) and HOG (Histogram of Oriented Gradients).

[0077] 10) Get mask information and

[0078] Among them, the text feature mask is:

[0079] Among them, the visual (graphic) feature mask is:

[0080] 11) After the above processing, the corresponding feature of each sub-image is expressed as: The feature of sub-image i is: Among them, other means i does not contain text, V i ∈R L , T i ∈R L Sample image features Based on the above calculations, the sample image database can be constructed.

[0081] The target image detection stage includes the following contents:

[0082] 1) Obtain the target image to be detected. The target image can come from the infringement detection requirements proposed by the user or from the automatic infringement detection request of the process.

[0083] 2) The system responds to the infringement detection request and calculates the target image features.

[0084] 3) The target image feature calculation uses the processing module in the model to obtain the target image feature information Where J represents the number of sub-images of the target image, V represents the visual sub-feature information and T represents the text sub-feature information. The visual sub-feature is represented by a vector Text sub-features are represented by vectors

[0085] 4) Obtain the mask information of each sub-image to be detected and

[0086] The text feature mask is:

[0087] The visual feature mask:

[0088] 5) Calculate the similarity between each sub-feature of the target image and the sub-element features of the image database. Here, the Euclidean distance is used to measure the similarity. The calculation formula is: in

[0089] 6) At the same time, if and When all are 1 vectors, the sample sub-image text structure information is calculated and the text structure information of the sub-image to be detected The edit distance ldist is used to measure the structural similarity of two text sub-images, using the formula Where sum means and The sum of the lengths of the strings, and ldist is the class edit distance.

[0090] 7) Obtain the top k sample sub-images of similarity of each sub-image to be detected, and perform voting calculation on the original images to which the k×J sub-images belong, and sort the original images in descending order of scores to obtain similar image information, where J is the total number of sub-images to be detected.

[0091] 8) When the voting score of the sample image is greater than the preset threshold, where the threshold setting can be defined according to different databases, it is considered that the target image and the sample image in the database image are similar. The system returns the result image and marks the similar area in the image result for evidence collection and next step operation.

[0092] 9) If it is identified that any image block in the sample image contains copyright information, then whether the image to be identified is an infringing image is determined based on the image block containing the copyright information. This can not only reduce the amount of calculation and improve the recognition efficiency, but also perform targeted image block infringement identification and improve the recognition accuracy.

[0093] This embodiment proposes a segmentation-based refined image feature matching scheme, which first performs image segmentation and then extracts sub-features, and then matches the sub-features to achieve refined infringement review; and proposes a mask-based adaptive cross-modal similarity calculation method to achieve mutual retrieval between images and images and texts; in addition, a retrieval voting strategy is proposed, which obtains the final retrieval ranking after voting on the intermediate retrieval results; and a multimodal feature extraction method with different semantics is proposed to achieve text infringement and image infringement detection.

[0094] This embodiment achieves refined review by segmenting the image, extracting sub-image information and corresponding semantic class information, and then encoding and extracting features for different semantic classes. The method includes: calculating feature information of the image to be detected, determining the suspected sub-image corresponding to the image to be detected; performing voting calculations on each suspected sub-image; arranging suspected homologous images in descending order according to the voting scores, and judging that the image to be detected is infringing if it meets the specified conditions, and determining the similar area between the image to be detected and the image that meets the infringement conditions as the infringement area, and marking it in the result map, thereby achieving refined infringement review of image data. This solves the problem that the prior art infringement detection method extracts global features of the image and then performs similarity calculations, resulting in poor detail retrieval accuracy.

[0095] The embodiments of the present application also provide an image data infringement determination device. It should be noted that the image data infringement determination device of the embodiments of the present application can be used to execute the image data infringement determination method provided by the embodiments of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation in hardware, or a combination of software and hardware, is also possible and conceivable.

[0096] The following is an introduction to the image data infringement determination device provided in an embodiment of the present application.

[0097] Figure 5 Schematic diagram of an image data infringement determination device according to an embodiment of the present application. Figure 5 As shown, the device includes: an acquisition unit 51, a processing unit 52, a construction unit 53, and a determination unit 54.

[0098] The acquisition unit 51 is used to acquire the image to be detected, and use the image segmentation technology to segment the image to be detected to obtain multiple sub-images to be detected, and extract the features of each sub-image to be detected to obtain feature information of each sub-image to be detected, wherein the feature information includes graphic feature information and text feature information;

[0099] Specifically, the image to be detected can come from the infringement detection requirements proposed by the user, or from the automatic infringement detection request of the process; image segmentation refers to obtaining the segmentation results of independent instances in the image by using image segmentation. Image segmentation technology can adopt panoramic segmentation based on deep learning, or it can use traditional segmentation methods based on edges, thresholds, graph theory, etc.

[0100] The processing unit 52 is used to perform feature mask processing on each sub-image to be detected to obtain mask information of each sub-image to be detected, wherein the mask information includes graphic feature mask information and text feature mask information;

[0101] Specifically, masking is a method for extracting and describing information about specific areas in an image. By defining a mask, the areas of interest in the image can be marked and feature information can be extracted from these areas. The mask can be a binary image, where white represents the areas of interest and black represents the areas of no interest. Using masks, images can be locally analyzed and features of specific areas can be extracted, thereby achieving more accurate image analysis and processing.

[0102] A construction unit 53, configured to construct an image database, wherein the image database includes a plurality of sample images and sample image data corresponding to each sample image, and the sample image data includes a sample sub-image, feature information of the sample sub-image, and mask information of the sample sub-image;

[0103] The image database can be constructed by grabbing a large amount of sample image data from the Internet, or it can be a customized database within the enterprise. The database is constructed as an image database to be protected, and infringement review will be carried out through searching the database later.

[0104] The determination unit 54 is used to determine the similarity between each sub-image to be detected and the sample sub-image according to the sample image data, the feature information of each sub-image to be detected and the mask information of each sub-image to be detected, and determine whether the image to be detected is infringing according to the similarity.

[0105] Specifically, by calculating the similarity between each sub-image to be detected and each sample sub-image in the image database, it is determined whether the image to be detected is infringing, and the similar area between the image to be detected and the image that meets the infringement conditions is determined as the infringing area and marked in the result image.

[0106] In this embodiment, the acquisition unit is used to acquire the image to be detected, and use image segmentation technology to segment the image to be detected to obtain multiple sub-images to be detected, and extract the features of each sub-image to be detected to obtain feature information of each sub-image to be detected, wherein the feature information includes graphic feature information and text feature information; the processing unit is used to perform feature mask processing on each sub-image to be detected to obtain mask information of each sub-image to be detected, wherein the mask information includes graphic feature mask information and text feature mask information; the construction unit is used to construct an image database, wherein the image database includes multiple sample images and sample image data corresponding to each sample image, and the sample image data includes sample sub-images, feature information of the sample sub-images, and mask information of the sample sub-images; the determination unit is used to determine the similarity between each sub-image to be detected and the sample sub-image according to the sample image data, the feature information of each sub-image to be detected and the mask information of each sub-image to be detected, and determine whether the image to be detected is infringing according to the similarity. By segmenting the image, extracting sub-image information and corresponding graphic feature information and text feature information, and calculating the similarity between the image to be detected and the sample image based on the extracted features, refined image infringement review is achieved, solving the problem of poor detail retrieval accuracy caused by the existing infringement detection methods that extract global features of the image and then perform similarity calculations.

[0107] As an optional solution, the determining unit includes a first determining module, which is used to determine the distance according to the Euclidean distance formula: Determine the similarity between the sub-image to be detected and the sample sub-image, where D ij is the similarity, q j is the feature information of the sub-image to be detected, is the graphic feature mask information of the sub-image to be detected, is the text feature mask information of the sub-image to be detected, k i is the feature information of the sample sub-image, is the graphic feature mask information of the sample sub-image, is the text feature mask information of the sample sub-image.

[0108] Specifically, the Euclidean distance is used to calculate the similarity between the sub-image to be detected and the sample sub-image. The Euclidean distance is an intuitive distance calculation method with a simple calculation process and high calculation efficiency. It is suitable for large-scale similarity calculation. Therefore, the Euclidean distance is used to calculate the similarity between the sub-image to be detected and the sample sub-image with high efficiency.

[0109] In an optional solution, the determination unit also includes a second determination module, a voting calculation module, and a third determination module; the second determination module is used to determine the similarity between each sub-image to be detected and each sample sub-image using Euclidean distance; the voting calculation module is used to sort the similarities between each sub-image to be detected and each sample sub-image in descending order, select a preset number of sample sub-images before the similarity of each sub-image to be detected to perform original image voting calculation, and obtain similarity voting scores between the image to be detected and multiple sample images; the third determination module is used to select a target sample image with a similarity voting score greater than a score threshold, and determine whether the image to be detected is infringing based on the target sample image.

[0110] Specifically, after using the Euclidean distance to determine the similarity between each sub-image to be detected and each sample sub-image, the similarities between each sub-image to be detected and each sample sub-image are sorted in descending order, and the top k sample sub-images that are similar to each sub-image to be detected are selected. The scores of the sample images to which the selected sample sub-images belong are calculated by voting. When the score of the sample image is greater than the set score threshold, the sample image is considered to be similar to the image to be detected.

[0111] In an optional solution, the third determination module includes a first determination submodule and a second determination submodule; the first determination submodule is used to determine whether the target sample image contains copyright information; the second determination submodule is used to determine whether the image to be detected is infringing when it is determined that the target sample image contains copyright information.

[0112] Specifically, when it is determined that the sample image and the image to be detected are similar, it is determined whether the sample image has copyright information. If copyright information exists, the image to be detected is considered to be infringing; if copyright information does not exist, the image to be detected is considered to have no infringement issue.

[0113] In an optional solution, the determination unit includes an acquisition module, a fourth determination module, and a fifth determination module; the acquisition module is used to acquire the structural coding information of the sub-image to be detected and the sample sub-image when it is determined that the text feature mask information of the sub-image to be detected and the text feature mask information of the sample sub-image are both 1, and determine the edit distance between the sub-image to be detected and the sample sub-image according to the structural coding information of the sub-image to be detected and the sample sub-image; the fourth determination module is used to determine the edit distance between the sub-image to be detected and the sample sub-image according to the text feature mask information of the sub-image to be detected and the sample sub-image; Determine the text structure similarity between the sub-image to be detected and the sample sub-image, where r ij is the text structure similarity, sum is the structural coding information of the sub-image to be detected and the structural coding information of the sample sub-image, and ldist is the edit distance; the fifth determination module is used to determine the text structure similarity according to Ar ij +BD ij =F ij, determine the final similarity between the sub-image to be detected and the sample sub-image, and determine whether the image to be detected is infringing based on the final similarity, where A and B are weight coefficients, A+B=1, F ij is the final similarity.

[0114] Specifically, the final similarity can also be determined by combining the Euclidean distance and the text structure similarity; the Euclidean distance is used to determine the similarity between the sub-image to be detected and each sample sub-image, and at the same time, when it is determined that the text mask information of the sub-image to be detected and the sample sub-image is a full 1 vector, the text structure similarity of each sample sub-image of the sub-image to be detected is calculated, and the final similarity is jointly determined by the Euclidean distance similarity and the text structure similarity. When it is determined that the text mask information of the sub-image to be detected and the sample sub-image is not a full 1 vector, the Euclidean distance similarity is used as the final similarity. The similarity calculated in this way is more accurate, which improves the accuracy of image infringement review.

[0115] An optional solution, the acquisition unit includes a recognition module, a coding module and a sixth determination module; the recognition module is used to recognize the text content of each sub-image to be detected to obtain the text information of each sub-image to be detected; the coding module is used to encode the keywords of the text content of each sub-image to be detected according to a word meaning dictionary to obtain the word meaning coding information of each sub-image to be detected, and the word meaning dictionary includes multiple groups of data pairs of keywords and word meaning codes; the sixth determination module is used to determine the text feature information of each sub-image to be detected according to the text information and word meaning coding information of each sub-image to be detected, and adopt a graphic feature extraction method to obtain the graphic feature information of each sub-image to be detected.

[0116] In an optional solution, the processing unit includes a seventh determination module and an eighth determination module; the seventh determination module is used to determine the text feature according to the mask formula: Determine the text feature mask information of each sub-image to be detected, where: is the text feature mask information, L is the feature vector length, and i is the current position subscript; the eighth determination module is used according to the graphic feature mask formula: Determine the text feature mask information of each sub-image to be detected, where: It is the graphic feature mask information.

[0117] The image data infringement determination device includes a processor and a memory. The acquisition unit, processing unit, construction unit, determination unit, etc. are all stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.

[0118] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters can be adjusted to solve the problem that the existing infringement detection method extracts the global features of the image and then performs similarity calculation, resulting in poor detail retrieval accuracy.

[0119] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0120] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the image data infringement determination method.

[0121] Specifically, the method for determining image data infringement includes:

[0122] Step S201, obtaining an image to be detected, and using image segmentation technology to segment the image to be detected to obtain multiple sub-images to be detected, and extracting features of each sub-image to be detected to obtain feature information of each sub-image to be detected, wherein the feature information includes graphic feature information and text feature information;

[0123] Step S202, performing feature mask processing on each sub-image to be detected to obtain mask information of each sub-image to be detected, wherein the mask information includes graphic feature mask information and text feature mask information;

[0124] Step S203, constructing an image database, wherein the image database includes a plurality of sample images and sample image data corresponding to each sample image, and the sample image data includes a sample sub-image, feature information of the sample sub-image, and mask information of the sample sub-image;

[0125] Step S204, determining the similarity between each sub-image to be detected and the sample sub-image based on the sample image data, the feature information of each sub-image to be detected and the mask information of each sub-image to be detected, and determining whether the image to be detected is infringing based on the similarity.

[0126] An embodiment of the present invention provides a processor, which is used to run a program, wherein the image data infringement determination method is executed when the program is running.

[0127] Specifically, the method for determining image data infringement includes:

[0128] Step S201, obtaining an image to be detected, and using image segmentation technology to segment the image to be detected to obtain multiple sub-images to be detected, and extracting features of each sub-image to be detected to obtain feature information of each sub-image to be detected, wherein the feature information includes graphic feature information and text feature information;

[0129] Step S202, performing feature mask processing on each sub-image to be detected to obtain mask information of each sub-image to be detected, wherein the mask information includes graphic feature mask information and text feature mask information;

[0130] Step S203, constructing an image database, wherein the image database includes a plurality of sample images and sample image data corresponding to each sample image, and the sample image data includes a sample sub-image, feature information of the sample sub-image, and mask information of the sample sub-image;

[0131] Step S204, determining the similarity between each sub-image to be detected and the sample sub-image based on the sample image data, the feature information of each sub-image to be detected and the mask information of each sub-image to be detected, and determining whether the image to be detected is infringing based on the similarity.

[0132] An embodiment of the present invention provides an electronic device, the electronic device comprising a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the following steps are implemented:

[0133] Step S201, obtaining an image to be detected, and using image segmentation technology to segment the image to be detected to obtain multiple sub-images to be detected, and extracting features of each sub-image to be detected to obtain feature information of each sub-image to be detected, wherein the feature information includes graphic feature information and text feature information;

[0134] Step S202, performing feature mask processing on each sub-image to be detected to obtain mask information of each sub-image to be detected, wherein the mask information includes graphic feature mask information and text feature mask information;

[0135] Step S203, constructing an image database, wherein the image database includes a plurality of sample images and sample image data corresponding to each sample image, and the sample image data includes a sample sub-image, feature information of the sample sub-image, and mask information of the sample sub-image;

[0136] Step S204, determining the similarity between each sub-image to be detected and the sample sub-image based on the sample image data, the feature information of each sub-image to be detected and the mask information of each sub-image to be detected, and determining whether the image to be detected is infringing based on the similarity.

[0137] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0138] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:

[0139] Step S201, obtaining an image to be detected, and using image segmentation technology to segment the image to be detected to obtain multiple sub-images to be detected, and extracting features of each sub-image to be detected to obtain feature information of each sub-image to be detected, wherein the feature information includes graphic feature information and text feature information;

[0140] Step S202, performing feature mask processing on each sub-image to be detected to obtain mask information of each sub-image to be detected, wherein the mask information includes graphic feature mask information and text feature mask information;

[0141] Step S203, constructing an image database, wherein the image database includes a plurality of sample images and sample image data corresponding to each sample image, and the sample image data includes a sample sub-image, feature information of the sample sub-image, and mask information of the sample sub-image;

[0142] Step S204, determining the similarity between each sub-image to be detected and the sample sub-image based on the sample image data, the feature information of each sub-image to be detected and the mask information of each sub-image to be detected, and determining whether the image to be detected is infringing based on the similarity.

[0143] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0144] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0145] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0146] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0147] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0148] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0149] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0150] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0151] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0152] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0153] 1) A method for determining infringement of image data of the present application includes: obtaining an image to be detected, and using image segmentation technology to segment the image to be detected to obtain multiple sub-images to be detected, and extracting features of each sub-image to be detected to obtain feature information of each sub-image to be detected, wherein the feature information includes graphic feature information and text feature information; performing feature mask processing on each sub-image to be detected to obtain mask information of each sub-image to be detected, wherein the mask information includes graphic feature mask information and text feature mask information; constructing an image database, wherein the image database includes multiple sample images and sample image data corresponding to each sample image, the sample image data includes sample sub-images, feature information of the sample sub-images, and mask information of the sample sub-images; determining the similarity between each sub-image to be detected and the sample sub-image based on the sample image data, the feature information of each sub-image to be detected and the mask information of each sub-image to be detected, and determining whether the image to be detected is infringing based on the similarity. By segmenting the image, extracting sub-image information and corresponding graphic feature information and text feature information, and calculating the similarity between the image to be detected and the sample image based on the extracted features, refined image infringement review is achieved, solving the problem of poor detail retrieval accuracy caused by the existing infringement detection methods that extract global features of the image and then perform similarity calculations.

[0154] 2) An image data infringement determination device of the present application includes: an acquisition unit, which is used to acquire an image to be detected, and use image segmentation technology to segment the image to be detected to obtain multiple sub-images to be detected, and extract the features of each sub-image to be detected to obtain feature information of each sub-image to be detected, wherein the feature information includes graphic feature information and text feature information; a processing unit, which is used to perform feature mask processing on each sub-image to be detected to obtain mask information of each sub-image to be detected, wherein the mask information includes graphic feature mask information and text feature mask information; a construction unit, which is used to construct an image database, wherein the image database includes multiple sample images and sample image data corresponding to each sample image, and the sample image data includes sample sub-images, feature information of the sample sub-images, and mask information of the sample sub-images; a determination unit, which is used to determine the similarity between each sub-image to be detected and the sample sub-image according to the sample image data, the feature information of each sub-image to be detected and the mask information of each sub-image to be detected, and determine whether the image to be detected is infringing according to the similarity. By segmenting the image, extracting sub-image information and corresponding graphic feature information and text feature information, and calculating the similarity between the image to be detected and the sample image based on the extracted features, refined image infringement review is achieved, solving the problem of poor detail retrieval accuracy caused by the existing infringement detection methods that extract global features of the image and then perform similarity calculations.

[0155] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining infringement of image data, characterized in that: include: Acquire an image to be detected, and use image segmentation technology to segment the image to be detected to obtain multiple sub-images to be detected, and extract features of each of the sub-images to be detected to obtain feature information of each of the sub-images to be detected, wherein the feature information includes graphic feature information and text feature information; Performing feature mask processing on each of the sub-images to be detected to obtain mask information of each of the sub-images to be detected, wherein the mask information includes graphic feature mask information and text feature mask information; Constructing an image database, wherein the image database includes a plurality of sample images and sample image data corresponding to each of the sample images, the sample image data including a sample sub-image, feature information of the sample sub-image, and the mask information of the sample sub-image; The similarity between each sub-image to be detected and the sample sub-image is determined based on the sample image data, the feature information of each sub-image to be detected and the mask information of each sub-image to be detected, and whether the image to be detected is infringing is determined based on the similarity.

2. The method according to claim 1, characterized in that Determining the similarity between each sub-image to be detected and the sample sub-image according to the sample image data, the feature information of each sub-image to be detected and the mask information of each sub-image to be detected, including: According to the Euclidean distance formula: Determine the similarity between the sub-image to be detected and the sample sub-image, where D ij is the similarity, q j is the feature information of the sub-image to be detected, is the graphic feature mask information of the sub-image to be detected, is the text feature mask information of the sub-image to be detected, k i is the characteristic information of the sample sub-image, is the graphic feature mask information of the sample sub-image, is the text feature mask information of the sample sub-image.

3. The method according to claim 1, characterized in that Determining the similarity between each sub-image to be detected and the sample sub-image according to the sample image data, the feature information of each sub-image to be detected and the mask information of each sub-image to be detected, and determining whether the image to be detected is infringing according to the similarity, including: Using Euclidean distance to determine the similarity between each of the sub-images to be detected and each of the sample sub-images; Sorting the similarities between each sub-image to be detected and each sample sub-image in descending order, selecting a preset number of sample sub-images before the similarity of each sub-image to be detected to perform original image voting calculation, and obtaining similarity voting scores between the image to be detected and the plurality of sample images; A target sample image whose similarity voting score is greater than a score threshold is selected, and whether the image to be detected is infringing is determined based on the target sample image.

4. The method according to claim 3, characterized in that Determining whether the image to be detected is infringing according to the target sample image includes: Determining whether the target sample image contains copyright information; In the case where it is determined that the target sample image contains the copyright information, it is determined that the image to be detected is infringing.

5. The method according to claim 2, characterized in that: Determining the similarity between each sub-image to be detected and the sample sub-image according to the sample image data, the feature information of each sub-image to be detected and the mask information of each sub-image to be detected, and determining whether the image to be detected is infringing according to the similarity, including: When it is determined that the text feature mask information of the sub-image to be detected and the text feature mask information of the sample sub-image are both 1, obtaining structural coding information of the sub-image to be detected and the sample sub-image, and determining the edit distance between the sub-image to be detected and the sample sub-image according to the structural coding information of the sub-image to be detected and the sample sub-image; according to Determine the text structure similarity between the sub-image to be detected and the sample sub-image, where r ij is the text structure similarity, sum is the structural coding information of the sub-image to be detected and the structural coding information of the sample sub-image, and ldist is the edit distance; According to Ar ij +BD ij =F ij , determine the final similarity between the sub-image to be detected and the sample sub-image, and determine whether the image to be detected is infringing according to the final similarity, where A and B are weight coefficients, A+B=1, F ij is the final similarity.

6. The method according to claim 1, characterized in that Extracting the features of each of the sub-images to be detected to obtain feature information of each of the sub-images to be detected includes: Recognizing the text content of each of the sub-images to be detected to obtain text information of each of the sub-images to be detected; Encoding the keywords of the text content of each sub-image to be detected according to a word meaning dictionary to obtain word meaning coding information of each sub-image to be detected, wherein the word meaning dictionary includes a plurality of data pairs of keywords and word meaning codes; The text feature information of each sub-image to be detected is determined according to the text information and the word meaning coding information of each sub-image to be detected, and the graphic feature information of each sub-image to be detected is acquired by using a graphic feature extraction method.

7. The method according to claim 1, characterized in that Performing feature mask processing on each of the sub-images to be detected to obtain mask information of each of the sub-images to be detected, including: According to the text feature mask formula: Determine the text feature mask information of each sub-image to be detected, wherein: is the text feature mask information, L is the feature vector length, and i is the current position subscript; According to the graphic feature mask formula: Determine the text feature mask information of each sub-image to be detected, wherein: is the graphic feature mask information.

8. An image data infringement determination device, characterized in that: include: an acquisition unit, used to acquire an image to be detected, and segment the image to be detected using an image segmentation technique to obtain a plurality of sub-images to be detected, and extract features of each of the sub-images to be detected to obtain feature information of each of the sub-images to be detected, wherein the feature information includes graphic feature information and text feature information; A processing unit, configured to perform feature mask processing on each of the sub-images to be detected to obtain mask information of each of the sub-images to be detected, wherein the mask information includes graphic feature mask information and text feature mask information; A construction unit, configured to construct an image database, wherein the image database includes a plurality of sample images and sample image data corresponding to each of the sample images, the sample image data including a sample sub-image, feature information of the sample sub-image, and the mask information of the sample sub-image; A determination unit is used to determine the similarity between each sub-image to be detected and the sample sub-image based on the sample image data, the feature information of each sub-image to be detected and the mask information of each sub-image to be detected, and determine whether the image to be detected is infringing based on the similarity.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for determining infringement of image data according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the image data infringement determination method described in any one of claims 1 to 7.

Images