Image processing method and device, computer readable medium and electronic equipment
By detecting the object features and position relationships in the image and combining the object relationship vector for image repetition detection, the problem of poor deduplication effect caused by loss of position information in the prior art is solved, and more accurate image repetition detection is achieved.
Patent Information
- Application Number
- CN202310546270.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-15
- Publication Date
- 2024-11-15
AI Technical Summary
In the process of image deduplication, the position information is lost due to feature pooling in deep learning methods, and it is difficult to distinguish images with different role positions, resulting in poor deduplication effect.
By detecting objects in the image, the object feature representation vector and object relationship vector are generated, the image repeat detection is performed based on object features and position relationships, and the object relationship extraction model is used for training to improve detection accuracy.
It improves the accuracy of image repetition detection, can more accurately represent image features, distinguish images with different role positions, thereby improving the deduplication effect.
Smart Images

Figure CN118968103A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer and communication technology, and in particular to an image processing method, device, computer-readable medium and electronic device. Background Art
[0002] In the image deduplication task, it is usually necessary to deduplicate images with the same background but different character relationships or different character actions. The deduplication solution based on deep learning image embedding representation proposed in the related art uses the average pooling method to compress the three-dimensional features with position information into one-dimensional features that are independent of position after extracting deep features, which loses the relationship between objects in the image, resulting in poor image deduplication effect. For example, in the game screen, the map is the same, the characters participating in the battle are also the same, but the relative positions of the characters are different, it will be difficult to distinguish whether these images are duplicate images through the solution proposed in the related art. Summary of the invention
[0003] The embodiments of the present application provide an image processing method, device, computer-readable medium and electronic device, which can improve the accuracy of image duplication detection.
[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.
[0005] According to one aspect of an embodiment of the present application, there is provided an image processing method, comprising: detecting each object contained in an image to be processed to obtain an object detection result; generating a feature representation vector of each object based on the object detection result; extracting a positional relationship between each object contained in the image to be processed based on the feature representation vector of each object to obtain an object relationship vector; and detecting whether the image to be processed is a duplicate of a template image in an image library based on the feature representation vector and the object relationship vector.
[0006] According to one aspect of an embodiment of the present application, there is provided an image processing device, comprising: a detection unit, configured to detect each object contained in an image to be processed and obtain an object detection result; a generation unit, configured to generate a feature representation vector of each object according to the object detection result; an extraction unit, configured to extract the positional relationship between each object contained in the image to be processed according to the feature representation vector of each object and obtain an object relationship vector; and a processing unit, configured to detect whether the image to be processed is a duplicate of a template image in an image library according to the feature representation vector and the object relationship vector.
[0007] In some embodiments of the present application, based on the aforementioned scheme, the extraction unit is configured to: perform quantization processing on each object according to the feature representation vector of each object to obtain a quantization value of each object; generate a quantization map corresponding to the image to be processed according to the position of each object in the image to be processed and the quantization value of each object; and extract the object relationship vector from the quantization map.
[0008] In some embodiments of the present application, based on the aforementioned scheme, the extraction unit performs quantization processing on each object according to the feature representation vector of each object to obtain a quantization value of each object, including: obtaining multiple cluster centers, where the cluster centers are obtained by clustering the feature representation vectors of the objects contained in the template image; determining the cluster center closest to the feature representation vector of each object according to the distance between the feature representation vector of each object and the multiple cluster centers; and generating the quantization value of each object according to the identification value corresponding to the cluster center closest to the feature representation vector of each object.
[0009] In some embodiments of the present application, based on the aforementioned scheme, the image processing device also includes: a clustering unit, configured to obtain the number of known objects and the number of associated objects of the known objects, and obtain the feature representation vector of the objects contained in the template image; determine the number of cluster centers according to the number of known objects and the number of associated objects; and perform clustering processing on the feature representation vectors of the objects contained in the template image according to the number of cluster centers to obtain the multiple cluster centers.
[0010] In some embodiments of the present application, based on the aforementioned scheme, the extraction unit generates a quantization map corresponding to the image to be processed according to the positions of the objects in the image to be processed and the quantization values of the objects, including: replacing the pixel values at the positions of the objects with the quantization values of the objects according to the positions of the objects in the image to be processed; replacing the pixel values in the image to be processed except for the positions of the objects with set values to generate a quantization map corresponding to the image to be processed.
[0011] In some embodiments of the present application, based on the aforementioned scheme, the extraction unit extracts the object relationship vector from the quantization map, including: inputting the quantization map into an object relationship extraction model, the object relationship extraction model includes multiple convolutional layers connected in sequence, and the quantization map is input to the first convolutional layer among the multiple convolutional layers; merging the outputs of at least two of the multiple convolutional layers after pooling processing to obtain the object relationship vector.
[0012] In some embodiments of the present application, based on the aforementioned scheme, the image processing device also includes: an acquisition unit, configured to acquire a triplet for training the object relationship extraction model, the triplet including an anchor image, a positive sample image and a negative sample image; wherein the anchor image, the positive sample image and the negative sample image are all quantized images; an input unit, configured to input the triplet into the object relationship extraction model to obtain a first embedding vector for the anchor image, a second embedding vector for the negative sample image and a third embedding vector for the positive sample image output by the object relationship extraction model; a calculation unit, configured to calculate the difference between the first distance and the second distance based on the first distance between the first embedding vector and the second embedding vector, and the second distance between the first embedding vector and the third embedding vector; an adjustment unit, configured to adjust the model parameters of the object relationship extraction model with the difference being greater than or equal to a set value as a training target.
[0013] In some embodiments of the present application, based on the aforementioned scheme, the adjustment unit is configured to: generate a loss value corresponding to the triplet according to the difference and the set value; add the loss values corresponding to a set number of triplets as the total loss for a round of training; and adjust the model parameters of the object relationship extraction model based on the total loss.
[0014] In some embodiments of the present application, based on the aforementioned scheme, the acquisition unit is configured to: acquire multiple quantization maps and object record information corresponding to each quantization map, wherein the object record information is obtained by arranging the quantization values of the objects contained in the quantization map in a set order; select a target quantization map from the multiple quantization maps, and according to the target object record information corresponding to the target quantization map, search from the multiple quantization maps for a first quantization map whose corresponding object record information is the same as the target object record information or contains the target object record information; use the target quantization map as an anchor image, use the second quantization map as a positive sample image corresponding to the target quantization map, and use other quantization maps among the multiple quantization maps except the target quantization map and the first quantization map as negative sample images corresponding to the target quantization map, to generate the triplet.
[0015] In some embodiments of the present application, based on the aforementioned scheme, the acquisition unit is configured to: acquire a selected anchor image; and perform data enhancement processing on the anchor image without changing the object relationship in the anchor image to obtain a positive sample image corresponding to the anchor image.
[0016] In some embodiments of the present application, based on the above solution, when the object relationship in the anchor image does not change, data enhancement processing is performed on the anchor image, including at least one of the following methods:
[0017] Performing overall movement processing on the anchor image or all objects contained in the anchor image;
[0018] Performing reduction or enlargement processing on the anchor point image;
[0019] Performing reduction or enlargement processing on a designated object in the anchor image;
[0020] The anchor image is cropped on the basis of retaining the object contained in the anchor image.
[0021] In some embodiments of the present application, based on the aforementioned scheme, the acquisition unit is configured to: acquire a selected anchor image; change at least one of an object and an object relationship in the anchor image to obtain a negative sample image corresponding to the anchor image.
[0022] In some embodiments of the present application, based on the above scheme, changing at least one of the object and the object relationship in the anchor image to obtain a negative sample image corresponding to the anchor image includes at least one of the following methods:
[0023] Performing movement processing on some objects in the anchor point image;
[0024] randomly replacing at least one object in the anchor image;
[0025] Replace other objects in the anchor image that are different from the specified object with the specified object in the similar image of the anchor image;
[0026] swapping positions of at least two objects in the anchor image;
[0027] The position of the designated object in the anchor image is adjusted to a set position.
[0028] In some embodiments of the present application, based on the aforementioned scheme, the processing unit is configured as follows: if the feature representation vector matches the feature representation vector corresponding to the specified template image, and the object relationship vector matches the object relationship vector corresponding to the specified template image, then it is determined that the image to be processed is a duplicate of the specified template image.
[0029] In some embodiments of the present application, based on the aforementioned scheme, the processing unit is configured as follows: if the feature representation vector does not match the feature representation vector corresponding to the specified template image, or the object relationship vector does not match the object relationship vector corresponding to the specified template image, then it is determined that the image to be processed does not overlap with the specified template image.
[0030] In some embodiments of the present application, based on the aforementioned scheme, the processing unit is further configured as follows: if it is detected that a template image needs to be stored in the image library, then each object contained in the template image is detected to obtain an object detection result corresponding to the template image; a feature representation vector of each object in the template image is generated according to the object detection result corresponding to the template image to obtain a feature representation vector corresponding to the template image; the positional relationship between each object contained in the template image is extracted according to the feature representation vector corresponding to each object in the template image to obtain an object relationship vector corresponding to the template image; and the template image is associated with the feature representation vector and the object relationship vector corresponding to the template image and stored in the image library.
[0031] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image processing method as described in the above embodiment is implemented.
[0032] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more computer programs, wherein when the one or more computer programs are executed by the one or more processors, the electronic device implements the image processing method as described in the above embodiments.
[0033] According to one aspect of the embodiments of the present application, a computer program product is provided, the computer program product comprising a computer program stored in a computer-readable storage medium. A processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the image processing method provided in the above various optional embodiments.
[0034] In the technical solutions provided in some embodiments of the present application, a feature representation vector of each object is generated according to the object detection result, and then the positional relationship between each object contained in the image to be processed is extracted according to the feature representation vector of each object to obtain an object relationship vector. Then, based on the feature representation vector of each object and the object relationship vector, it is detected whether the image to be processed and the template image in the image library are repeated. When extracting image features for duplication detection, on the one hand, the feature representation vector of each object contained in the image is taken into account, and on the other hand, the positional relationship between each object can also be taken into account. Then, the feature representation vector of the object and the object relationship vector can be used to more accurately characterize the features of the image, which is beneficial to improving the accuracy of image duplication detection.
[0035] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 A flow chart of a method for extracting image features is shown;
[0037] Figure 2 A flow chart of a method for extracting image features is shown;
[0038] Figure 3 A schematic diagram showing an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied;
[0039] Figure 4 A flowchart of an image processing method according to an embodiment of the present application is shown;
[0040] Figure 5 A flowchart of an image processing method according to an embodiment of the present application is shown;
[0041] Figure 6 A schematic diagram showing the comparison relationship between an image and a quantization graph according to an embodiment of the present application is shown;
[0042] Figure 7 A structural diagram of a relation extraction model according to an embodiment of the present application is shown;
[0043] Figure 8 A schematic diagram of the training process of an object relationship extraction model according to an embodiment of the present application is shown;
[0044] Fig. 9 A schematic diagram of an operation interface related to image processing according to an embodiment of the present application is shown;
[0045] Fig.10 A flowchart of an image processing method according to an embodiment of the present application is shown;
[0046] Fig.11 A flowchart of an image processing method according to an embodiment of the present application is shown;
[0047] Fig.12 A comparison diagram of an example action state before and after a change according to an embodiment of the present application is shown;
[0048] Fig.13 A schematic diagram showing the comparison relationship between an image and a quantization graph according to an embodiment of the present application is shown;
[0049] Fig.14 A schematic diagram showing the comparison relationship between quantization images before and after image translation according to an embodiment of the present application is shown;
[0050] Fig.15 A schematic diagram showing the comparison relationship between an image before and after cropping according to an embodiment of the present application is shown;
[0051] Fig.16 A block diagram of an image processing device according to an embodiment of the present application is shown;
[0052] Fig.17 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION
[0053] The exemplary embodiments are now described in a more comprehensive manner with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in various forms and should not be understood as being limited to these examples; on the contrary, the purpose of providing these embodiments is to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.
[0054] In addition, the features, structures or characteristics described in the present application may be combined in one or more embodiments in any suitable manner. In the following description, there are many specific details so that the embodiments of the present application can be fully understood. However, those skilled in the art will appreciate that when implementing the technical scheme of the present application, all the detailed features in the embodiments may not be needed, one or more specific details may be omitted, or other methods, elements, devices, steps, etc. may be adopted.
[0055] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0056] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.
[0057] It should be noted that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.
[0058] It is understandable that in the specific implementation of this application, data related to image processing is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
[0059] Image deduplication refers to confirming whether two images are the same, such as finding out whether an image in an image library is a duplicate of a sample image in the image library. Usually, when performing image deduplication, the image features (such as embedding features) of the query image (i.e., the image that needs to be queried whether it is a duplicate of the sample image) and the sample image in the image library are extracted, and then the distance between the query image and the sample image is calculated through the image features (such as the Euclidean distance between the image features, etc.) to determine whether the query image and the sample image are duplicates. For example, if the distance between the query image and the sample image is less than or equal to the set value, then the query image and the sample image are considered to be duplicates.
[0060] In an image feature extraction scheme, such as Figure 1 As shown, image 101 (which can be a query image or a sample image) is subjected to feature extraction by a CNN (Convolutional Neural Networks) model to obtain a feature space heat map, and then average or maximum pooling is performed to obtain one-dimensional image features. However, this solution loses the object position information in the image when pooling deep features (i.e., features represented by the feature space heat map) into image features, making it impossible to distinguish between two images with the same character but different character positions. For example, two game images contain two game characters that appear in the same map area, but the positions of the two game characters are different in the two game images. In this case, Figure 1 The method shown cannot distinguish that the positions of the characters in the two images are different.
[0061] At the same time, due to Figure 1 The method shown extracts the global features of image 101, but the global features cannot accurately represent the fine-grained character change information. Therefore, if the character is relatively small relative to the entire image, the character information will be lost, which will result in the obtained image features being difficult to truly represent the real information in image 101. For example, in a game image, a game character occupies 3.9% of the entire image area. If the CNN model finally compresses the 224×224 image to 14×14 (compression ratio 224 / 14=16) through convolution, then the pixel value of the game character in the feature space heat map is 224×0.039 / 16=0.5, that is, the representation of the game character in the feature space heat map is less than 1 pixel value. In other words, the heat map no longer has the representation information of the position, action, orientation, etc. of the game character.
[0062] like Figure 2 As shown, in another image feature extraction scheme, image 201 (which can be a query image or a sample image) is firstly subjected to image role detection, and then the detected roles are respectively extracted through a CNN model to obtain role representations, and the obtained role representations are used as image features of the entire image to perform similarity calculation for deduplication processing. Figure 2 Although the scheme shown can detect the character information in the image, it cannot obtain the relative position information of the character in the image, and further it is difficult to distinguish images with the same character but different character positions.
[0063] Regarding the above Figure 1 and Figure 2 To solve the problems in the above scheme, the embodiment of the present application proposes a new image processing scheme, which can accurately represent the features of the image through the feature representation vector of the object and the object relationship vector, thereby facilitating the improvement of the accuracy of image duplication detection. Specifically, Figure 3 The exemplary system architecture block diagram applying the technical solution of the present application is schematically shown.
[0064] like Figure 3As shown, an exemplary system architecture applying the technical solution of the present application may include a terminal device 310, a network 320 and a server 330. The terminal device 310 may include a smart phone, a tablet computer, a laptop computer, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, and the like. The server 330 may be a server that provides various services, which may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The network 320 may be a communication medium of various connection types that can provide a communication link between the terminal device 310 and the server 330, such as a wired communication link or a wireless communication link.
[0065] According to the implementation needs, the system architecture in the embodiment of the present application can have any number of terminal devices, networks and servers. For example, the server 330 can be a server group composed of multiple server devices. In addition, the technical solution provided in the embodiment of the present application can be applied to the terminal device 310, can also be applied to the server 330, or can be implemented by the terminal device 310 and the server 330 together, and the present application does not make special restrictions on this.
[0066] For example, the terminal device 310 may provide the image to be processed to the server 330 through the network 320. The image to be processed may be, for example, an image that needs to be checked for duplicates. After acquiring the image to be processed, the server 330 may detect each object contained in the image to be processed, obtain an object detection result, and then generate a feature representation vector of each object based on the object detection result. After obtaining the feature representation vector of each object, the positional relationship between each object contained in the image to be processed may be extracted based on the feature representation vector of each object to obtain an object relationship vector, and then the image to be processed may be detected based on the feature representation vector of each object and the obtained object relationship vector to determine whether the template image in the image library is repeated.
[0067] It should be noted that, after acquiring the image that needs to be checked for duplicates, the terminal device 310 can also detect the various objects contained in the image to be processed and generate a feature representation vector for each object, and then extract the positional relationship between the various objects contained in the image to be processed based on the feature representation vector of each object to obtain an object relationship vector, and then detect whether the image to be processed is a duplicate of the template image in the image library based on the feature representation vector of each object and the obtained object relationship vector.
[0068] In some optional embodiments, it can be determined whether the image to be processed and the template image in the image library are repeated in the following manner: if the feature representation vectors of each object in the image to be processed match the feature representation vectors corresponding to the specified template image, and the object relationship vector corresponding to the image to be processed matches the object relationship vector corresponding to the specified template image, then it is determined that the image to be processed and the specified template image are repeated. However, if the feature representation vectors of each object in the image to be processed do not match the feature representation vectors corresponding to the specified template image, or the object relationship vectors corresponding to the image to be processed do not match the object relationship vectors corresponding to the specified template image, then it is determined that the image to be processed and the specified template image are not repeated.
[0069] It can be seen that the technical solution of the embodiment of the present application takes into account the feature representation vectors of each object contained in the image when extracting image features for duplicate detection, and on the other hand, can also take into account the positional relationship between each object. Then, the image features can be more accurately represented by the object's feature representation vector and the object relationship vector, which is beneficial to improving the accuracy of image duplicate detection.
[0070] The implementation details of the technical solution of the embodiment of the present application are described in detail below:
[0071] Figure 4 A flowchart of an image processing method according to an embodiment of the present application is shown. The image processing method can be executed by a server, or by a terminal device, or by both a server and a terminal device. Figure 4 As shown, the image processing method at least includes steps S410 to S440, which are described in detail as follows:
[0072] In step S410, each object contained in the image to be processed is detected to obtain an object detection result.
[0073] In some optional embodiments, the image to be processed may be an image that needs to be matched with a template image in an image library to determine whether it is a duplicate of the template image. For example, the image to be processed may be an image of a game screen, or an image of a sports match screen, an animation image, etc. The object contained in the image to be processed refers to the person, object, virtual character, etc. contained in the image to be processed. For example, if an image is a battle screen of two game characters, then the objects contained in the image may be the two game characters and other objects present in the battle screen.
[0074] In some optional embodiments, each object contained in the image to be processed can be detected by machine learning. Among them, machine learning (Machine Learning, referred to as ML) is a multi-field cross-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all fields of artificial intelligence. Machine learning and deep learning generally include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, teaching learning and other technologies. Optionally, a YOLO detection model can be selected to detect each object contained in the image to be processed.
[0075] In step S420, a feature representation vector of each object is generated according to the object detection result.
[0076] In some optional embodiments, after obtaining the object detection results, the feature representation vectors of each object can be extracted through a machine learning model. Figure 2 In the method shown, after detecting each object contained in the image to be processed, the feature representation vector of each object is extracted by the CNN model.
[0077] It should be noted that the feature representation vector of an object is a combination used to represent the features of the object, and can be an n-dimensional numerical vector (n≥1) used to distinguish different objects. In different business scenarios, the feature representation vectors of objects are different. For example, in a game scenario, the feature representation vector of an object is used to distinguish different game characters in the game screen. In this case, the feature representation vector can specifically include the facial features, dress features, and features of the props used by the game characters; in a sports competition scenario, the feature representation vector of an object is used to distinguish different athletes on the field. In this case, the feature representation vector can specifically include the facial features, sportswear number features, and hairstyle features of the athletes.
[0078] In step S430, the positional relationship between the objects contained in the image to be processed is extracted according to the feature representation vector of each object to obtain an object relationship vector.
[0079] In some optional embodiments, after obtaining the feature representation vector of each object, the object relationship vector can be extracted through the object relationship extraction model. For example, after obtaining the feature representation vector of each object, a feature representation graph corresponding to the image to be processed can be obtained according to the position of each object in the image to be processed and the feature representation vector of each object, and then the feature representation graph is input into the object relationship extraction model to obtain the object relationship vector.
[0080] It should be noted that the object relationship vector is a feature combination used to characterize the relationship between objects. It can be an n-dimensional numerical vector (n≥1) used to characterize the position relationship (such as relative position) and action relationship between multiple objects. In different business scenarios, object relationship vectors are different. For example, in a game scenario, an object relationship vector can represent the position relationship between multiple game characters in the game screen, whether they are in a battle state, and the way they attack each other; while in a sports competition scenario, an object relationship vector can represent the position relationship, attack and defense status, etc. between multiple athletes in the sports competition screen.
[0081] In step S440, based on the feature representation vectors and object relationship vectors of each object, it is detected whether the image to be processed is repeated with the template image in the image library.
[0082] In some optional embodiments, based on the feature representation vector and the object relationship vector, detecting whether the image to be processed and the template image in the image library are repeated may specifically be: if the feature representation vectors of each object of the image to be processed match the feature representation vector corresponding to the specified template image, and the object relationship vector of the image to be processed matches the object relationship vector corresponding to the specified template image, then it is determined that the image to be processed is repeated with the specified template image. If the feature representation vectors of each object of the image to be processed do not match the feature representation vector corresponding to the specified template image, or the object relationship vector of the image to be processed does not match the object relationship vector corresponding to the specified template image, then it is determined that the image to be processed and the specified template image are not repeated.
[0083] In some optional embodiments, the feature representation vector and object relationship vector corresponding to the template image in the image library can be obtained in the same processing method as the image to be processed, and can be processed when a template image is detected to be stored in the library. For example, if it is detected that a template image needs to be stored in the image library, each object contained in the template image is detected to obtain the object detection result corresponding to the template image, and then the feature representation vector of each object in the template image is generated according to the object detection result corresponding to the template image to obtain the feature representation vector corresponding to the template image, and then the positional relationship between each object contained in the template image is extracted according to the feature representation vector corresponding to each object in the template image to obtain the object relationship vector corresponding to the template image; then the template image is associated with the feature representation vector and the object relationship vector corresponding to the template image and stored in the image library.
[0084] Figure 5 A flowchart of an image processing method according to an embodiment of the present application is shown. The image processing method can be executed by a server, or by a terminal device, or by both a server and a terminal device. Figure 5 As shown, the image processing method at least includes steps S510 to S560, which are described in detail as follows:
[0085] In step S510, each object contained in the image to be processed is detected to obtain an object detection result.
[0086] Optionally, the specific implementation method of this step is similar to the aforementioned step S410 and is not repeated here.
[0087] In step S520, a feature representation vector of each object is generated according to the object detection result.
[0088] Optionally, the specific implementation method of this step is similar to the aforementioned step S420 and is not repeated here.
[0089] In step S530, each object is quantized according to its characteristic representation vector to obtain a quantized value of each object.
[0090] It should be noted that the quantization process in the embodiment of the present application refers to converting the feature representation vector of the object into a numerical value, thereby more accurately representing the position of the object on the image through the numerical value. The following is a detailed description:
[0091] In some optional embodiments, each object is quantized according to its feature characterization vector, and the process of obtaining the quantized value of each object may be: obtaining multiple cluster centers, which are obtained by clustering the feature characterization vectors of the objects contained in the template image; determining the cluster center closest to the feature characterization vector of each object according to the distance between the feature characterization vector of each object and the multiple cluster centers; generating the quantized value of each object according to the identification value corresponding to the cluster center closest to the feature characterization vector of each object.
[0092] Optionally, the template image may be an image in an image library, and the feature representation vectors of the objects contained in the template image may be extracted through a machine learning model. After extracting the feature representation vectors of the objects contained in multiple template images in the image library, the feature representation vectors of these objects may be clustered using a K-means clustering algorithm to obtain multiple cluster centers, and the identification value of the cluster center may be the serial number of the cluster center (such as 1, 2, 3, etc.). After determining the cluster center closest to the feature representation vector of each object, the identification value corresponding to the cluster center closest to the feature representation vector of each object may be used as the quantitative value of each object.
[0093] For example, assuming there are three cluster centers, the identification values of these three cluster centers are 1, 2, and 3 respectively. If the distance between the feature representation vector of object 1 in the image to be processed and cluster center 2 is the shortest, then the identification value 2 of cluster center 2 is used as the quantized value of object 1.
[0094] In some optional embodiments, when clustering the feature representation vectors of the objects contained in the template image, the number of known objects and the number of associated objects of the known objects can be obtained, and then the number of cluster centers can be determined based on the number of known objects and the number of associated objects. Then, based on the determined number of cluster centers, the feature representation vectors of the objects contained in the template image are clustered to obtain multiple cluster centers.
[0095] Optionally, known objects refer to the number of objects that are known and may exist in the image to be processed. For example, if the image to be processed is a game screen of a certain game, then the known objects may be game characters in the game; for another example, if the image to be processed is a sports match screen, then the known objects may be athletes participating in the sports match. The associated objects of a known object may be objects that have an associated relationship with the known object and affect the identification of the known object. For example, if the known object is a game character in a certain game, then the associated objects may be the skin of the game character. This is because the same game character presents different visual effects when wearing different skins. Therefore, in order to ensure that game characters wearing different skins can be distinguished, the number of cluster centers can be determined based on the number of game characters and the number of skins.
[0096] Optionally, in the scene of the game screen, assuming that there are x game characters, the number of skins of these x game characters can be accumulated, and the accumulated value can be used as the number of cluster centers.
[0097] In step S540, a quantization map corresponding to the image to be processed is generated according to the position of each object in the image to be processed and the quantization value of each object.
[0098] In some optional embodiments, according to the position of each object in the image to be processed and the quantization value of each object, the process of generating a quantization map corresponding to the image to be processed may be: according to the position of each object in the image to be processed, using the quantization value of each object to replace the pixel value at the position of each object; then replacing the pixel value of the image to be processed except the position of each object with the set value to generate a quantization map corresponding to the image to be processed.
[0099] Optionally, the set value can be 0 or other values. Figure 6 In the example shown, the image to be processed 601 contains three objects, and the quantization values of these three objects are 1, 2, and 3 respectively. Then, according to the positions of these three objects in the image to be processed 601, the pixel values at the position of each object can be replaced with the corresponding quantization values, and the pixel values at other positions in the image to be processed can be replaced with 0, and finally a quantization map 602 is obtained (in order to clearly show the positions of these three objects, 0 is not marked in the quantization map 602, that is, the pixel value of the blank area in the quantization map 602 is 0).
[0100] In step S550 , an object relationship vector is extracted from the quantization graph.
[0101] In some optional embodiments, extracting the object relationship vector from the quantization graph may be inputting the quantization graph into an object relationship extraction model, and then using the output embedding vector of the object relationship extraction model as the extracted object relationship vector, wherein the training process of the object relationship extraction model is as follows: Figure 8 shown.
[0102] Optionally, the object relationship extraction model may include multiple convolutional layers connected in sequence, and the quantization map is input into the first convolutional layer among the multiple convolutional layers. In order to ensure that features of different resolutions in the image can be accurately represented, the outputs of at least two of the multiple convolutional layers can be merged after pooling processing to obtain the object relationship vector.
[0103] For example, Figure 7 The object relationship extraction model shown in the figure contains five sequentially connected convolutional layers, namely conv1, conv2, conv3, conv4, and conv5. The quantization map can be input into conv1, and then the outputs of conv3, conv4, and conv5 are merged after pooling, and the merged result is used as the object relationship vector. In this way, the features of different resolutions can be fully considered to ensure that the obtained object relationship vector can more accurately represent the features of the image. It should be noted that if the feature map sizes of the outputs of conv3, conv4, and conv5 are different after pooling, they can be converted into feature maps of the same size through the embedding layer before merging.
[0104] In step S560, based on the feature representation vectors and object relationship vectors of each object, it is detected whether the image to be processed is repeated with the template image in the image library.
[0105] Optionally, the specific implementation method of this step is similar to the aforementioned step S440 and is not repeated here.
[0106] The following combination Figure 8 The training process of the object relationship extraction model mentioned in the above embodiment is described in detail.
[0107] Figure 8 The diagram shows a training process of an object relationship extraction model according to an embodiment of the present application. In specific implementation, it can be executed by a server, or by a terminal device, or by both a server and a terminal device. Figure 8 As shown, the training process of the object relationship extraction model according to an embodiment of the present application includes the following steps S810 to S840, which are described in detail as follows:
[0108] In step S810, a triplet for training an object relationship extraction model is obtained, the triplet comprising an anchor image, a positive sample image, and a negative sample image; wherein the anchor image, the positive sample image, and the negative sample image are all quantized images.
[0109] Optionally, the process of generating the anchor image, the positive sample image and the negative sample image in the form of a quantization map is similar to the process of generating the quantization map corresponding to the image to be processed in the aforementioned embodiment, and will not be described in detail.
[0110] In some optional embodiments, when obtaining a triple for training an object relationship extraction model, a selection may be made from a given plurality of quantization graphs. Specifically, a plurality of quantization graphs and object record information corresponding to each quantization graph may be obtained, and the object record information is obtained by arranging the quantization values of the objects contained in the quantization graph in a set order, for example, traversing row by row from the upper left corner of the quantization graph, and then arranging the instance quantization values in the order in which the instances appear, such as for Figure 6 In the quantization graph shown, when traversing row by row starting from the upper left corner of the quantization graph, an instance with an instance quantization value of 1 appears first, followed by an instance with an instance quantization value of 2, and then an instance with an instance quantization value of 3, so the corresponding object record information can be expressed as [1,2,3].
[0111] After obtaining multiple quantization maps and the object record information corresponding to each quantization map, a target quantization map can be selected from the multiple quantization maps, and then, based on the target object record information corresponding to the target quantization map, the first quantization map whose corresponding object record information is the same as the target object record information or contains the target object record information can be searched from the multiple quantization maps. Optionally, the target quantization map can be any one of the multiple quantization maps. For example, a quantization map can be randomly selected from the multiple quantization maps as the target quantization map. Among them, if the object record information corresponding to the two quantization maps is the same, it means that the two quantization maps are similar to a large extent; if the object record information corresponding to the two quantization maps has an inclusion relationship, it means that the two quantization maps are largely a subgraph of one map, such as a quantization map. Figure 1 The corresponding object record information is [1,2,3], quantized Figure 2 The corresponding object record information is [1,2,3,4], which means that the quantization Figure 1 To a large extent, quantitative Figure 2 The subgraph of .
[0112] After searching for the first quantization map whose corresponding object record information is the same as the target object record information or contains the target object record information from multiple quantization maps, the target quantization map can be used as the anchor image, the second quantization map can be used as the positive sample image corresponding to the target quantization map, and the other quantization maps except the target quantization map and the first quantization map can be used as the negative sample images corresponding to the target quantization map to generate a triple. For example, there are 5 quantization maps in total, of which quantization Figure 1 is the anchor image, quantized Figure 2 and quantification Figure 3 is a positive sample image, quantized Figure 4 and quantification Figure 5 is a negative sample image, then the generated triplet can be (quantized Figure 1 , quantification Figure 2 , quantification Figure 4 ), (Quantitative Figure 1 , quantification Figure 2 , quantification Figure 5 ), (Quantitative Figure 1 , quantification Figure 3 , quantification Figure 4 ), (Quantitative Figure 1 , quantification Figure 3 , quantification Figure 5 ).
[0113] In some optional embodiments, when obtaining a triplet for training an object relationship extraction model, a selected anchor image may be obtained first, and then, when the object relationship in the anchor image is unchanged, data enhancement processing may be performed on the anchor image to obtain a positive sample image corresponding to the anchor image. Optionally, the selected anchor image may be an image randomly selected from an image library.
[0114] Optionally, when the object relationship in the anchor image does not change, the data enhancement processing for the anchor image may be to perform overall movement processing on the anchor image or all objects contained in the anchor image, such as horizontal movement, vertical movement, or movement in a specified direction.
[0115] Optionally, when the object relationship in the anchor image does not change, the data enhancement processing on the anchor image may be to reduce or enlarge the anchor image, or to reduce or enlarge a designated object in the anchor image.
[0116] Optionally, when the object relationship in the anchor image does not change, performing data enhancement processing on the anchor image may be performing cropping processing on the anchor image on the basis of retaining the object contained in the anchor image.
[0117] In some optional embodiments, when obtaining a triplet for training an object relationship extraction model, a selected anchor image may be obtained first, and then at least one of an object and an object relationship in the anchor image may be changed to obtain a negative sample image corresponding to the anchor image. Optionally, the selected anchor image may be an image randomly selected from an image library.
[0118] Optionally, changing at least one of the object and the object relationship in the anchor image to obtain the negative sample image corresponding to the anchor image may be to move part of the object in the anchor image, such as moving an object horizontally, vertically, or in a specified direction.
[0119] Optionally, changing at least one of an object and an object relationship in the anchor image to obtain a negative sample image corresponding to the anchor image may be randomly replacing at least one object in the anchor image.
[0120] Optionally, changing at least one of the object and the object relationship in the anchor image to obtain the negative sample image corresponding to the anchor image may be by replacing other objects in the anchor image that are different from the specified object with the specified object in an image similar to the anchor image.
[0121] Optionally, changing at least one of an object and an object relationship in the anchor image to obtain a negative sample image corresponding to the anchor image may be swapping positions of at least two objects in the anchor image.
[0122] Optionally, changing at least one of the object and the object relationship in the anchor image to obtain the negative sample image corresponding to the anchor image may be adjusting the position of the specified object in the anchor image to a set position, such as adjusting the position of the specified object to the edge of the image.
[0123] In step S820, the triplet is input into the object relationship extraction model to obtain a first embedding vector for the anchor image, a second embedding vector for the negative sample image, and a third embedding vector for the positive sample image output by the object relationship extraction model.
[0124] Alternatively, the object relation extraction model can be Figure 7 The model shown, that is, through Figure 7 The model shown extracts the first embedding vector for the anchor image, the second embedding vector for the negative sample image, and the third embedding vector for the positive sample image.
[0125] In step S830, a difference between the first distance and the second distance is calculated according to the first distance between the first embedding vector and the second embedding vector, and the second distance between the first embedding vector and the third embedding vector.
[0126] Optionally, the first distance and the second distance may be L2 distance, i.e., Euclidean distance, or L1 distance, i.e., Manhattan distance, or other distances.
[0127] In step S840, the model parameters of the object relationship extraction model are adjusted with the difference between the first distance and the second distance being greater than or equal to the set value as a training target.
[0128] In some optional embodiments, the process of adjusting the model parameters of the object relationship extraction model with the difference between the first distance and the second distance being greater than or equal to the set value as the training target can be: generating the loss value corresponding to the triple according to the difference between the first distance and the second distance and the set value, and then adding the loss values corresponding to the set number of triplets respectively as the total loss of a round of training, and then adjusting the model parameters of the object relationship extraction model based on the total loss. In this embodiment, the model parameters of the object relationship extraction model are adjusted by the total loss of a group of triplets (i.e., a set number of triplets). In other embodiments of the present application, the model parameters of the object relationship extraction model can also be adjusted by the loss value of each triple. Optionally, the set number can be 5, 10 or other numerical values.
[0129] Optionally, the loss value corresponding to the triplet generated according to the difference between the first distance and the second distance and the set value can be specifically: if the sum of the opposite number of the difference and the set value is greater than 0, then the sum of the opposite number of the difference and the set value is used as the loss value; if the sum of the opposite number of the difference and the set value is less than or equal to 0, then 0 is used as the loss value.
[0130] The above describes the image processing scheme in the embodiment of the present application. When extracting image features for duplicate detection, on the one hand, the feature representation vectors of each object contained in the image can be taken into account, and on the other hand, the positional relationship between each object can also be taken into account. Then, the image features can be more accurately represented by the object feature representation vector and the object relationship vector, which is conducive to improving the accuracy of image duplicate detection. In order to further explain the implementation details that are not detailed in the embodiment of the present application, the following is combined with Figures 9 to 15 , the technical solution of the embodiment of the present application is described again:
[0131] like Fig. 9The operation interface related to image processing in the embodiment of the present application is shown, and the operation of image storage and image duplication checking are provided in the operation interface. Specifically, one or more images can be uploaded by clicking control 901. After uploading the image, if control 902 is clicked, it is determined that the uploaded image needs to be stored in the image library, and then the feature representation vector and object relationship vector of the object contained in the image can be extracted, and then stored in the image library in association with the image. After uploading the image, if control 903 is clicked, it is determined that the operation of duplication checking is performed, then the uploaded image will be compared and calculated with the image in the image library, and then the duplicate image obtained by the query is displayed in window 904.
[0132] Fig. 9 The processing process of the operation interface is as follows Fig.10 As shown, in the storage process, the stored image first passes through the instance detection model to obtain the detection instance result (wherein the instance is the object mentioned in the aforementioned embodiment), and then passes through the instance representation model to obtain the instance representation (that is, the feature representation vector of the object) and stores it in the image instance representation library; and after obtaining the instance representation, an instance quantization graph is generated and the instance relationship representation (that is, the object relationship vector) is extracted and stored in the image instance relationship library. In the duplicate checking process, for the query image, the instance representation and instance relationship representation are also extracted, and then the instance representation is compared with the image instance representation library to determine whether the instance representation is repeated. If there is a repeated image in the instance representation, then it is determined whether the instance relationship representation of the image and the query image is also repeated. When both are repeated, yes is returned (that is, there is a repeated image, and the repeated image can be returned), otherwise no is returned (that is, there is no repeated image).
[0133] It can be seen that in the embodiment of the present application, for both the stored image and the query image, the instance relationship representation is obtained through a similar processing process, as shown in the following example: Fig.11 As shown in the figure, the image (including the stored image and the query image) is passed through the instance detection model to obtain the detection instance result, and then based on the instance detection result, the instance quantization processing is performed to obtain the entity quantization graph, and then the instance relationship representation corresponding to the image is extracted through the relationship extraction model.
[0134] In some optional embodiments, the instance detection model may adopt a YOLO detection model trained based on an open source coco dataset, and may be fine-tuned based on business data in actual application scenarios (such as game characters, sports figures, etc.).
[0135] In some optional embodiments, the instance representation model is used to output the embedding representation of the detected instance, and the feature vector output by the pooling layer of the resnet101 model trained based on the open source ImageNet can be used as the representation, or it can be obtained by training the CNN model with business data in actual application scenarios (such as game characters, sports figures, etc.). Optionally, the Resnet101 network structure parameters used in the embodiment of the present application are shown in Table 1 below:
[0136]
[0137] Table 1
[0138] As shown in Table 1, the residual network Resnet101 used in the embodiment of the present application mainly includes multiple convolutional layers (i.e., conv1, Conv2_x, Conv3_x, Conv4_x, and Conv5_x) and a pooling layer (Pool). The size of the feature map output by each convolutional layer is reduced in sequence, and finally after pooling processing, an embedding representation of size 1×2048 is generated as an instance representation.
[0139] In one embodiment of the application, after the instance characterization is obtained, the result based on the instance characterization can be quantified to obtain an entity quantization graph. Specifically, since multiple detection results will be generated for the same target after instance detection, instance detection results far greater than the actual number of instance categories will be generated. For example, if there is a certain game character in a game, then the game character may be detected in the game screen under multiple game skins, and may be in multiple action states such as firing skills, turning around, walking, being knocked out, etc. In this case, if when extracting the instance relationship in the game screen, an instance relationship is extracted for each instance in each action state, the extracted instance relationship will eventually be too detailed to cope with the state change of the game character. This is also because the changed game character will be regarded as a new instance in this case, which will cause the extraction of the instance relationship to no longer be accurate. Fig.12 As shown, instance 2 in (a) changes to the state shown in (b) after the action state changes. Then, due to the change in the action state of instance 2, it will be considered as a new instance, namely instance 4, which will cause the extraction of instance relations to no longer be accurate.
[0140] In view of the problem that different instance states lead to multiple instances of the same role, thus causing inaccurate instance relationships, the embodiment of the present application proposes to improve the accuracy of instance representation through a quantization method. The specific process is: extract instances from images in the image library, obtain instance representations for all extracted instances, and then cluster the instance representations with M centers using the K-means clustering method (the number of M is defined by the business data of the actual application scenario, such as in the game scenario, it can be determined according to the number of game characters and the number of game skins. Specifically, the number of skins of the game characters can be accumulated, and the accumulated value is used as the number of cluster centers), and the representations of M cluster centers are obtained as quantization points. The quantization points are marked with codes 1, 2, ..., M, that is, the code of the first cluster center is 1, and so on. Then, for a certain image to be processed, after the instance is detected, the instance representation of each instance is obtained, and then the nearest quantization point is found for each instance representation, and the code of the quantization point is used as the pixel value at the instance position on the image to generate an instance quantization map.
[0141] for example, Figure 6 The game screen shown in , after instance detection, instance characterization, and quantization point search, the obtained quantization map is shown in Figure 602, where the pixel value of the blank area in the quantization map 602 is 0, indicating that there is no instance, all 1s indicate that there is a hero with a quantization value of 1 at that position in the image, all 2s indicate that there is a hero with a quantization value of 2 at that position in the image, and all 3s indicate that there is a hero with a quantization value of 3 at that position in the image. In this representation method, when the action state of a certain game character changes, the instance of the same game character can be found more accurately, avoiding the problem of multiple instances of the same game character in different action states. For example Fig.13 As shown, when Figure 6 The position of instance 2 in the image shown is changed to obtain image 1301. Since only the action state and position of instance 2 have changed, it is still represented by 2 in the quantization diagram 1302, but the position of the pixels occupied has changed.
[0142] In one embodiment of the present application, the structure of the relation extraction model for extracting instance relation representations can be as follows: Figure 7 As shown, convolution blocks conv1, conv2, conv3, cov4, and conv5 are used to extract deep features of the image, and the model parameters can be initialized using the parameters of the resnet101 model pre-trained based on ImageNet.
[0143] Considering that features at different resolutions in deep convolutional networks have different representation capabilities for instances in images, for example, conv1 is closest to the input image, with a small convolution receptive field and low resolution, while conv5 has a large convolution receptive field and high resolution, this may cause the instance to be too small relative to the image and the conv5 receptive field to be too large, making the instance unable to be represented after conv5. At the same time, shallow networks can retain more detailed textures in the image, while deep networks can learn the existence of instances in the image. Therefore, considering the problem that instances may occupy too small an area in the image, in Figure 7 The model structure shown adds a multi-granularity bypass representation union to avoid the problem that the instance cannot be represented by the last network layer. In other words, in the embodiment of the present application, the outputs of conv3, conv4 and conv5 can be merged after pooling and embedding, and the merged result is used as the extracted instance relationship representation.
[0144] Optionally, in one embodiment of the present application, Figure 7 The parameters of the model structure shown are shown in Table 2:
[0145]
[0146]
[0147] Table 2
[0148] As shown in Table 2, the outputs of conv3, conv4, and conv5 are respectively processed by pooling and embedding to obtain an embedding representation of size 1×64, and then the merged embedding representation is used as the instance relationship representation.
[0149] In one embodiment of the present application, when training the relationship extraction model, triple data can be used for training, but because the relationship extraction model needs to learn instance relationship representations, and it is difficult to obtain a large number of such training samples in the real world, this is because only a certain proportion of image repetition samples are relational repetitions, and it is difficult to label image pairs with the same instance relationship on a large scale from massive data. Therefore, in the embodiment of the present application, triplets for training the relationship extraction model can be obtained through data labeling and data enhancement.
[0150] The triplet in the embodiment of the present application includes (anchor point anchor, positive sample positive, negative sample negative), where anchor and positive are extremely similar samples (in the image deduplication scenario), and anchor and negative are dissimilar samples. Since the embodiment of the present application is used in the image deduplication scenario, two image samples need to be extremely similar to be considered similar samples. The embodiment of the present application needs to find more valuable triplets from a large number of samples during the data preparation stage.
[0151] In some optional embodiments, triple data can be obtained by data annotation. Specifically, firstly, the instance is detected from the image to be annotated, then the instance representation is obtained, the instance representation on all images is obtained, and the instance quantization map is obtained, and then the association list between the instance quantization map and the instance record is established. It should be noted that the instance record is obtained by arranging the instance quantization values in the instance quantization map in a certain order. For example, the instance record can be traversed row by row starting from the upper left corner of the quantization map, and then the instance quantization values are arranged in the order in which the instances appear, such as for Figure 6 In the quantization graph shown, when traversing row by row starting from the upper left corner of the quantization graph, the instance with instance quantization value 1 appears first, then the instance with instance quantization value 2 appears, and then the instance with instance quantization value 3 appears, so the corresponding instance records can be expressed as [1,2,3]. Based on this, the association list between instance quantization graphs and instance records can be: quantization graph of graph i - [1,2,3]; quantization graph of graph j - [1,3,4]; ...
[0152] For a randomly selected instance quantization map, query the associated list for images that have duplicate instances in the instance quantization map. Figure 1 -[1,2,3], 10 quantization graphs are found, and the association relationships between these 10 quantization graphs and the association list are: Quantization Figure 2 -[1,2,3], Quantization Figure 3 -[2,1,3], Quantization Figure 4 -[1,2,3,6], Quantization Figure 5 -[1,2,6], Quantization Figure 6 -[1,3,4], Quantization Figure 7 -[2,3], Quantization Figure 8 -[2,3], Quantization Fig. 9 -[2,6], Quantization Fig.10 -[2] Quantization Fig.11 -[2]. The instance quantization images corresponding to the instance records with exactly the same instance ranking in the search results are marked as identical / extremely similar images, such as quantization Figure 2 -[1,2,3] and quantization Figure 1-[1,2,3] are exactly the same; while quantization Figure 4 -[1,2,3,6] contains [1,2,3], i.e. quantization Figure 1 It is quantitative Figure 4 subgraph, so the quantization Figure 2 and quantification Figure 4 As a quantification Figure 1 The corresponding positive sample; and other quantization maps are used as quantization Figure 1 The corresponding negative samples.
[0153] In some optional embodiments, triple data may be obtained by data enhancement, which may include translation, random reduction, random expansion, random cropping, and the like.
[0154] Translation: The translation mode (1,2) and translation amount (x,y) can be randomly generated; the (1,2) in the translation mode represents the translation of the entire image and the translation of any instance, respectively, and (x,y) represents the translation amount along the length and width of the image. The corresponding instance is selected according to the translation mode, and the instance is translated. When the translated instance reaches the edge of the image, the maximum position that can be moved to in the translation direction is used as the final position. When the translated instance overlaps with another instance, the instance closer to the bottom of the image is used as the near view, and the instance slightly farther from the bottom of the image is used as the far view. The occlusion result is generated based on the principle that the near view occludes the far view. Fig.14 As shown, the quantization map corresponding to image 1401 is 1402; instance 2 in image 1401 is translated to the right to obtain image 1401', in which instance 2 occludes instance 1, but because instance 2 is closer to the bottom of the image, instance 2 is taken as the near view and instance 1 is taken as the far view. Based on the principle that the near view occludes the far view, the quantization map 1402' is obtained, that is, the quantization value of instance 2 occludes the quantization value of instance 1. It should be noted that, except for Fig.14 The horizontal translation method shown can also be a vertical translation or a translation method in any direction.
[0155] Random reduction: You can randomly generate reduction modes (1, 2) and reduction parameters ratio and scale. The reduction modes (1, 2) represent full image processing and arbitrary instance processing respectively; scale represents the reduction range (for example, it can take a value between 1 and 3 pixels), which means that the pixels within the scale range of the instance edge are reduced; ratio represents the reduction probability (for example, it can take a value between 0 and 1), which means that a pixel within the reduction range is reduced with a ratio probability. When the probability is less than the ratio, it is reduced, and when it is greater than the ratio, it is not reduced.
[0156] Random expansion: Similar to the random reduction operation, it converts the reduction operation into a pixel expansion operation.
[0157] Random cropping: Cropping the original image to any image between the minimum image containing the instance and the original image size. After cropping, the instance relationship in the original image is still contained. For example, Fig.15 The image 1501 shown in is cropped into image 1502. Assuming that w1, w2, h1, and h2 represent the maximum cropping amounts on the left, right, top, and bottom, respectively, four random numbers less than or equal to the four maximum cropping amounts can be generated during cropping as the final cropping amounts.
[0158] In one embodiment of the present application, for any quantized image, when constructing a positive sample, one of the above data enhancement methods can be selected: full image translation, random reduction, random expansion, and random cropping methods to perform data enhancement processing to obtain a positive sample. This is because these data enhancement methods do not change the relationship between instances in the image, so the enhanced image and the original image constitute a positive sample pair.
[0159] In one embodiment of the present application, for any quantization map, as long as the instance position of the image changes or the instance changes, a negative sample can be generated. Therefore, the construction of the negative sample can randomly select one of the following methods to operate: translate the specified instance; randomly select an instance in the image and change it to another random instance (or by finding the instance quantization map that is most similar to the image but not the same as the image in the image instance record list, and use the newly appeared instance in the similar map); randomly exchange the positions of multiple instances in the image; move the specified instance to a specified position (such as the lower right corner of the image, the bottom of the image, etc.).
[0160] In one embodiment of the present application, when the relation extraction model is trained, bs (for example, 10) triples can be selected as a round of model training process. In the model parameter initialization stage, the resnet101 model parameters trained based on the large open source data set ImageNet can be used as the convolutional layer parameters for initialization (or the parameters of other models can also be used for initialization), and other newly added structures such as embedding are initialized with a Gaussian distribution with a variance of 0.01 and a mean of 0. When training, the N quantized graphs are divided into at least one batch (batch, the network is updated in batches, and the network parameters are updated once for a batch of data), each batch contains bs quantized graphs, and a total of N / bs model learning is performed to complete a round of iteration (an epoch).
[0161] Specifically, for each batch of bs quantized graphs, first obtain n×bs (n is a set constant, such as 10) triplets according to the above method, and then input the triplet samples into the relation extraction model to perform forward calculation of the model and calculate the corresponding loss value. Then calculate the total loss of a batch, and use the SGD (stochastic gradient descent) method to calculate the gradient of each parameter in the relation extraction model; update the model parameters according to the gradient and the current learning rate, and complete an epoch of training after completing the full N / bs updates; end after a total of K epochs of training (or end when the average loss of 10 consecutive epochs does not decrease).
[0162] Optionally, when updating the model parameters, all parameters in the relationship extraction model can be learned, and the network parameters involved in the learning can be set to a learning rate of 0.0005, and the learning rate is adjusted to 0.1 times the original every 10 rounds.
[0163] In one embodiment of the present application, the loss of the relation extraction model can adopt triplet loss. Specifically, after determining the triplet (anchor, positive, negative), the embedding features of these triplet are used to calculate the triplet loss (i.e., triplet loss). The calculation formula of triplet loss is as follows:
[0164] L triplet =max(||D a -D p ||-||D a -D n ||+α, 0)
[0165] Among them, L triplet represents triple loss; ||D a -D p || represents the L2 distance (i.e., Euclidean distance) between the anchor embedding and the positive embedding; || D a -D n || represents the L2 distance between the anchor embedding and the negative embedding; α represents the set difference, for example, it can be 0.6. The purpose of model training is to make the distance between the anchor embedding and the negative embedding greater than the distance between the anchor embedding and the positive embedding by margin.
[0166] In the above embodiments of the present application, a model for extracting the relationship between instances can be quickly constructed, that is, through limited labeled samples, data enhancement processing between instance relationships, and self-supervised similarity learning methods, the model for extracting the relationship between instances can be quickly constructed, avoiding the problem of a large amount of manual labeling leading to a longer model update cycle, and thus being more friendly to image relationship representation learning in migration services. In addition, by mapping instances to quantized space, the amount of instance representation data is effectively compressed, reducing the difficulty of relationship learning and avoiding the problem of inaccurate relationship learning.
[0167] After training the relation extraction model, you can Fig. 9 As shown, the library images or query images are uploaded, and then the instance representations and instance relationship representations can be extracted according to the technical solutions of the above embodiments of the present application, thereby realizing the duplicate checking of the images.
[0168] The following describes an apparatus embodiment of the present application, which can be used to execute the image processing method in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned embodiment of the image processing method of the present application.
[0169] Fig.16 FIG. 1 is a block diagram of an image processing device according to an embodiment of the present application. Fig.16 As shown, an image processing device 1600 according to an embodiment of the present application includes: a detection unit 1602 , a generation unit 1604 , an extraction unit 1606 and a processing unit 1608 .
[0170] Among them, the detection unit 1602 is configured to detect each object contained in the image to be processed and obtain an object detection result; the generation unit 1604 is configured to generate a feature representation vector of each object based on the object detection result; the extraction unit 1606 is configured to extract the positional relationship between each object contained in the image to be processed based on the feature representation vector of each object to obtain an object relationship vector; the processing unit 1608 is configured to detect whether the image to be processed is repeated with a template image in an image library based on the feature representation vector and the object relationship vector.
[0171] In some embodiments of the present application, based on the aforementioned scheme, the extraction unit 1606 is configured to: perform quantization processing on each object according to the feature representation vector of each object to obtain a quantization value of each object; generate a quantization map corresponding to the image to be processed according to the position of each object in the image to be processed and the quantization value of each object; and extract the object relationship vector from the quantization map.
[0172] In some embodiments of the present application, based on the aforementioned scheme, the extraction unit 1606 performs quantization processing on each object according to the feature representation vector of each object to obtain a quantization value of each object, including: obtaining multiple cluster centers, where the cluster centers are obtained by clustering the feature representation vectors of the objects contained in the template image; determining the cluster center closest to the feature representation vector of each object according to the distance between the feature representation vector of each object and the multiple cluster centers; and generating the quantization value of each object according to the identification value corresponding to the cluster center closest to the feature representation vector of each object.
[0173] In some embodiments of the present application, based on the aforementioned scheme, the image processing device also includes: a clustering unit, configured to obtain the number of known objects and the number of associated objects of the known objects, and obtain the feature representation vector of the objects contained in the template image; determine the number of cluster centers according to the number of known objects and the number of associated objects; and perform clustering processing on the feature representation vectors of the objects contained in the template image according to the number of cluster centers to obtain the multiple cluster centers.
[0174] In some embodiments of the present application, based on the aforementioned scheme, the extraction unit 1606 generates a quantization map corresponding to the image to be processed according to the position of each object in the image to be processed and the quantization value of each object, including: according to the position of each object in the image to be processed, using the quantization value of each object to replace the pixel value at the position of each object; replacing the pixel value in the image to be processed except the position of each object with a set value to generate a quantization map corresponding to the image to be processed.
[0175] In some embodiments of the present application, based on the aforementioned scheme, the extraction unit 1606 extracts the object relationship vector from the quantization map, including: inputting the quantization map into an object relationship extraction model, the object relationship extraction model includes a plurality of convolutional layers connected in sequence, and the quantization map is input to the first convolutional layer among the plurality of convolutional layers; merging the outputs of at least two of the plurality of convolutional layers after pooling processing to obtain the object relationship vector.
[0176] In some embodiments of the present application, based on the aforementioned scheme, the image processing device also includes: an acquisition unit, configured to acquire a triplet for training the object relationship extraction model, the triplet including an anchor image, a positive sample image and a negative sample image; wherein the anchor image, the positive sample image and the negative sample image are all quantized images; an input unit, configured to input the triplet into the object relationship extraction model to obtain a first embedding vector for the anchor image, a second embedding vector for the negative sample image and a third embedding vector for the positive sample image output by the object relationship extraction model; a calculation unit, configured to calculate the difference between the first distance and the second distance based on the first distance between the first embedding vector and the second embedding vector, and the second distance between the first embedding vector and the third embedding vector; an adjustment unit, configured to adjust the model parameters of the object relationship extraction model with the difference being greater than or equal to a set value as a training target.
[0177] In some embodiments of the present application, based on the aforementioned scheme, the adjustment unit is configured to: generate a loss value corresponding to the triplet according to the difference and the set value; add the loss values corresponding to a set number of triplets as the total loss for a round of training; and adjust the model parameters of the object relationship extraction model based on the total loss.
[0178] In some embodiments of the present application, based on the aforementioned scheme, the acquisition unit is configured to: acquire multiple quantization maps and object record information corresponding to each quantization map, wherein the object record information is obtained by arranging the quantization values of the objects contained in the quantization map in a set order; select a target quantization map from the multiple quantization maps, and according to the target object record information corresponding to the target quantization map, search from the multiple quantization maps for a first quantization map whose corresponding object record information is the same as the target object record information or contains the target object record information; use the target quantization map as an anchor image, use the second quantization map as a positive sample image corresponding to the target quantization map, and use other quantization maps among the multiple quantization maps except the target quantization map and the first quantization map as negative sample images corresponding to the target quantization map, to generate the triplet.
[0179] In some embodiments of the present application, based on the aforementioned scheme, the acquisition unit is configured to: acquire a selected anchor image; and perform data enhancement processing on the anchor image without changing the object relationship in the anchor image to obtain a positive sample image corresponding to the anchor image.
[0180] In some embodiments of the present application, based on the above solution, when the object relationship in the anchor image does not change, data enhancement processing is performed on the anchor image, including at least one of the following methods:
[0181] Performing overall movement processing on the anchor image or all objects contained in the anchor image;
[0182] Performing reduction or enlargement processing on the anchor point image;
[0183] Performing reduction or enlargement processing on a designated object in the anchor image;
[0184] The anchor image is cropped on the basis of retaining the object contained in the anchor image.
[0185] In some embodiments of the present application, based on the aforementioned scheme, the acquisition unit is configured to: acquire a selected anchor image; change at least one of an object and an object relationship in the anchor image to obtain a negative sample image corresponding to the anchor image.
[0186] In some embodiments of the present application, based on the above scheme, changing at least one of the object and the object relationship in the anchor image to obtain a negative sample image corresponding to the anchor image includes at least one of the following methods:
[0187] Performing movement processing on some objects in the anchor point image;
[0188] randomly replacing at least one object in the anchor image;
[0189] Replace other objects in the anchor image that are different from the specified object with the specified object in the similar image of the anchor image;
[0190] swapping positions of at least two objects in the anchor image;
[0191] The position of the designated object in the anchor image is adjusted to a set position.
[0192] In some embodiments of the present application, based on the aforementioned scheme, the processing unit 1608 is configured as follows: if the feature representation vector matches the feature representation vector corresponding to the specified template image, and the object relationship vector matches the object relationship vector corresponding to the specified template image, then it is determined that the image to be processed is a duplicate of the specified template image.
[0193] In some embodiments of the present application, based on the aforementioned scheme, the processing unit 1608 is configured to: if the feature representation vector does not match the feature representation vector corresponding to the specified template image, or the object relationship vector does not match the object relationship vector corresponding to the specified template image, then determine that the image to be processed is not repeated with the specified template image.
[0194] In some embodiments of the present application, based on the aforementioned scheme, the processing unit 1608 is further configured to: if it is detected that a template image needs to be stored in the image library, then detect each object contained in the template image to obtain the object detection result corresponding to the template image; generate a feature representation vector for each object in the template image according to the object detection result corresponding to the template image to obtain the feature representation vector corresponding to the template image; extract the positional relationship between each object contained in the template image according to the feature representation vector corresponding to each object in the template image to obtain the object relationship vector corresponding to the template image; associate the template image with the feature representation vector and the object relationship vector corresponding to the template image and store them in the image library.
[0195] Fig.17 A schematic diagram showing the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. The electronic device may be Figure 3 The terminal device 310 or the server 330 shown in FIG.
[0196] It should be noted that Fig.17 The computer system 1700 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0197] like Fig.17 As shown, the computer system 1700 may include a central processing unit (CPU) 1701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1702 or the program loaded from the storage part 1708 to the random access memory (RAM) 1703, such as executing the method described in the above embodiment. In RAM 1703, various programs and data required for system operation are also stored. CPU 1701, ROM 1702 and RAM 1703 are connected to each other through bus 1704. Input / output (I / O) interface 1705 is also connected to bus 1704.
[0198] The following components can be connected to the I / O interface 1705: an input section 1706 including a keyboard, a mouse, etc.; an output section 1707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1708 including a hard disk, etc.; and a communication section 1709 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1709 performs communication processing via a network such as the Internet. A drive 1710 is also connected to the I / O interface 1705 as needed. A removable medium 1711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1710 as needed so that a computer program read therefrom is installed into the storage section 1708 as needed.
[0199] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program is used to perform the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1709, and / or installed from a removable medium 1711. When the computer program is executed by a central processing unit (CPU) 1701, various functions defined in the system of the present application are executed.
[0200] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a computer program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0201] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and a computer program.
[0202] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.
[0203] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more computer programs, and when the above one or more computer programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.
[0204] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.
[0205] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable an electronic device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation method of the present application.
[0206] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.
[0207] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. An image processing method, characterized in that: include: Detecting each object contained in the image to be processed and obtaining an object detection result; Generating a feature representation vector of each object according to the object detection result; Extracting the positional relationship between the various objects contained in the image to be processed according to the feature representation vectors of the various objects to obtain an object relationship vector; According to the feature representation vector and the object relationship vector, it is detected whether the image to be processed is repeated with a template image in an image library.
2. The image processing method according to claim 1, characterized in that: Extracting the positional relationship between the objects contained in the image to be processed according to the feature representation vectors of the objects to obtain the object relationship vector, including: Quantizing each object according to the feature representation vector of each object to obtain a quantized value of each object; Generating a quantization map corresponding to the image to be processed according to the position of each object in the image to be processed and the quantization value of each object; The object relationship vector is extracted from the quantization graph.
3. The image processing method according to claim 2, characterized in that: According to the feature representation vectors of the objects, the objects are quantized to obtain quantized values of the objects, including: Acquire multiple cluster centers, where the cluster centers are obtained by clustering feature representation vectors of objects contained in the template image; Determine the cluster center closest to the feature characterization vector of each object according to the distance between the feature characterization vector of each object and the multiple cluster centers; The quantized value of each object is generated according to the identification value corresponding to the cluster center closest to the feature representation vector of each object.
4. The image processing method according to claim 3, characterized in that: The image processing method further comprises: Acquire the number of known objects and the number of objects associated with the known objects, and acquire feature representation vectors of the objects contained in the template image; Determining the number of cluster centers according to the number of the known objects and the number of the associated objects; According to the number of the cluster centers, clustering processing is performed on the feature representation vectors of the objects contained in the template image to obtain the multiple cluster centers.
5. The image processing method according to claim 2, characterized in that: Generating a quantization map corresponding to the image to be processed according to the position of each object in the image to be processed and the quantization value of each object, including: According to the position of each object in the image to be processed, using the quantized value of each object to replace the pixel value at the position of each object; The pixel values in the image to be processed except for the positions where the various objects are located are replaced with set values to generate a quantization map corresponding to the image to be processed.
6. The image processing method according to claim 2, characterized in that: Extracting the object relationship vector from the quantization graph includes: Inputting the quantization map into an object relationship extraction model, wherein the object relationship extraction model includes a plurality of convolutional layers connected in sequence, and the quantization map is input into a first convolutional layer among the plurality of convolutional layers; Outputs of at least two convolutional layers among the multiple convolutional layers are respectively pooled and then merged to obtain the object relationship vector.
7. The image processing method according to claim 6, characterized in that: The image processing method further comprises: Acquire a triplet for training the object relationship extraction model, wherein the triplet includes an anchor image, a positive sample image, and a negative sample image; wherein the anchor image, the positive sample image, and the negative sample image are all quantized images; Inputting the triplet into the object relationship extraction model to obtain a first embedding vector for the anchor image, a second embedding vector for the negative sample image, and a third embedding vector for the positive sample image output by the object relationship extraction model; Calculate a difference between the first distance and the second distance based on a first distance between the first embedding vector and the second embedding vector and a second distance between the first embedding vector and the third embedding vector; The model parameters of the object relationship extraction model are adjusted with the difference being greater than or equal to a set value as a training target.
8. The image processing method according to claim 7, characterized in that: Taking the difference being greater than or equal to a set value as a training target, adjusting the model parameters of the object relationship extraction model includes: Generate a loss value corresponding to the triple according to the difference and the set value; Add the loss values corresponding to the set number of triplets as the total loss for one round of training; Model parameters of the object relationship extraction model are adjusted based on the total loss.
9. The image processing method according to claim 7, characterized in that: The image processing method further comprises: Acquire multiple quantization maps and object record information corresponding to each quantization map, wherein the object record information is obtained by arranging the quantization values of the objects included in the quantization map according to a set order; Selecting a target quantization map from the multiple quantization maps, and searching, according to the target object record information corresponding to the target quantization map, for a first quantization map whose corresponding object record information is the same as the target object record information or includes the target object record information from the multiple quantization maps; The target quantization map is used as an anchor image, the second quantization map is used as a positive sample image corresponding to the target quantization map, and other quantization maps among the multiple quantization maps except the target quantization map and the first quantization map are used as negative sample images corresponding to the target quantization map to generate the triplet.
10. The image processing method according to claim 7, characterized in that: Acquiring a triplet for training the object relationship extraction model, including: Get the selected anchor image; When the object relationship in the anchor image does not change, data enhancement processing is performed on the anchor image to obtain a positive sample image corresponding to the anchor image.
11. The image processing method according to claim 10, characterized in that: When the object relationship in the anchor image does not change, data enhancement processing is performed on the anchor image, including at least one of the following methods: Performing overall movement processing on the anchor image or all objects contained in the anchor image; Performing reduction or enlargement processing on the anchor point image; Performing reduction or enlargement processing on a designated object in the anchor point image; The anchor image is cropped on the basis of retaining the object contained in the anchor image.
12. The image processing method according to claim 7, characterized in that: Acquiring a triplet for training the object relationship extraction model, including: Get the selected anchor image; At least one of an object and an object relationship in the anchor image is changed to obtain a negative sample image corresponding to the anchor image.
13. The image processing method according to claim 12, characterized in that: Changing at least one of an object and an object relationship in the anchor image to obtain a negative sample image corresponding to the anchor image includes at least one of the following methods: Performing movement processing on some objects in the anchor point image; randomly replacing at least one object in the anchor image; Replace other objects in the anchor image that are different from the specified object with the specified object in the similar image of the anchor image; swapping positions of at least two objects in the anchor image; The position of the designated object in the anchor image is adjusted to a set position.
14. The image processing method according to any one of claims 1 to 13, characterized in that: Detecting whether the image to be processed is repeated with a template image in an image library according to the feature representation vector and the object relationship vector includes: If the feature characterization vector matches the feature characterization vector corresponding to the designated template image, and the object relationship vector matches the object relationship vector corresponding to the designated template image, it is determined that the image to be processed overlaps with the designated template image.
15. The image processing method according to any one of claims 1 to 13, characterized in that: Detecting whether the image to be processed is repeated with a template image in an image library according to the feature representation vector and the object relationship vector includes: If the feature characterization vector does not match the feature characterization vector corresponding to the designated template image, or the object relationship vector does not match the object relationship vector corresponding to the designated template image, it is determined that the image to be processed does not overlap with the designated template image.
16. The image processing method according to any one of claims 1 to 13, characterized in that: The image processing method further comprises: If it is detected that there is a template image that needs to be stored in the image library, each object contained in the template image is detected to obtain an object detection result corresponding to the template image; Generate a feature representation vector of each object in the template image according to the object detection result corresponding to the template image, and obtain the feature representation vector corresponding to the template image; Extracting the positional relationship between the objects contained in the template image according to the feature representation vector corresponding to each object in the template image, and obtaining the object relationship vector corresponding to the template image; The template image and the feature representation vector and the object relationship vector corresponding to the template image are associated and stored in the image library.
17. An image processing device, characterized in that: include: A detection unit, configured to detect each object contained in the image to be processed and obtain an object detection result; A generating unit, configured to generate a feature representation vector of each object according to the object detection result; an extraction unit configured to extract the positional relationship between the various objects contained in the image to be processed according to the feature representation vectors of the various objects to obtain an object relationship vector; The processing unit is configured to detect whether the image to be processed is repeated with a template image in an image library according to the feature representation vector and the object relationship vector.
18. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 16 is implemented.
19. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more computer programs, which, when executed by the one or more processors, enables the electronic device to implement the image processing method as described in any one of claims 1 to 16.
20. A computer program product, characterized in that The computer program product includes a computer program, which is stored in a computer-readable storage medium. A processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the image processing method according to any one of claims 1 to 16.
