Image detection method, device and equipment, computer readable storage medium and computer program product

Abstract

The invention provides an image detection method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the steps that first text recognition is carried out on an image to obtain a first text, second text recognition is carried out on the image to obtain a second text, the first text is used for describing attribute information of a target object in the image, and the second text is used for describing a counterfeit trace of the image; performing feature extraction on the first text to obtain a first text feature, performing feature extraction on the second text to obtain a second text feature, and performing visual feature extraction on the image to obtain a first visual feature; performing feature decoupling on the first visual feature based on the first text feature and the second text feature to obtain a second visual feature related to the attribute information and a third visual feature related to the counterfeit trace; and carrying out feature fusion on the second text feature and the third visual feature, and carrying out authenticity detection on the image based on the fused feature to obtain a first detection result.

Description

Technical Field

[0001] The present application relates to computer perception technology, and in particular to an image detection method, apparatus, device, storage medium, and program product. Background Art

[0002] With the development of computer technology, more and more business scenarios require user identity verification through images in order to provide corresponding services to users after passing identity authentication. On this basis, illegal attackers often forge verification images for identity authentication to "trick" the identity authentication system into treating the forged verification images as normal verification images, thereby passing the identity authentication of illegal attackers, increasing the risk of information confidentiality. Summary of the Invention

[0003] The embodiments of the present application provide an image detection method, device, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of authenticity detection of images.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present invention provides an image detection method, including:

[0006] Performing first text recognition on an image to obtain a first text, and performing second text recognition on the image to obtain a second text, wherein the first text is used to describe attribute information of a target object in the image, and the second text is used to describe traces of forgery in the image;

[0007] Performing feature extraction on the first text to obtain a first text feature, performing feature extraction on the second text to obtain a second text feature, and performing visual feature extraction on the image to obtain a first visual feature;

[0008] performing feature decoupling on the first visual feature based on the first text feature and the second text feature to obtain a second visual feature related to the attribute information and a third visual feature related to the forgery trace;

[0009] The second text feature and the third visual feature are fused to obtain a first fused feature, and authenticity detection is performed on the image based on the first fused feature to obtain a first detection result.

[0010] The present invention provides an image detection device, comprising:

[0011] a recognition module configured to perform first text recognition on an image to obtain a first text, and perform second text recognition on the image to obtain a second text, wherein the first text is used to describe attribute information of a target object in the image, and the second text is used to describe traces of forgery in the image;

[0012] an extraction module, configured to perform feature extraction on the first text to obtain a first text feature, perform feature extraction on the second text to obtain a second text feature, and perform visual feature extraction on the image to obtain a first visual feature;

[0013] a decoupling module, configured to perform feature decoupling on the first visual feature based on the first text feature and the second text feature to obtain a second visual feature related to the attribute information and a third visual feature related to the forgery trace;

[0014] The fusion module is used to perform feature fusion on the second text feature and the third visual feature to obtain a first fusion feature, and perform authenticity detection on the image based on the first fusion feature to obtain a first detection result.

[0015] An embodiment of the present application provides an electronic device, including:

[0016] a memory for storing computer-executable instructions or computer programs;

[0017] The processor is used to implement the image detection method provided in the embodiment of the present application when executing the computer-executable instructions or computer programs stored in the memory.

[0018] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or a computer program for implementing the image detection method provided in the embodiment of the present application when executed by a processor.

[0019] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the image detection method provided in the embodiment of the present application is implemented.

[0020] The embodiments of the present application have the following beneficial effects:

[0021] By using the embodiment of the present application, a first text recognition is performed on an image to obtain a first text, and a second text recognition is performed on the image to obtain a second text, wherein the first text is used to describe the attribute information of the target object in the image, and the second text is used to describe the forgery traces of the image. By distinguishing the second text related to forgery and the first text unrelated to forgery in the image, information unrelated to forgery can be eliminated in the subsequent forgery detection of the image, thereby saving computing resources and improving the accuracy of authenticity detection; feature extraction is performed on the first text to obtain a first text feature, feature extraction is performed on the second text to obtain a second text feature, and visual feature extraction is performed on the image to obtain a first visual feature; feature extraction of the first visual feature is performed based on the first text feature and the second text feature Decoupling to obtain a second visual feature related to attribute information and a third visual feature related to forgery traces; decoupling the third visual feature related to forgery and the second visual feature not related to forgery in the first visual feature by distinguishing whether the text is related to forgery, can improve the accuracy of the decoupled visual features, thereby improving the accuracy of subsequent authenticity detection of the image, performing feature fusion on the second text feature and the third visual feature to obtain a first fused feature, and performing authenticity detection on the image based on the first fused feature to obtain a first detection result, performing authenticity detection only through the second text feature related to forgery and the third visual feature related to forgery, saving computing resources and avoiding the interference of irrelevant information on authenticity detection, thereby improving the accuracy of authenticity detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 1 is a schematic diagram of the architecture of the image detection system 100 provided in an embodiment of the present application;

[0023] Figure 2A is a structural diagram of an electronic device 500-1 provided in an embodiment of the present application;

[0024] Figure 2B is a schematic structural diagram of an electronic device 500-2 provided in an embodiment of the present application;

[0025] Figure 3A This is a schematic diagram of the process of the image detection method provided in the embodiment of the present application. Figure 1 ;

[0026] Figure 3B Schematic diagram 2 of the flow chart of the image detection method provided in the embodiment of the present application;

[0027] Figure 4 Schematic diagram of the training process of the image detection model provided in the embodiment of the present application;

[0028] Figure 5 This is an architecture diagram of the image detection model provided in the embodiment of the present application;

[0029] Figure 6 is a schematic diagram of a labeled sample provided in an embodiment of the present application;

[0030] Figure 7 is a schematic diagram of a text recognition network provided in an embodiment of the present application;

[0031] Figure 8 This is a working principle diagram of the image detection model provided in the embodiment of the present application;

[0032] Figure 9 Schematic diagram of the algorithm evaluation protocol provided in the embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0034] It is understandable that in the embodiments of the present application, when user information and other related data are involved, when the embodiments of the present application are applied to specific products or technologies, user permission or consent must be obtained, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.

[0035] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0036] In the following description, the terms "first\second..." are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second..." can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0037] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0039] 1) Forgery: Image forgery refers to the manipulation or synthesis of an image by digital means to change the original content of the image or create a false visual effect. Such tampering may include adding, deleting, modifying objects or scenes in the image, or changing the context of the image. The purposes of image forgery vary, and may be to entertain, mislead the public, defraud, hide evidence, or express political positions. The following are some common image forgery techniques: using deep learning technology, especially generative adversarial networks, to create highly realistic forged videos or images. These technologies can change a person's expression, movement, or even identity; combining elements from multiple images to create a new scene, which can be achieved through techniques such as layer masks; stitching parts of different images together to create a completely new image, which may be used to create false news photos or scenes; changing the temporal context of the image by changing the temporal elements in the image (such as light, shadow, weather, etc.).

[0040] 2) Attribute information: Attribute information usually refers to the specific attributes of the target object contained in the image. These attributes may include but are not limited to the following: the gender of the target object in the image, which is usually divided into male and female; the age of the target object in the image, which can be a specific age value or a classification of age groups (such as children, teenagers, adults, and the elderly); the race or ethnic group of the target object in the image, such as Africans; the skin color of the target object in the image, which is usually a continuous value indicating the depth of the skin color; the type or style of clothing worn by the target object in the image, such as casual, formal, and sports; the expression of the target object in the image, such as happiness, sadness, surprise, and anger; the posture of the target object in the image, such as standing, sitting, and lying down; the identification of specific target objects in the image, such as celebrities and public figures; the actions of the target object in the image, such as walking, running, and jumping; the scene or environment in which the target object is located, such as indoors, outdoors, in the city, or in the countryside.

[0041] In today's digital age, facial recognition technology has been widely adopted in various fields, from unlocking mobile phones to airport security checks. Its convenience and efficiency are self-evident. However, as technology advances, this system faces unprecedented risks and challenges. High-precision facial forgery technologies, such as deep learning-driven deepfakes, have achieved incredibly realistic levels of realism, significantly enhancing their ability to mimic real faces, making it difficult to distinguish between the real and the fake. Furthermore, current liveness detection technology is not yet fully developed, making it difficult to effectively identify and block sophisticated fake masks or manipulated videos.

[0042] The embodiments of the present application provide an image detection method, device, electronic device, computer-readable storage medium and computer program product, which can improve the accuracy of image detection. The exemplary applications of the electronic devices provided by the embodiments of the present application are described below. The electronic devices provided by the embodiments of the present application can be implemented as various types of user terminals such as laptops, tablet computers, desktop computers, set-top boxes, mobile devices (for example, mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), smart phones, smart speakers, smart watches, smart TVs, vehicle-mounted terminals, security terminals, etc., and can also be implemented as servers. Below, the exemplary applications when the device is implemented as a terminal will be described.

[0043] See also Figure 1 , Figure 1 This is a schematic diagram of the architecture of the image detection system 100 provided in an embodiment of the present application. To support an exemplary application, the terminal 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0044] In some embodiments, the terminal 400 is provided with a client or application with an image detection function, which is used to implement the image detection method provided in the embodiments of the present application, such as performing a first text recognition on the image to obtain a first text, and performing a second text recognition on the image to obtain a second text, wherein the first text is used to describe the attribute information of the target object in the image, and the second text is used to describe the forgery traces of the image; performing feature extraction on the first text to obtain a first text feature, performing feature extraction on the second text to obtain a second text feature, and performing visual feature extraction on the image to obtain a first visual feature; performing feature decoupling on the first visual feature based on the first text feature and the second text feature to obtain a second visual feature related to the attribute information and a third visual feature related to the forgery traces; performing feature fusion on the second text feature and the third visual feature to obtain a first fusion feature, and performing authenticity detection on the image based on the first fusion feature to obtain a first detection result.

[0045] In some embodiments, the image detection method provided in the embodiments of the present application can also be implemented by a terminal and a server, that is, the terminal 400 obtains an image to be detected and sends the obtained image to the server 200; the server 200 performs a first text recognition on the image to obtain a first text, and performs a second text recognition on the image to obtain a second text, wherein the first text is used to describe the attribute information of the target object in the image, and the second text is used to describe the forgery traces of the image; feature extraction is performed on the first text to obtain a first text feature, feature extraction is performed on the second text to obtain a second text feature, and visual feature extraction is performed on the image to obtain a first visual feature. Based on the first text feature and the second text feature, the first visual feature is feature decoupled to obtain a second visual feature related to the attribute information and a third visual feature related to the forgery traces; the second text feature and the third visual feature are feature fused to obtain a first fused feature; and the image is authenticated based on the first fused feature to obtain a first detection result. The final first detection result is returned to the terminal 400.

[0046] In some embodiments, the terminal 400 is provided with a client or application having a training function of an image detection model, which is used to implement the training method of the image detection model provided in the embodiment of the present application, such as determining a training sample, wherein the training sample is marked with a first label, a second label and a third label, the first label indicates the first attribute information of the object sample in the training sample, the second label indicates the first forgery trace of the training sample, and the third label indicates whether the training sample is a forged image; through the text recognition network of the image detection model, text recognition is performed on the training sample to obtain a third text and a fourth text, wherein the third text is used to describe the second attribute information of the object sample in the training sample, and the fourth text is used to describe the second forgery trace of the training sample; through the feature extraction network of the image detection model, Feature extraction is performed on the third text to obtain a third text feature, feature extraction is performed on the fourth text to obtain a fourth text feature, and visual feature extraction is performed on the training sample to obtain a fourth visual feature; through the feature decoupling network of the image detection model, feature decoupling is performed on the fourth visual feature based on the third text feature and the fourth text feature to obtain a fifth visual feature related to the second attribute information and a sixth visual feature related to the second forgery trace; through the classification network of the image detection model, feature fusion is performed on the fourth text feature and the sixth visual feature to obtain a second fused feature, and authenticity detection is performed on the training sample based on the second fused feature to obtain a second detection result; the image detection model is trained based on the third text, the fourth text, the fifth visual feature, the sixth visual feature and the second detection result.

[0047] In some embodiments, the training of the image detection model provided in the embodiments of the present application can also be implemented by the terminal and the server, that is, the terminal 400 obtains the training sample and sends the obtained training sample to the server 200; the server 200 performs text recognition on the training sample through the text recognition network of the image detection model to obtain a third text and a fourth text, wherein the third text is used to describe the second attribute information of the object sample in the training sample, and the fourth text is used to describe the second forgery trace of the training sample; through the feature extraction network of the image detection model, feature extraction is performed on the third text to obtain a third text feature, feature extraction is performed on the fourth text to obtain a fourth text feature, and visual feature extraction is performed on the training sample to obtain a fourth visual feature; through the feature decoupling network of the image detection model, feature decoupling is performed on the fourth visual feature based on the third text feature and the fourth text feature to obtain a fifth visual feature related to the second attribute information and a sixth visual feature related to the second forgery trace; through the classification network of the image detection model, feature fusion is performed on the fourth text feature and the sixth visual feature to obtain a second fused feature, and authenticity detection is performed on the training sample based on the second fused feature to obtain a second detection result; the image detection model is trained based on the third text, the fourth text, the fifth visual feature, the sixth visual feature and the second detection result.

[0048] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in the embodiments of the present application.

[0049] See also Figure 2A , Figure 2A This is a schematic diagram of the structure of the electronic device 500-1 provided in the embodiment of the present application, with the electronic device 500-1 as an example. Figure 1 For example, Figure 2A The electronic device 500-1 shown includes: at least one processor 510-1, a memory 550-1, at least one network interface 520-1 and a user interface 530-1. The various components in the electronic device 500-1 are coupled together via a bus system 540-1. It is understood that the bus system 540-1 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 540-1 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 540-1 is not described in detail. Figure 2AIn the figure, various buses are labeled as bus system 540 - 1 .

[0050] Processor 510-1 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0051] The memory 550-1 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550-1 described in the embodiments of the present application is intended to include any suitable type of memory. The memory 550-1 may optionally include one or more storage devices physically remote from the processor 510-1.

[0052] In some embodiments, the memory 550 - 1 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0053] The operating system 551-1 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic businesses and process hardware-based tasks; the network communication module 552-1 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 520-1. Exemplary network interfaces 520-1 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.

[0054] In some embodiments, the image detection device provided in the embodiments of the present application can be implemented in software. The image detection device provided in the embodiments of the present application can be provided as various software embodiments, including various forms including applications, software, software modules, scripts or codes. Figure 2A An image detection device 555-1 stored in the memory 550-1 is shown, which can be software in the form of programs and plug-ins, and includes a series of modules, including a recognition module 5551, an extraction module 5552, a decoupling module 5553 and a fusion module 5554. These modules are logical and can therefore be arbitrarily combined or further split according to the functions implemented.

[0055] See also Figure 2B , Figure 2B This is a schematic diagram of the structure of the electronic device 500-2 provided in the embodiment of the present application, with the electronic device 500-2 as an example. Figure 1 For example, Figure 2B The electronic device 500-2 shown includes: at least one processor 510-2, a memory 550-2, at least one network interface 520-2 and a user interface 530-2. The various components in the electronic device 500-2 are coupled together via a bus system 540-2. It is understood that the bus system 540-2 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 540-2 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 540-2 is not described in detail. Figure 2B In the figure, various buses are labeled as bus system 540 - 2 .

[0056] Processor 510-2 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0057] The memory 550-2 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550-2 described in the embodiments of the present application is intended to include any suitable type of memory. The memory 550-2 may optionally include one or more storage devices physically remote from the processor 510-2.

[0058] In some embodiments, the memory 550 - 2 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0059] The operating system 551-2 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic businesses and process hardware-based tasks; the network communication module 552-2 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 520-2. Exemplary network interfaces 520-2 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.

[0060] In some embodiments, the image detection device provided in the embodiments of the present application can be implemented in software. The image detection device provided in the embodiments of the present application can be provided as various software embodiments, including various forms including applications, software, software modules, scripts or codes. Figure 2B A training device 555-2 for an image detection model stored in a memory 550-2 is shown, which may be software in the form of a program and a plug-in, and includes a series of modules, including a determination module 5555, an identification module 5556, an extraction module 5557, a decoupling module 5558, a fusion module 5559 and a training module 5550. These modules are logical and therefore can be arbitrarily combined or further split according to the functions implemented.

[0061] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image detection method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0062] In some embodiments, the terminal or server can implement the image detection method provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as car navigation APP, security APP; it can also be a small program that can be embedded in any APP, that is, a program that only needs to be downloaded to a browser environment and can be run. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0063] The image detection method provided by the embodiment of the present application will be described in conjunction with the exemplary application and implementation of the electronic device provided by the embodiment of the present application. As mentioned above, the image detection method provided by the embodiment of the present application can be implemented by various types of electronic devices, for example, Figure 1 The terminal and the server 200 in the embodiment can be executed by either one of them separately or by Figure 1 Therefore, the execution entities of each step will not be described again in the following.

[0064] It should be noted that in the following example of image detection, images are used as an example. Based on their understanding of the following, those skilled in the art can apply the image detection method provided in the embodiment of the present application to the detection of multiple continuously captured video frames.

[0065] See also Figure 3A , Figure 3A This is a flow chart of the image detection method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0066] In step 101, a first text recognition is performed on an image to obtain a first text, and a second text recognition is performed on the image to obtain a second text, wherein the first text is used to describe attribute information of a target object in the image, and the second text is used to describe forgery traces of the image.

[0067] In actual implementation, the image can be an image including the target object captured by the image acquisition device of the terminal, or an image including the target object obtained from a database and stored in a database, or an image including the target object uploaded by a user through an interface, or an image including the target object downloaded from the Internet.

[0068] In some embodiments, the first text obtained by performing first text recognition on the image in step 101 can be obtained by Figure 3B Steps 1011 to 1014 shown implement:

[0069] In step 1011, object recognition is performed on the image to obtain the target object included in the image.

[0070] In actual implementation, the attributes of the target object may be specific attributes of the target object contained in the image, such as the target object's gender, age, skin color, clothing, lighting, and expression.

[0071] As an example, a method for performing object recognition on an image can be to first preprocess the image, such as adjusting the size and resolution of the image, and then use a pre-trained object detection model to perform object detection on the preprocessed image. The object detection model outputs a series of bounding boxes and corresponding object categories and confidence levels, and then applies a non-maximum suppression algorithm to remove overlapping bounding boxes, retain the most likely object area, and use the object area as the target object included in the image.

[0072] In step 1012, attribute features of the target object are extracted to obtain the attribute features of the target object.

[0073] In actual implementation, the process of extracting attribute features of the target object can be to use color histogram, color moment, color moment, etc. to describe the color features of the target object; use texture energy map, gray level co-occurrence matrix, local binary pattern, etc. to describe the texture features of the target object; use the object's bounding box, principal component analysis, etc. to describe the shape features of the target object; use convolutional neural network to extract the visual features of the target object, and finally input the various features of the extracted target object into the fully connected layer, and the fully connected layer outputs the attribute features of the target object.

[0074] In step 1013, attribute recognition is performed on the target object based on the attribute features to obtain attribute information of the target object.

[0075] In actual implementation, the method of identifying the attributes of the target object can use a preset target detection algorithm (such as a single-shot multi-frame detection algorithm, a regional convolutional neural network, etc.) to detect the target object in the image, and extract features from the area obtained by target detection to obtain the image features of the area. Then, the image features of the area are decoded to obtain the attribute information of the target object.

[0076] As an example, attribute recognition of a target object based on attribute features and obtaining attribute information of the target object can also be achieved through a pre-trained decoder. Specifically, the obtained attribute features of the target object can be input into the decoder, and the decoder decodes the attribute features to obtain the attribute information of the target object.

[0077] In step 1014 , a first text is generated based on the attribute information.

[0078] In actual implementation, the process of generating text from an image and obtaining the first text can be achieved by combining image analysis and natural language processing technology. Specifically, the image file is first loaded using an image detection library, and then necessary adjustments are made to the image, such as scaling and cropping, to meet the needs of subsequent processing; then, a suitable text description template is selected based on the analysis results, and the extracted image feature information is filled into the description template to obtain the first text.

[0079] As an example, the method of generating the first text based on attribute information can also be to splice the basic information to obtain the first text. For example, if the attribute information is age: 38, gender: male, expression: angry, the generated first text can be [38, male, angry].

[0080] In some embodiments, performing second text recognition on the image to obtain the second text in step 101 can be achieved by the following technical solution: identifying forgery traces on the image through at least one of the following operations: performing metadata analysis on the image to obtain metadata analysis results; performing color feature extraction on the image to obtain color features of the image, and determining the color distribution of the image based on the color features; performing texture feature extraction on the image to obtain texture features of the image, and identifying the texture pattern of the image based on the texture features; performing compression trace detection on the image to obtain compression traces in the image; performing lighting consistency detection on the image to obtain lighting effects in the image; performing object consistency detection on the image to obtain object detection results; determining the forgery traces of the image based on at least one of the metadata analysis results, color distribution, texture pattern, compression traces, lighting effects, and objects in the image that do not conform to the actual situation.

[0081] In actual implementation, metadata analysis is performed on the image, and the metadata analysis results can be obtained by inputting the image into a professional tool or programming library that can read and analyze image metadata, and the professional tool or programming library outputs the metadata analysis results, wherein the professional tool or programming library can be a Python image processing library, an Interchangeable Image Text Format Reader, a Joint Image Experts Format Metadata Processing Library, etc.

[0082] When determining the color distribution of an image, color features of the image can be extracted to obtain the color features of the image, and the color distribution of the image can be determined based on the color features. The specific processing process may include: first determining the color value (such as RGB value, etc.) of each pixel in the image, calculating the frequency of occurrence of each color in the image, forming a color histogram, calculating the first-order moment, second-order moment and third-order moment of the image to obtain information such as the average brightness, variance and skewness of the image color, using a clustering algorithm (such as k-means) to determine the main colors in the image, and finally, analyzing the color distribution of the image based on the extracted color features; for the color histogram, the color distribution can be observed by visualizing the histogram; for color clustering, the main colors and distribution of the image can be determined based on the clustering results.

[0083] When determining the texture type in an image, texture features can be extracted through algorithms such as gray-level co-occurrence matrix and local binary pattern. The texture pattern in the image can then be identified through edge detection and texture analysis. The texture pattern of the image is analyzed and interpreted based on the recognition results, which may include determining the texture type in the image (such as smooth, rough, striped, etc.).

[0084] When determining the compression traces of an image, compression trace detection can use edge detection algorithms to detect edges in the image. Compression traces may appear as unnatural transitions near the edges; it can also analyze artifacts caused by compression in the image, such as mosquito effect (mosquito noise) or blocking effect. These artifacts are usually caused by block coding and quantization in compression. Based on the detected features and patterns, it is evaluated whether the image contains compression traces.

[0085] When performing lighting detection on an image machine, the process of performing lighting consistency detection on the image can be as follows: first, the image can be converted into a grayscale image, and then the gradient of the image can be calculated, and the distribution of the gradient can be analyzed to estimate the lighting direction, and the high-frequency details in the image can be analyzed to estimate the uniformity of the lighting. The edge detection algorithm is used to identify the edges in the image, and the continuity of the lighting changes at the edges is analyzed. Then, the shadow direction of the objects in the image can be checked for consistency. Inconsistency may indicate unnatural lighting conditions. The brightness of different areas of the image is analyzed. Uneven brightness distribution may indicate inconsistent lighting. If there is a change in color temperature in the image, it may indicate that the light sources in different areas are different.

[0086] When performing consistency detection on an image, the process of performing object consistency detection on the image can be as follows: first, use the object detection algorithm to identify the objects in the image, then check whether the size of the object is consistent with the actual scene, analyze the occlusion relationship between objects, check whether they are natural, evaluate whether the lighting affects the detection and recognition of the object, and check the logical relationship between objects, such as whether an object should be in a specific position. Based on the results of the consistency check, analyze which objects in the image may have been tampered with. If, during the above detection process, the size of the object does not match the actual scene, or the occlusion between objects is unnatural, or there is a problem with the logical relationship between objects, it can be determined that the image of the object does not match the actual situation, and the presence of forgery traces on the object is used as the detection result of the object.

[0087] In actual implementation, if it is determined that there is an abnormality in at least one of the metadata analysis results, color distribution, texture pattern, compression traces, lighting effects, and objects in the image that do not conform to the actual situation, the abnormal part can be regarded as a trace of image forgery.

[0088] In actual implementation, generating a second text based on the forged traces can be achieved by combining image analysis and natural language processing technology. Specifically, the forged traces can be preprocessed first. The preprocessing process can be to adjust the size and resolution of the forged traces. Then, a suitable text description template can be selected according to the forged traces. After selecting the text description template, the information of the forged traces can be filled into the text description template according to the format of the text description template. Finally, the generated description text can be corrected to ensure its accuracy and readability, so that the corrected description text can be output to obtain the second text.

[0089] In step 102 , feature extraction is performed on the first text to obtain a first text feature, feature extraction is performed on the second text to obtain a second text feature, and visual feature extraction is performed on the image to obtain a first visual feature.

[0090] In some embodiments, the feature extraction of the first text to obtain the first text feature in step 102 can be achieved by the following technical solution: dividing the first text into segments to obtain multiple text segments; extracting text features from each text segment to obtain text features of each text segment; determining the weight of each text segment, and based on the weight, performing feature fusion on the text features of each text segment to obtain the first text feature.

[0091] In actual implementation, the process of segmenting the first text to obtain multiple text segments can find the positions of the first text that can be used as segmentation points based on the grammatical structure and semantic logic of Chinese. For example, punctuation marks such as commas, periods, and semicolons included in the first text, as well as words or phrases that can form independent meanings, can be used to divide the first text into multiple segments according to the positions of the segmentation points.

[0092] In actual implementation, text features of text fragments can be extracted separately through neural networks to obtain text features corresponding to each text fragment. Then, the weight of each text fragment can be determined according to the importance of the information included in each text fragment. For example, the first text is [38, male, angry], where 38 represents the age of the target object, male represents the gender of the target object, and angry represents the expression of the target object. In the process of determining the attribute information of the target object, the importance of age can be set greater than the importance of gender, and the importance of gender can be set greater than the importance of expression. It can be seen that the weight of the text fragment corresponding to "38" is greater than the weight of the text fragment corresponding to "male", and the weight of the text fragment corresponding to "male" is greater than the weight of the text fragment corresponding to "angry"; after obtaining the weight of each text fragment, the text features of each text fragment can be weighted fused based on the weight of each text fragment to obtain the fused first text feature.

[0093] In some implementations, extracting visual features from the image to obtain the first visual feature in step 102 can be achieved through the following technical solutions: extracting visual features from the image to obtain the second visual feature; partitioning the image to obtain multiple visual areas, and extracting visual features from each visual area to obtain the third visual feature of each visual area; based on the second visual feature, estimating the weight of each visual area to obtain the weight of each visual area; based on the weight of each visual area, performing feature fusion on each third visual feature and the second visual feature to obtain the first visual feature.

[0094] In actual implementation, the visual feature extraction method of an image can be to extract the color histogram, color moment, etc. of the image, extract the texture features of the image, such as Local Binary Patterns (LBP), etc., extract the shape description of the object in the image, such as edges, corners, contours, etc., and automatically learn the feature representation of the image by training the neural network model.

[0095] In actual implementation, the process of partitioning the image can be to adopt a preset partitioning algorithm, set a pixel value range, and regard adjacent pixels in the image that are in the same pixel value range as the same visual area, based on which multiple visual areas are obtained; it is also possible to first identify the area of ​​rapid change in pixel intensity in the image, and use this area as the boundary area of ​​each visual area, and divide the image into multiple visual areas based on the boundary area); it is also possible to first arbitrarily select multiple target pixel points in the image, and by determining the similarity between each target pixel point and the adjacent pixel points, regard the pixel points with similarity higher than the similarity threshold as the pixel points belonging to the same visual area as the target pixel point, based on which the image is divided into multiple visual partitions; it is also possible to determine multiple local minima of the pixels of the visual data, connect the local minima, obtain the boundary line of the visual area, and divide the image into multiple visual areas according to the obtained boundary line.

[0096] In actual implementation, feature extraction may be performed on each region included in the partitioned visual data to obtain a third visual feature.

[0097] In actual implementation, since the importance of each visual area in the image is different, more accurate feature extraction is required for important visual areas to obtain more detailed features of the visual area. For unimportant visual areas, only the basic features of the visual area can be extracted without wasting computing resources.

[0098] In actual implementation, in order to determine the importance of each visual area, the attention mechanism can be used to determine the weight of each area. Then, based on the weight of each area, the third visual feature corresponding to each area can be weighted fused to obtain the fusion result. Finally, the third visual feature and the second visual feature are feature fused to obtain the first visual feature.

[0099] In step 103 , feature decoupling is performed on the first visual feature based on the first text feature and the second text feature to obtain a second visual feature related to the attribute information and a third visual feature related to the forgery trace.

[0100] In some embodiments, the feature decoupling of the first visual feature based on the first text feature and the second text feature in step 103 to obtain the second visual feature related to the attribute information and the third visual feature related to the forgery trace can be achieved by the following technical solution: based on the first text feature, the first visual feature is decoded to obtain the second visual feature related to the attribute information; based on the second text feature, the first visual feature is feature decoded to obtain the third visual feature related to the forgery trace.

[0101] In actual implementation, the visual features are first standardized or normalized to facilitate subsequent decoupling operations. After that, the first text feature and the second text feature can be converted into a vector form that can be used for machine learning, and a deep learning model, such as an autoencoder or an adversarial network, is used to learn how to decouple the visual features.

[0102] As an example, two independent encoder and decoder paths can be designed, one for learning features related to the attribute information of the target object, and the other for learning features related to forged traces. Specifically, a shared encoder can be designed, which is responsible for extracting the first visual features of the image, and two different decoders can be designed, one for reconstructing visual features related to person attributes, and the other for reconstructing visual features related to forged traces; then, through adversarial training, the model can be enabled to effectively distinguish between person attribute information and forged traces (specifically, the first text feature can be used as the label of the person attribute information, and the second text feature can be used as the label of the attack clue, and the model can be trained to distinguish between these two types of information). After that, the model can be trained using a labeled image dataset, and the model parameters can be optimized until the model can effectively distinguish and reconstruct the two types of features, and the trained model can be used to encode and decouple the visual features (specifically, two independent decoders can be used to reconstruct features related to person attributes and features related to attack clues, respectively).

[0103] In step 104 , the second text feature and the third visual feature are fused to obtain a first fused feature, and authenticity detection is performed on the image based on the first fused feature to obtain a first detection result.

[0104] In actual implementation, the second text feature and the third visual feature are fused to obtain the first fused feature by splicing the second text feature and the third visual feature to obtain the first fused feature.

[0105] In some embodiments, the authenticity detection of the image based on the first fused feature in step 104 to obtain the first detection result can be achieved by the following technical solution: linearly transforming the first fused feature to obtain a transformed feature; performing nonlinear mapping on the transformed feature to obtain a first detection result, wherein the first detection result is used to indicate whether the image is a forged image, and if the image is a forged image, indicates the forgery type of the forged image.

[0106] In actual implementation, the first fusion feature can be linearly changed by multiplying the first fusion feature with a linear change matrix to obtain a transformation feature to adjust the dimension of the first fusion feature so that the features included in the first fusion feature are in the same dimension.

[0107] In actual implementation, the process of performing nonlinear mapping on the transformed features can be to input the transformed features into an activation function to obtain nonlinear mapped features, and then input the nonlinear mapped features into a classification network, and the classification network outputs a first detection result. When the first detection result characterizes that the image is a forged image, the classification network outputs the forgery type of the forged image, such as forgery, tampering, synthesis, addition, compression, etc.

[0108] See also Figure 4 , Figure 4 This is a diagram of the training process of the image detection model provided in the embodiment of the present application, which will be combined with Figure 4 The steps shown are explained.

[0109] In step 201, a training sample is determined, wherein the training sample is labeled with a first label, a second label, and a third label, the first label indicates first attribute information of an object sample in the training sample, the second label indicates a first forgery trace of the training sample, and the third label indicates whether the training sample is a forged image.

[0110] In actual implementation, the training sample is also an image sample, and the object included in the image sample is the object sample. The first label of the training sample indicates the attribute information of the object sample, the second label indicates the first forgery trace of the training sample (such as whether there is a forgery trace, and if there is a forgery trace, the type of forgery trace), and the third label indicates whether the image sample is a forged image (such as 1 represents a forged image, and 0 represents a non-forged image).

[0111] In step 202, text recognition is performed on the training sample through the text recognition network of the image detection model to obtain a third text and a fourth text, wherein the third text is used to describe the second attribute information of the object sample in the training sample, and the fourth text is used to describe the second forgery trace of the training sample.

[0112] In actual implementation, the image detection model can be as follows Figure 5 The structure shown, Figure 5 This is an architectural diagram of the image detection model provided in an embodiment of the present application. The image detection model includes a text recognition network, which inputs training samples into the text recognition network. The text recognition network can output a third text and a fourth text. Since the training samples often include both attribute information of the object sample and forgery traces related to forgery, the text recognition network can divide the recognized text into a third text related to the attribute information and a fourth text related to the forgery traces.

[0113] In actual application, the text recognition network in the image detection model can be trained separately. Specifically, a text recognition sample can be obtained, and the text recognition sample has a first text label and a second text label. The text recognition sample is input into the text recognition network of the image detection model, and the text recognition network outputs the first predicted text and the second predicted text of the text recognition sample. A first text recognition loss function is constructed based on the first predicted text and the first text label, and a second text recognition loss function is constructed based on the second predicted text and the second text label. Finally, the text recognition network of the image detection model is trained based on the first text recognition loss function and the second text recognition loss function.

[0114] In step 203, feature extraction is performed on the third text through the feature extraction network of the image detection model to obtain a third text feature, feature extraction is performed on the fourth text to obtain a fourth text feature, and visual feature extraction is performed on the training sample to obtain a fourth visual feature.

[0115] In actual implementation, the image detection model can be as follows Figure 5 The structure shown can input the obtained third text and fourth text into the text feature extraction network of the image detection model, and the text feature extraction network outputs the third text feature and the fourth text feature. At the same time, the training sample can be input into the visual feature extraction network in the image detection model, and the visual feature extraction network outputs the fourth visual feature.

[0116] In actual application, the text feature extraction network in the image detection model can be trained separately. Specifically, a labeled text sample can be obtained and the text sample can be input into the text feature extraction network of the image detection model. The text feature extraction network outputs the predicted text features, and a text feature extraction loss function is constructed based on the predicted text features and the labels of the text samples. Finally, the text feature extraction network of the image detection model is trained based on the text feature extraction loss function.

[0117] In actual application, the visual feature extraction network in the image detection model can be trained separately. Specifically, labeled visual samples can be obtained and input into the visual feature extraction network of the image detection model. The visual feature extraction network outputs the predicted visual features, and a visual feature extraction loss function is constructed based on the predicted visual features and the labels of the visual samples. Finally, the visual feature extraction network of the image detection model is trained based on the visual feature extraction loss function.

[0118] In step 204, the fourth visual feature is feature decoupled based on the third text feature and the fourth text feature through the feature decoupling network of the image detection model to obtain a fifth visual feature related to the second attribute information and a sixth visual feature related to the second forgery trace.

[0119] In actual implementation, the image detection model can be as follows Figure 5 The structure shown can input the obtained third text feature, fourth text feature and fourth visual feature into the feature decoupling network of the image detection model, and the feature decoupling network outputs the fifth visual feature and the sixth visual feature.

[0120] In actual application, the feature decoupling network in the image detection model can be trained separately. Specifically, a decoupling network training sample can be constructed first. The training sample is annotated with a first label and a second label. The first label is a text feature used to describe the attribute information of the object sample in the training sample, and the second label is a text feature used to describe the forgery traces of the training sample; the first visual feature of the training sample is decoded by the first decoder to obtain a first decoding feature, and the first visual feature of the training sample is decoded by the second decoder to obtain a second decoding feature; the first decoupling network loss is constructed based on the first decoding feature and the first label of the decoupling network training sample, and the second decoupling network loss is constructed based on the second decoding feature and the second label of the decoupling network training sample. Finally, the decoding model is adversarially trained based on the first decoupling network loss and the second decoupling network loss.

[0121] In step 205, the fourth text feature and the sixth visual feature are fused through the classification network of the image detection model to obtain a second fused feature, and the authenticity of the training sample is detected based on the second fused feature to obtain a second detection result.

[0122] In actual implementation, the image detection model can be as follows Figure 5 The structure shown can input the obtained fourth text feature and sixth visual feature into the classification network of the image detection model, and the feature fusion layer in the classification network fuses the fourth text feature and the sixth visual feature to obtain a second fused feature. The second fused feature is then input into the classification layer in the classification network, and the classification layer outputs the second detection result.

[0123] In actual application, the classification network in the image detection model can be trained separately. Specifically, the classified text samples and classified visual samples of the classification network can be constructed first. The classified text samples and classified visual samples have the same classification labels. The classification labels represent whether the classified samples are forged images. The classified text samples and classified visual samples are input into the classification network, and the classification network outputs the predicted detection results. The classification loss is constructed based on the predicted detection results and the classification labels, and finally the classification network is trained based on the classification loss.

[0124] In step 206 , the image detection model is trained based on the third text, the fourth text, the fifth visual feature, the sixth visual feature, and the second detection result.

[0125] In some embodiments, the training of the image detection model based on the third text, the fourth text, the fifth visual feature, the sixth visual feature and the second detection result in step 206 can be achieved by the following technical solution: constructing a first loss based on the third text and the first label, constructing a second loss based on the fourth text and the second label, and constructing a third loss of the text recognition network based on the first loss and the second loss; constructing a fourth loss based on the text feature of the first attribute information indicated by the fifth visual feature and the first label, constructing a fifth loss based on the text feature of the first forgery trace indicated by the sixth visual feature and the second label, and constructing a sixth loss of the feature decoupling network based on the fourth loss and the fifth loss; constructing a seventh loss of the classification network based on the second detection result and the third label; and training the image detection model based on the third loss, the sixth loss and the seventh loss.

[0126] In actual implementation, the first loss is constructed based on the third text and the first label, the second loss is constructed based on the fourth text and the second label, the fourth loss is constructed based on the text features of the first attribute information indicated by the fifth visual feature and the first label, the fifth loss is constructed based on the text features of the first forgery trace indicated by the sixth visual feature and the second label, and the seventh loss of the classification network is constructed based on the second detection result and the third label. All of these can be constructed using one of the following methods of constructing loss functions: mean square error loss, cross entropy loss, binary cross entropy loss, logarithmic loss, absolute error loss, structural similarity index loss, triangular loss, and focal loss, etc.

[0127] In actual implementation, the third loss of the text recognition network constructed based on the first loss and the second loss can be adopted as follows (1):

[0128] L3=L1+L2 (1)

[0129] In formula (1), L3 is the third loss, L1 is the first loss, and L2 is the second loss.

[0130] In actual implementation, the sixth loss of the feature decoupling network constructed based on the fourth loss and the fifth loss can be adopted as the following formula (2):

[0131] L6=L4+L5 (2)

[0132] In formula (2), L6 is the sixth loss, L4 is the fourth loss, and L5 is the fifth loss.

[0133] In actual implementation, based on the third loss, the sixth loss and the seventh loss, the process of training the image detection model can adopt the comprehensive loss function provided by the following formula (3) to train the image detection model:

[0134] L loss =L3+L6+L7 (3)

[0135] In formula (3), L loss is the comprehensive loss function, L3 is the third loss, L6 is the sixth loss, and L7 is the seventh loss.

[0136] In some embodiments, the following technical solution may also be implemented: determining a test sample, where the test sample satisfies at least one of the following: an object sample in the test sample is different from an object sample in the training sample, the forgery trace of the test sample is different from the forgery trace of the training sample, and the quality of the test sample is different from the quality of the training sample; performing text recognition on the test sample through a text recognition network of an image detection model to obtain a fifth text and a sixth text, wherein the fifth text is used to describe third attribute information of the object sample in the test sample, and the sixth text is used to describe a third forgery trace of the test sample; performing feature extraction on the fifth text through a feature extraction network of the image detection model to obtain a fifth text feature, performing feature extraction on the sixth text to obtain a sixth text feature, and performing visual feature extraction on the test sample to obtain a seventh visual feature; performing feature decoupling on the seventh visual feature based on the fifth text feature and the sixth text feature through a feature decoupling network of the image detection model to obtain an eighth visual feature related to the third attribute information and a ninth visual feature related to the third forgery trace; performing feature fusion on the sixth text feature and the ninth visual feature through a classification network of the image detection model to obtain a third fused feature, and performing authenticity detection on the test sample based on the third fused feature to obtain a third detection result, so as to evaluate the image detection model based on the third detection result.

[0137] In actual implementation, after the image detection model is trained, in the process of testing the image detection model, in order to test the generalization ability of the image detection model, the object samples, forgery traces and sample quality included in the constructed test samples are different from the training samples in the training image detection model. For example, if the object sample in the training sample is person A, then the object sample in the test sample cannot include person A, and the object sample in the test sample can be person B, person C, etc. For another example, if the forgery trace in the training sample is image splicing, then the forgery trace in the test sample cannot include image splicing, and can be image synthesis, image tampering, etc. For another example, if the sample quality in the training sample is the quality of compression level c23, then the sample quality in the test sample can be the quality of compression level c30.

[0138] In actual implementation, the testing process of the image detection model can be as follows: input the test sample into the text recognition network of the image detection model to obtain the fifth text and the sixth text, and then use the text feature extraction network of the image detection model to extract the fifth text feature of the fifth text and the sixth text feature of the sixth text, and extract the seventh visual feature of the test sample through the visual feature extraction network of the image detection model, and then input the fifth text feature, the sixth text feature and the seventh visual feature into the feature decoupling network of the image detection model to obtain the eighth visual feature related to the third attribute information and the ninth visual feature related to the third forgery trace, and then use the classification network of the image detection model to fuse the sixth text feature and the ninth visual feature related to the forgery trace to obtain the third fused feature, and the classification network of the image detection model outputs the third detection result corresponding to the third fused feature, and finally the image detection model is evaluated by comparing the third detection result with the test label of the test sample.

[0139] Below, exemplary applications of the embodiments of the present application in actual application scenarios will be described.

[0140] First, training samples need to be constructed. Specifically, fine-grained text generation is performed on the data used to construct the training samples. The small sample learning ability of the existing large image understanding model can be used to achieve semi-automatic text annotation. For each different type of attack, 10 samples can be randomly selected and a question template is defined. Known information such as sample category, attack type (i.e., the aforementioned forgery traces) and basic attributes of the sample (i.e., the aforementioned attribute information) (gender, age, race, whether wearing glasses, posture, lighting, etc.) are embedded in the question. In the answer template, the given basic information and the attack clues in the image are described. These question-answer pairs and the corresponding images are used as examples of a small number of samples, enabling the large image understanding model to refer to the given samples and complete the automatic annotation of other samples of that type.

[0141] Specific participation Figure 6 , Figure 6 It is a schematic diagram of the labeled sample provided in the embodiment of the present application.

[0142] exist Figure 6 In the figure, image 601 is a sample with manual annotations, 602 is the annotation content of the sample, the manually annotated sample is input into the image understanding model, and the image understanding model learns the annotation method. Then, the annotated image 603 is input into the image understanding model, and the image understanding model outputs the sample 604 after annotating image 603.

[0143] The resulting fine-grained text description can then be divided into a description of the character's basic attributes (i.e., the first text mentioned above) (gender, age, race, skin color, lighting, expression, posture, etc., which are unrelated to the authenticity of the image) and a description of the attack clues (i.e., the second text mentioned above) (which is directly related to the authenticity of the image);

[0144] join Figure 7 , Figure 7 Schematic diagram of the text recognition network provided by the embodiment of the present application. Using the constructed training samples, an image description generation model (i.e., the text recognition network of the above-mentioned image detection model) is trained. The model has two independent outputs, one for outputting text descriptions that are irrelevant to image classification (person attributes) (i.e., Figure 7 The text 1 in , that is, the first text mentioned above) and the text description that is irrelevant to the classification (attack clues) (that is, Figure 7 The image description generation model can adopt the network structure of convolutional network + long short-term memory network to obtain better performance. The loss function of the image description generation model adopts cross entropy loss to optimize the difference between the generated text and the real text label.

[0145] join Figure 8 , Figure 8 : This is a working principle diagram of the image detection model provided in the embodiment of the present application. A pre-trained text feature extraction model (i.e., the feature extraction network in the above-mentioned image detection model), such as a contrastive learning inference projection model or a long sequence contrastive learning inference projection model, is used to extract text features, and two groups of text features TF1 (i.e., the third text feature mentioned above) and TF2 (i.e., the fourth text feature mentioned above) are obtained respectively.

[0146] The image itself passes through a visual feature extractor (i.e., the feature decoupling network in the above-mentioned image detection model), which can be a convolutional neural network structure. The output feature L2 passes through a nonlinear feature transformation module to obtain two sets of image features IF1 (i.e., the seventh visual feature mentioned above) and IF2 (i.e., the eighth visual feature mentioned above).

[0147] In the visual feature extractor, text-guided supervised training can be used to achieve feature decoupling: feature TF1 extracted from text related to person attributes is used to supervise image feature IF1, and feature TF2 extracted from text related to true and false face classification (attack clues) is used to supervise image feature IF2. Two cross-entropy losses are used to constrain the features so that the image features are decoupled: IF1 is related to person attributes, while IF2 is related to true and false face classification.

[0148] Finally, we can use only the image features related to classification (i.e., the ninth visual feature) and text features (i.e., the sixth text feature) to concatenate and obtain fused features (i.e., the third fused feature) and input them into a multi-classification head for classification supervision training. The overall loss function is shown in the following formula (4):

[0149] L loss =L3+L6+L7 (3)

[0150] In formula (3), L loss is a comprehensive loss function, L3 is the classification-independent feature decoupling supervision loss (i.e., the third loss mentioned above), L6 is the classification-related feature decoupling supervision loss (i.e., the sixth loss mentioned above), and L7 is the multi-classification cross entropy loss (i.e., the seventh loss mentioned above).

[0151] During the testing of image detection models, the generalization ability of the model across attack methods and quality is usually tested. Therefore, two protocols that are more in line with actual application scenarios can be defined to fully evaluate the performance of the detection model in real scenarios, as follows Figure 9 As shown, Figure 9 Schematic diagram of the algorithm evaluation protocol provided in the embodiment of the present application.

[0152] exist Figure 9 In the data, the 2000, 500, and 2500 corresponding to identities represent the number of target objects in different data subsets (training / validation / testing). The target identities in different subsets are unique, assessing the system's cross-identity detection capabilities. The numbers 1, 2, 6, and 23 in the Type (authentic, presentational, adversarial, deepfake) represent the specific methods included in each attack type. The "√" in the Quality column indicates the quality of the sample. The last column (Total) shows the total number of samples for each test subset and protocol.

[0153] In the P1 protocol, all subjects were divided into three independent groups. The adversarial sample attacks and deep fake attacks were divided into two groups according to the time when the attack algorithms were proposed. They were used for training / validation and testing respectively to test the generalization ability of the detection model in the face of different object samples (i.e., cross-identity) and the generalization ability in the face of different types of forgery traces.

[0154] In the P2 protocol (i.e., P2.1 and P2.2 in the figure): Based on protocol P1, testing is performed using test sets with different compression levels (c30) and compression methods (h5). Specifically, the training set is of c23 quality, while the test sets are of c30 and h5 quality, respectively. This verifies the image detection model's generalization ability across different object samples, different types of forgery traces, and varying sample quality (i.e., varying compression levels).

[0155] So far, the image detection method provided by the embodiment of the present application has been described in combination with the exemplary application and implementation of the electronic device provided by the embodiment of the present application. The following will continue to describe the cooperation of various modules in the image detection device 555-1 provided by the embodiment of the present application to implement the image detection scheme.

[0156] a recognition module 5551 configured to perform first text recognition on an image to obtain first text, and to perform second text recognition on the image to obtain second text, wherein the first text is used to describe attribute information of a target object in the image, and the second text is used to describe traces of forgery in the image;

[0157] An extraction module 5552 is configured to perform feature extraction on the first text to obtain a first text feature, perform feature extraction on the second text to obtain a second text feature, and perform visual feature extraction on the image to obtain a first visual feature;

[0158] a decoupling module 5553 configured to perform feature decoupling on the first visual feature based on the first text feature and the second text feature to obtain a second visual feature related to the attribute information and a third visual feature related to the forgery trace;

[0159] The fusion module 5554 is configured to perform feature fusion on the second text feature and the third visual feature to obtain a first fusion feature, and perform authenticity detection on the image based on the first fusion feature to obtain a first detection result.

[0160] In some embodiments, the above-mentioned recognition module 5551 is also used to perform attribute recognition on the target object in the image, obtain attribute information of the target object, and generate a first text based on the attribute information; perform forgery trace recognition on the image, obtain forgery traces of the image, and generate a second text based on the forgery traces.

[0161] In some embodiments, the above-mentioned recognition module 5551 is also used to perform object recognition on the image to obtain the target object included in the image; perform attribute feature extraction on the target object to obtain the attribute features of the target object; perform attribute recognition on the target object based on the attribute features to obtain the attribute information of the target object, and generate a first text based on the attribute information.

[0162] In some embodiments, the above-mentioned recognition module 5551 is also used to identify traces of forgery on the image through at least one of the following operations: performing metadata analysis on the image to obtain metadata analysis results; performing color feature extraction on the image to obtain color features of the image, and determining the color distribution of the image based on the color features; performing texture feature extraction on the image to obtain texture features of the image, and identifying the texture pattern of the image based on the texture features; performing compression trace detection on the image to obtain compression traces in the image; performing lighting consistency detection on the image to obtain lighting effects in the image; performing object consistency detection on the image to obtain detection results of the object; determining traces of forgery on the image based on at least one of the metadata analysis results, the color distribution, the texture pattern, the compression traces, the lighting effects, and the detection results of the object; and generating a second text based on the forgery traces.

[0163] In some embodiments, the above-mentioned extraction module 5552 is also used to divide the first text into segments to obtain multiple text segments; perform text feature extraction on each of the text segments to obtain text features of each of the text segments; determine the weight of each of the text segments, and based on the weight, perform feature fusion on the text features of each of the text segments to obtain the first text feature.

[0164] In some embodiments, the extraction module 5552 is further used to extract visual features from the image to obtain a second visual feature; partition the image to obtain a plurality of visual areas, and extract visual features from each of the visual areas to obtain a third visual feature of each of the visual areas; based on the second visual feature, perform weight estimation on each of the visual areas to obtain a weight of each of the visual areas; based on the weight of each of the visual areas, perform feature fusion on each of the third visual features and the second visual feature to obtain the first visual feature.

[0165] In some embodiments, the above-mentioned decoupling module 5553 is also used to decode the first visual feature based on the first text feature to obtain a second visual feature related to the attribute information; and to feature decode the first visual feature based on the second text feature to obtain a third visual feature related to the forgery trace.

[0166] In some embodiments, the above-mentioned fusion module 5554 is also used to perform a linear transformation on the first fusion feature to obtain a transformation feature; and perform a nonlinear mapping on the transformation feature to obtain a first detection result, wherein the first detection result is used to indicate whether the image is a forged image, and if the image is a forged image, indicate the forgery type of the forged image.

[0167] The following further describes the cooperation of various modules in the image detection model training device 555-2 provided in the embodiment of the present application to implement the image detection solution.

[0168] a determination module 5555 configured to determine a training sample, wherein the training sample is annotated with a first label, a second label, and a third label, wherein the first label indicates first attribute information of an object sample in the training sample, the second label indicates a first forgery trace of the training sample, and the third label indicates whether the training sample is a forged image;

[0169] a recognition module 5556 configured to perform text recognition on the training sample using a text recognition network of the image detection model to obtain a third text and a fourth text, wherein the third text is used to describe second attribute information of the object sample in the training sample, and the fourth text is used to describe a second forgery trace of the training sample;

[0170] an extraction module 5557 configured to perform feature extraction on the third text to obtain a third text feature, perform feature extraction on the fourth text to obtain a fourth text feature, and perform visual feature extraction on the training sample to obtain a fourth visual feature, using a feature extraction network of the image detection model;

[0171] a decoupling module 5558 configured to perform feature decoupling on the fourth visual feature based on the third text feature and the fourth text feature using a feature decoupling network of the image detection model to obtain a fifth visual feature related to the second attribute information and a sixth visual feature related to the second forgery trace;

[0172] a fusion module 5559 configured to perform feature fusion on the fourth text feature and the sixth visual feature through the classification network of the image detection model to obtain a second fused feature, and perform authenticity detection on the training sample based on the second fused feature to obtain a second detection result;

[0173] The training module 5550 is used to train the image detection model based on the third text, the fourth text, the fifth visual feature, the sixth visual feature and the second detection result.

[0174] In some embodiments, the above-mentioned training module 5550 is also used to construct a first loss based on the third text and the first label, construct a second loss based on the fourth text and the second label, and construct a third loss of the text recognition network based on the first loss and the second loss; construct a fourth loss based on the fifth visual feature and the text feature of the first attribute information indicated by the first label, construct a fifth loss based on the sixth visual feature and the text feature of the first forgery trace indicated by the second label, and construct a sixth loss of the feature decoupling network based on the fourth loss and the fifth loss; construct a seventh loss of the classification network based on the second detection result and the third label; and train the image detection model based on the third loss, the sixth loss and the seventh loss.

[0175] In some embodiments, the training module 5550 is further used to determine a test sample, wherein the test sample satisfies at least one of the following conditions: the object sample in the test sample is different from the object sample in the training sample, the forgery trace of the test sample is different from the forgery trace of the training sample, and the quality of the test sample is different from the quality of the training sample; the text recognition network of the image detection model is used to perform text recognition on the test sample to obtain a fifth text and a sixth text, wherein the fifth text is used to describe the third attribute information of the object sample in the test sample, and the sixth text is used to describe the third forgery trace of the test sample; the feature extraction network of the image detection model is used to perform feature extraction on the fifth text to obtain a fifth text. Text features, feature extraction is performed on the sixth text to obtain a sixth text feature, and visual feature extraction is performed on the test sample to obtain a seventh visual feature; through the feature decoupling network of the image detection model, feature decoupling is performed on the seventh visual feature based on the fifth text feature and the sixth text feature to obtain an eighth visual feature related to the third attribute information and a ninth visual feature related to the third forgery trace; through the classification network of the image detection model, feature fusion is performed on the sixth text feature and the ninth visual feature to obtain a third fusion feature, and authenticity detection is performed on the test sample based on the third fusion feature to obtain a third detection result, so as to evaluate the image detection model based on the third detection result.

[0176] The present invention provides a computer program product including a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the image detection method described in the present invention.

[0177] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the image detection method provided by the embodiment of the present application, for example, Figure 3A The image detection method is shown.

[0178] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0179] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0180] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0181] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0182] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. An image detection method, characterized in that the method include: Performing first text recognition on an image to obtain a first text, and performing second text recognition on the image to obtain a second text, wherein the first text is used to describe attribute information of a target object in the image, and the second text is used to describe traces of forgery in the image; Performing feature extraction on the first text to obtain a first text feature, performing feature extraction on the second text to obtain a second text feature, and performing visual feature extraction on the image to obtain a first visual feature; performing feature decoupling on the first visual feature based on the first text feature and the second text feature to obtain a second visual feature related to the attribute information and a third visual feature related to the forgery trace; The second text feature and the third visual feature are fused to obtain a first fused feature, and authenticity detection is performed on the image based on the first fused feature to obtain a first detection result.

2. The method according to claim 1, characterized in that The performing first text recognition on the image to obtain the first text includes: Performing object recognition on the image to obtain a target object included in the image; Extracting attribute features of the target object to obtain attribute features of the target object; Performing attribute recognition on the target object based on the attribute features to obtain attribute information of the target object; A first text is generated based on the attribute information.

3. The method according to claim 1, characterized in that The performing second text recognition on the image to obtain the second text includes: Identify forgery traces on the image by performing at least one of the following operations: Performing metadata analysis on the image to obtain metadata analysis results; performing color feature extraction on the image to obtain color features of the image, and determining color distribution of the image based on the color features; Extracting texture features of the image to obtain texture features of the image, and identifying a texture pattern of the image based on the texture features; Performing compression trace detection on the image to obtain compression traces in the image; Performing a lighting consistency test on the image to obtain a lighting effect in the image; Performing object consistency detection on the image to obtain a detection result of the object; determining, based on at least one of the metadata analysis result, the color distribution, the texture pattern, the compression traces, the lighting effect, and the object detection result, whether the image is forged; A second text is generated based on the forgery trace.

4. The method according to claim 1, wherein The extracting features of the first text to obtain first text features includes: Segmenting the first text to obtain multiple text segments; Performing text feature extraction on each of the text segments to obtain text features of each of the text segments; The weight of each of the text segments is determined, and based on the weight, the text features of each of the text segments are fused to obtain a first text feature.

5. The method according to claim 1, wherein The extracting visual features from the image to obtain the first visual feature includes: performing visual feature extraction on the image to obtain a second visual feature; Partitioning the image to obtain a plurality of visual areas, and extracting visual features from each of the visual areas to obtain a third visual feature of each of the visual areas; Based on the second visual feature, weight estimation is performed on each of the visual areas to obtain a weight of each of the visual areas; Based on the weight of each visual area, feature fusion is performed on each third visual feature and the second visual feature to obtain the first visual feature.

6. The method according to claim 1, wherein The performing feature decoupling on the first visual feature based on the first text feature and the second text feature to obtain a second visual feature related to the attribute information and a third visual feature related to the forgery trace includes: decoding the first visual feature based on the first text feature to obtain a second visual feature related to the attribute information; Based on the second text feature, the first visual feature is feature decoded to obtain a third visual feature related to the forgery trace.

7. The method according to claim 1, characterized in that The performing authenticity detection on the image based on the first fusion feature to obtain a first detection result includes: Performing a linear transformation on the first fused feature to obtain a transformed feature; Nonlinear mapping is performed on the transformed features to obtain a first detection result, wherein the first detection result is used to indicate whether the image is a forged image, and if the image is a forged image, indicates the forgery type of the forged image.

8. A training method for an image detection model, characterized in that: The method comprises: Determining a training sample, wherein the training sample is labeled with a first label, a second label, and a third label, the first label indicating first attribute information of an object sample in the training sample, the second label indicating a first forgery trace of the training sample, and the third label indicating whether the training sample is a forged image; Performing text recognition on the training sample using a text recognition network of the image detection model to obtain a third text and a fourth text, wherein the third text is used to describe second attribute information of the object sample in the training sample, and the fourth text is used to describe the second forgery trace of the training sample; Performing feature extraction on the third text to obtain a third text feature, performing feature extraction on the fourth text to obtain a fourth text feature, and performing visual feature extraction on the training sample to obtain a fourth visual feature, using a feature extraction network of the image detection model; performing feature decoupling on the fourth visual feature based on the third text feature and the fourth text feature using a feature decoupling network of the image detection model to obtain a fifth visual feature related to the second attribute information and a sixth visual feature related to the second forgery trace; Performing feature fusion on the fourth text feature and the sixth visual feature to obtain a second fused feature through the classification network of the image detection model, and performing authenticity detection on the training sample based on the second fused feature to obtain a second detection result; The image detection model is trained based on the third text, the fourth text, the fifth visual feature, the sixth visual feature and the second detection result.

9. The method according to claim 8, characterized in that The training of the image detection model based on the third text, the fourth text, the fifth visual feature, the sixth visual feature, and the second detection result includes: Constructing a first loss based on the third text and the first label, constructing a second loss based on the fourth text and the second label, and constructing a third loss of the text recognition network based on the first loss and the second loss; constructing a fourth loss based on the fifth visual feature and the text feature of the first attribute information indicated by the first label, constructing a fifth loss based on the sixth visual feature and the text feature of the first forgery trace indicated by the second label, and constructing a sixth loss of the feature decoupling network based on the fourth loss and the fifth loss; constructing a seventh loss of the classification network based on the second detection result and the third label; The image detection model is trained based on the third loss, the sixth loss, and the seventh loss.

10. The method according to claim 8, characterized in that After training the image detection model, the method further includes: Determine a test sample, where the test sample satisfies at least one of the following: an object sample in the test sample is different from an object sample in the training sample, a forgery trace of the test sample is different from a forgery trace of the training sample, and a quality of the test sample is different from a quality of the training sample; Performing text recognition on the test sample using a text recognition network of the image detection model to obtain a fifth text and a sixth text, wherein the fifth text is used to describe third attribute information of the object sample in the test sample, and the sixth text is used to describe a third forgery trace of the test sample; Performing feature extraction on the fifth text to obtain a fifth text feature, performing feature extraction on the sixth text to obtain a sixth text feature, and performing visual feature extraction on the test sample to obtain a seventh visual feature, using a feature extraction network of the image detection model; performing feature decoupling on the seventh visual feature based on the fifth text feature and the sixth text feature using a feature decoupling network of the image detection model to obtain an eighth visual feature related to the third attribute information and a ninth visual feature related to the third forgery trace; Through the classification network of the image detection model, the sixth text feature and the ninth visual feature are fused to obtain a third fused feature, and the authenticity of the test sample is detected based on the third fused feature to obtain a third detection result, so as to evaluate the image detection model based on the third detection result.

11. An image detection device, characterized in that: The device comprises: a recognition module, configured to perform first text recognition on an image to obtain a first text, and perform second text recognition on the image to obtain a second text, wherein the first text is used to describe attribute information of a target object in the image, and the second text is used to describe traces of forgery in the image; an extraction module, configured to perform feature extraction on the first text to obtain a first text feature, perform feature extraction on the second text to obtain a second text feature, and perform visual feature extraction on the image to obtain a first visual feature; a decoupling module, configured to perform feature decoupling on the first visual feature based on the first text feature and the second text feature to obtain a second visual feature related to the attribute information and a third visual feature related to the forgery trace; The fusion module is used to perform feature fusion on the second text feature and the third visual feature to obtain a first fusion feature, and perform authenticity detection on the image based on the first fusion feature to obtain a first detection result.

12. A training device for an image detection model, characterized in that: The device comprises: a determination module, configured to determine a training sample, wherein the training sample is annotated with a first label, a second label, and a third label, wherein the first label indicates first attribute information of an object sample in the training sample, the second label indicates a first forgery trace of the training sample, and the third label indicates whether the training sample is a forged image; a recognition module, configured to perform text recognition on the training sample using a text recognition network of the image detection model to obtain a third text and a fourth text, wherein the third text is used to describe second attribute information of the object sample in the training sample, and the fourth text is used to describe a second forgery trace of the training sample; an extraction module, configured to perform feature extraction on the third text to obtain a third text feature, perform feature extraction on the fourth text to obtain a fourth text feature, and perform visual feature extraction on the training sample to obtain a fourth visual feature, using a feature extraction network of the image detection model; a decoupling module configured to perform feature decoupling on the fourth visual feature based on the third text feature and the fourth text feature using a feature decoupling network of the image detection model to obtain a fifth visual feature related to the second attribute information and a sixth visual feature related to the second forgery trace; a fusion module, configured to perform feature fusion on the fourth text feature and the sixth visual feature through a classification network of the image detection model to obtain a second fused feature, and perform authenticity detection on the training sample based on the second fused feature to obtain a second detection result; A training module is used to train the image detection model based on the third text, the fourth text, the fifth visual feature, the sixth visual feature and the second detection result.

13. An electronic device, characterized in that: include: a memory for storing computer-executable instructions or computer programs; A processor, configured to implement the image detection method described in any one of claims 1 to 7 or the training method of the image detection model described in any one of claims 8 to 10 when executing the computer-executable instructions or computer program stored in the memory.

14. A computer-readable storage medium, characterized in that Computer-executable instructions or computer programs are stored, and when the computer-executable instructions or computer programs are executed by a processor, the image detection method described in any one of claims 1 to 7 or the training method of the image detection model described in any one of claims 8 to 10 are implemented.

15. A computer program product comprising a computer program or computer executable instructions, characterized in that When the computer program or computer executable instructions are executed by a processor, the image detection method described in any one of claims 1 to 7 or the training method of the image detection model described in any one of claims 8 to 10 is implemented.

Images