Image processing method and device

Abstract

The invention discloses an image processing method and device, and the method comprises the steps: carrying out the recognition processing of a target image in response to a target triggering operation, and enabling the target image to be an image inputted to a target application or an image currently displayed by the target application, the target application is an application capable of executing the identification processing or calling a target program file to execute the identification processing; and outputting a recognition result for the target image, wherein the recognition result at least can indicate source information of at least part of image content of the target image. According to the invention, authenticity and source information of at least part of image content of the image can be identified more accurately, so that a richer and more credible identification result is provided for a user, and the identification reliability and interactivity are improved.

Description

Technical Field

[0001] The present application relates to an image processing method and device. Background Art

[0002] The rapid development of a new generation of artificial intelligence technology, represented by generative artificial intelligence technology, has enabled people to easily generate a high-quality image or edit and tamper with a real image, thereby producing deep fake images. These high-quality deep fake images are highly realistic, and it is usually difficult for people to tell whether the target image is a real image or a deep fake image with the naked eye.

[0003] Deep fake technology and deep fake images may be abused by criminals, which poses a great threat to user safety. Taking video conferencing scenarios as an example, users cannot accurately distinguish the authenticity of the characters in images and videos, which makes users trust the other party, leading to adverse consequences such as property loss or even endangerment of life safety. Summary of the invention

[0004] The purpose of the embodiment of the present application is to provide an image processing method, including:

[0005] In response to a target trigger operation, performing recognition processing on a target image, the target image being an image input to a target application or an image currently displayed by the target application, the target application being an application capable of executing the recognition processing or calling a target program file to execute the recognition processing;

[0006] A recognition result for the target image is output, where the recognition result can at least indicate source information of at least a portion of image content of the target image.

[0007] As an optional embodiment, in response to the target triggering operation, the target image is identified and processed, including one of the following:

[0008] In response to obtaining an operation of a target user inputting a target image into a target application, performing recognition processing on the target image;

[0009] In response to obtaining target question information for a target image displayed by a target application, performing recognition processing corresponding to the target question information on the target image;

[0010] In response to obtaining a selection operation of a question option for a target image displayed by a target application, performing recognition processing corresponding to the selection operation on the target image;

[0011] In response to obtaining an operation of a target user inputting a target image into a target application and target question information for the target image, performing recognition processing corresponding to the target question information on the target image;

[0012] In response to detecting a change in usage information of the electronic device, a target image displayed by a target application is identified.

[0013] As an optional embodiment, the target image is recognized and processed, including:

[0014] Acquire image feature data of the target image, wherein the image feature data includes overall information and local information of the target image;

[0015] In the case where the target question information is not obtained, the image feature data is inferred to obtain the recognition result of the target image; or,

[0016] When target question information is obtained, text feature data of the target question information is obtained, the image feature data and the text feature data are fused to obtain fused feature data, and the fused feature data is inferred to obtain a recognition result for the target image.

[0017] As an optional embodiment, obtaining image feature data of the target image includes:

[0018] Dividing the target image into a plurality of image blocks, and processing the pixel points of the image blocks into image pixel sequences;

[0019] Concatenating the image pixel sequence with the position coding information corresponding to each image block to obtain a first image block;

[0020] Acquire a second image block representing source information;

[0021] The first image block and the second image block are encoded using a first encoder to obtain the image feature data.

[0022] As an optional embodiment, the inference of the image feature data to obtain the recognition result of the target image includes at least one of the following:

[0023] Inputting the image features into a classification model composed of a multi-layer perceptron and a first activation function to obtain source information of at least part of the image content of the target image;

[0024] The image features are input into a multi-layer perceptron and an output layer formed by a second activation function to obtain source information of at least a portion of the image content of the target image and an explanation of the source information.

[0025] As an optional embodiment, extracting text feature data of the target question information includes:

[0026] Segmenting the target question information into a text character sequence based on semantic information of the target question information, wherein a mark separator is inserted into the text character sequence;

[0027] Representing the text character sequence and the token separator as word vector data;

[0028] The word vector data is subjected to word embedding encoding processing to output the text feature data.

[0029] As an optional embodiment, the inference of the fused feature data to obtain a recognition result for the target image includes:

[0030] The fused feature data is decoded using an autoregressive decoding module to obtain response information for question information regarding at least a portion of the image content of the target image.

[0031] As an optional embodiment, outputting the recognition result for the target image includes at least one of the following:

[0032] Outputting source information of at least a portion of the image content of the target image and an explanation of the source information;

[0033] Outputting source information for at least a portion of the image content of the target image and a first control, wherein the first control can be triggered to display an explanation for the source information;

[0034] Outputting source information of at least a portion of the image content of the target image, and, when the source information is triggered, outputting an explanation of the source information;

[0035] Outputting source information for at least a portion of image content of the target image and first response information for first question information for at least a portion of the image content;

[0036] The source information of at least a part of the image content of the target image and a second control are output, and the second control can be triggered to display the associated information of the source information.

[0037] As an optional embodiment, outputting a recognition result for the target image includes:

[0038] Outputting source information of at least a portion of image content of the target image;

[0039] Target content data associated with the source information is output based on the category information of the source information.

[0040] The embodiment of the present application aims to provide an image processing device, comprising:

[0041] a recognition module configured to perform recognition processing on a target image in response to a target trigger operation, wherein the target image is an image input to a target application or an image currently displayed by the target application, and the target application is an application capable of performing the recognition processing or calling a target program file to perform the recognition processing;

[0042] An output module is configured to output a recognition result for the target image, wherein the recognition result can at least indicate source information of at least a portion of image content of the target image.

[0043] The purpose of an embodiment of the present application is to also provide an electronic device, including: a memory and a processor, wherein an executable program is stored in the memory, and the processor executes the executable program to implement the steps of any method provided in the embodiment of the present application.

[0044] The purpose of the embodiments of the present application is to provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any image processing method provided in the embodiments of the present application are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flowchart of a graphics processing method according to an embodiment of the present application;

[0046] Figure 2 This is a flowchart of step S10 of the graphics processing method according to an embodiment of the present application;

[0047] Figure 3 This is a flowchart of step S101 of the graphics processing method of an embodiment of the present application;

[0048] Figure 4 This is a flowchart of step S103 of the graphics processing method of an embodiment of the present application;

[0049] Figure 5 This is a flowchart of step S20 of the graphics processing method according to an embodiment of the present application;

[0050] Figure 6 is a structural block diagram of a graphics processing device according to an embodiment of the present application;

[0051] Figure 7 The process of the graphics processing method of the embodiment of the present application is as follows Figure 1 ;

[0052] Figure 8 The process of the graphics processing method of the embodiment of the present application is as follows Figure 2 ;

[0053] Fig. 9 This is a flowchart of obtaining image feature data according to an embodiment of the present application;

[0054] Fig.10 A flowchart of obtaining image classification according to an embodiment of the present application;

[0055] Fig.11 The process of obtaining text feature data in the embodiment of the present application Figure 1 ;

[0056] Fig.12 The process of obtaining text feature data in the embodiment of the present application Figure 2 ;

[0057] Fig.13 This is a flowchart of obtaining fusion feature data according to an embodiment of the present application;

[0058] Fig.14 This is a flow chart of autoregressive decoding fusion feature data in an embodiment of the present application;

[0059] Fig.15 A flowchart of the self-attention mechanism of an embodiment of the present application;

[0060] Fig.16 A flowchart of the causal attention mechanism of an embodiment of the present application;

[0061] Fig.17 A flowchart of a cross-attention mechanism according to an embodiment of the present application;

[0062] Fig.18 A schematic diagram of a graphics processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0063] Various aspects and features of the present application are described herein with reference to the accompanying drawings.

[0064] It should be understood that various modifications may be made to the embodiments of the present application. Therefore, the above description should not be considered as limiting, but only as an example of an embodiment. Other modifications within the scope and spirit of the present application will occur to those skilled in the art.

[0065] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0066] These and other characteristics of the present application will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.

[0067] It should also be understood that although the present application has been described with reference to some specific examples, those skilled in the art will be able to readily implement many other equivalent forms of the present application.

[0068] The above and other aspects, features and advantages of the present application will become more apparent in view of the following detailed description when taken in conjunction with the accompanying drawings.

[0069] Specific embodiments of the present application are described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments applied for are merely examples of the present application, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that obscure the present application. Therefore, the specific structural and functional details applied for herein are not intended to be limiting, but merely serve as a basis and representative basis for the claims to teach those skilled in the art to use the present application in a variety of ways with substantially any suitable detailed structure.

[0070] This specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present application.

[0071] How to effectively detect and identify deep fake images, and attribute the target images and the technology used to generate the deep fake images, is of great significance to ensuring user safety and protecting network security.

[0072] Currently, when determining whether a target image is a real image or a deep fake image, relevant technologies can determine the authenticity of the target image.

[0073] However, when users face a target image, in addition to wanting to know the authenticity of the target image as a whole, they usually have further, personalized needs. For example, users want to know the authenticity of a local area in the target image, and the reason why the target image is judged to be a real image or a deep fake image. Users also want to ask personalized questions about the authenticity of the target image and expect to get detailed, easy-to-understand answers. These requirements cannot be met by existing technical solutions.

[0074] Therefore, an image processing method according to an embodiment of the present application is as follows: Figure 1 As shown, including:

[0075] S10, in response to a target trigger operation, performing recognition processing on a target image, wherein the target image is an image input to a target application or an image currently displayed by the target application, and the target application is an application capable of executing the recognition processing or calling a target program file to execute the recognition processing;

[0076] In this embodiment, the target trigger operation refers to an action performed by the user or a system event, which triggers the subsequent processing flow. Specifically, the target trigger operation includes but is not limited to:

[0077] 1) User input of target image

[0078] For example, a user clicks the "Upload Picture" button on the interactive interface of the target application or a picture sharing social platform (the platform uses the technical solution disclosed in this application, or the platform is an associated platform of the target application mentioned in this application, and the platform needs to mark the source information of the displayed pictures, and therefore needs to identify the source information of the uploaded pictures), and then selects a picture from the mobile phone album and uploads it to the platform. This upload action can be regarded as a target trigger operation.

[0079] Alternatively, a user may use a mobile phone camera to take a picture of an item in a shopping app and then upload it directly to the app for identification. The action of taking the picture and uploading it triggers image recognition.

[0080] 2) Questioning the target image

[0081] The questioning operation may be in the form of voice questioning operation, text questioning operation, etc. For example, a user enters text in a search bar of an image recognition application: "Is this picture real?" This text input action is an operation of asking a question, which triggers the target application to identify the animal in the uploaded picture.

[0082] The target image refers to the image to be recognized, including the image currently displayed by the target application and the image input to the target application. Specifically, the target image can be an image input by the user or an image currently displayed by an application. For example, a photo uploaded by a user to social media or a picture displayed on a web page.

[0083] 3) Selection operation of question options on the application interface

[0084] The user starts the image-related processing or analysis process by selecting a specific question option on the application interface. The application provides one or more preset questions or question options related to image processing, which are usually related to the image analysis, recognition or search functions of the application. The user tells the application the specific image processing task they want to perform by clicking, touching or selecting these preset question options.

[0085] For example, when a user opens an app, they see a list of options, including "Is this picture real?", "What object is this?", "Where is this scene?", "What device took this picture?", and "Who took this picture?". The user selects the option "Is this picture real?". After receiving the user's selection, the app triggers the image recognition function to analyze the uploaded picture.

[0086] 4) Image recognition based on scene switching or recognition-determined triggers

[0087] It usually refers to automatically starting the image recognition process after a specific scene changes or the target scene is recognized. For example, an intelligent security monitoring system is used as the technical solution of this application. When the surveillance camera captures a scene change, such as a human face suddenly appears in a certain area, the system will automatically start the image recognition process and analyze the captured image to determine whether it is forged, tampered, or generated by an AI model (such as a fake image generated by GAN).

[0088] The recognition process can identify the source of the target image, such as which generation model the target image is generated from, which target camera the target image is generated from, which terminal the target image is generated from, or which place the target image is generated from.

[0089] The target image is an image input to the target application or an image currently displayed by the target application, wherein the target application refers to an application that can perform recognition processing or call other program files to perform recognition processing. For example, a recognition software that can identify the source of an image can be used by the user to identify the source of the object in the picture, such as whether the image is taken from the real physical world or generated from a model, or from which camera, terminal, or place, etc.

[0090] That is, the target image can be an image input by the user into the image authenticity identification application for image identification, or it can be an authenticity identification or image content source identification directly performed on the display content in the application interface of the current display output, such as identification of images displayed in WeChat, a browser, or a website.

[0091] For example, if a user receives a picture in WeChat and wants to verify its authenticity, the user can save the picture and upload it to a dedicated image authenticity identification application (image source identification application) or directly press and hold in the WeChat interface to trigger the image authenticity identification application (image source identification application) to identify the picture. This application helps users identify whether the picture has been tampered with by analyzing the ELA value of the image.

[0092] The target application for performing the recognition processing is an application having the capability of identifying the source of an image, such as image authenticity recognition and / or local authenticity identification.

[0093] The target program file may be an application, a program code or a model file. The target application cannot directly implement image source or authenticity identification, but can implement image authenticity identification or image traceability by calling an image authenticity identification application or code file or model file.

[0094] S20: Output a recognition result for the target image, where the recognition result can at least indicate source information of at least a portion of image content of the target image.

[0095] In this embodiment, the output form of the recognition result obtained after the recognition process is not limited, and can be in the form of text output, image output, voice broadcast, etc.

[0096] The at least part of the image content includes a foreground image or a background image or a part of an object in the foreground or background. For example, if a user uploads a photo of a park, the foreground image is identified as a bench, the background image is identified as a lake in the distance, and the part of the object is identified as a children's slide in the foreground or a boat in the background.

[0097] The source information refers to the source of the target image, such as which generation model or camera or terminal or location the image is generated from, etc. In addition, the source information may also include the name, location, time, etc. of at least part of the target image.

[0098] For example, the output form of the recognition result is text output, and the output content is "the picture is real" or "the foreground image is real and is identified as a German shepherd" or "the image is fake and is forged by the StyleGAN model."

[0099] When using this application, the user clicks the "Upload Image" button in an image recognition application, selects a picture from the mobile phone album (which can be taken by the user or downloaded from the Internet), and uploads it to the platform. This upload action triggers the image recognition process.

[0100] The photo uploaded by the user becomes the target image and is input into the image recognition application. The image recognition application receives the uploaded photo and prepares to perform the recognition process. The application has the ability to perform the recognition process, or can call built-in program files (such as deep learning models) to assist in recognition.

[0101] The app starts analyzing the photo, using a deep learning model to identify the foreground image (such as a bench, a children's slide), the background image (such as a lake, a boat), and the source information of the image (such as the camera model, the location and time of shooting). After the app analyzes the photo, it displays the following text information: the foreground image is fake, the foreground image is generated by the StyleGAN model, and the background image is real.

[0102] The present application can more accurately identify the authenticity and source information of at least part of the image content of an image, so as to provide users with richer and more convincing recognition results and improve recognition reliability and interactivity.

[0103] In one embodiment, in response to the target triggering operation, the target image is identified and processed, including one of the following:

[0104] 1) In response to obtaining an operation of a target user inputting a target image into a target application, performing recognition processing on the target image.

[0105] In this embodiment, the user performs an action, that is, inputting image data into the target application. The target application receives the image data input by the user and processes the received image data to identify the content or features in the image. The operation of the user inputting the target image into the target application is that the user uploads the image to be identified.

[0106] For example, there is an image recognition application that can identify the authenticity of an image, which model or device / photographer it comes from, etc. The user opens the application and clicks the "Select Picture" button on the application interface. The user selects a garden photo from the mobile phone album and uploads it to the application. After the application receives the photo uploaded by the user, it automatically starts analyzing the photo. Using the technical solution of this application, it is recognized that the photo is authentic.

[0107] 2) In response to obtaining target question information for a target image displayed by a target application, performing recognition processing corresponding to the target question information on the target image.

[0108] In this embodiment, the image currently displayed by the target application can be used as the target image, or it can be a specific image, such as an image sent by a specific object, or a specific related image (an image about the Great Wall, a photo of a ** person).

[0109] The currently displayed image can be automatically loaded by the application or the result of the user's previous browsing or operation. For example, a user is using a news reading application to read an article about natural scenery. The article contains a picture of a waterfall. This picture of the waterfall is the image currently displayed by the target application and can be used as the target image for recognition processing, such as identifying the authenticity of the image and identifying which model or device / photographer the image comes from.

[0110] Or, a user receives a photo of a pet dog from a friend via WeChat and asks if the dog's fur color is real. This photo of the pet dog is the image sent by the specific object and can be used as the target image for recognition processing to determine that part of the image (fur color) is fake and was generated by the StyleGAN model.

[0111] Or, if a user is interested in the history of the Great Wall and enters the keyword "Great Wall" in the search engine, multiple pictures about the Great Wall appear in the search results. These images about the Great Wall are specific and relevant images and can be used as target images for recognition processing, such as identifying which model or device / photographer the image comes from.

[0112] The target question information includes questions about the source of the image, questions about the authenticity of the image, etc. Users ask questions about the source of the image, such as who took the image, where the image was taken, when the image was taken, the equipment used to take the image, etc. Users question the authenticity of the image, and want to confirm whether the image has been tampered with or edited, and whether it is the original, unmodified version.

[0113] For example, a user sees an eye-catching news photo on social media and wants to verify whether the source of the photo is reliable. The user may ask: "When and where was this photo taken?"

[0114] Or, if a user sees a photo of a politician on a news site but suspects that the photo may have been manipulated to influence public opinion, the user might ask, “Is this photo original?” or “Has this photo been photoshopped?”

[0115] The recognition process corresponding to the target question information may be a default recognition process or a recognition process for the question information. The default recognition process means that no matter what kind of question the user asks, the system will execute a preset recognition process. For example, the system may perform image authenticity recognition processing on all uploaded images by default.

[0116] In practice, an image processing application is set by default to authenticate all uploaded images. No matter what questions the user asks after uploading the image, the application will first authenticate the image to see if it has been tampered with or edited. For example, a user uploads a landscape photo and asks, "Where was this photo taken?" Even though the user's question has nothing to do with the authenticity of the image, the application will still authenticate the image first and then try to answer the user's specific question.

[0117] The recognition processing for question information means that the system selects the corresponding recognition processing flow according to the specific question of the user. If the user's question involves a specific aspect of the image, the system will perform the recognition processing related to that aspect.

[0118] For example, a user uploads the same photo of a celebrity and asks, “When was this photo taken?” The app recognizes that this is a question about when the image was taken, so it performs different recognition processing, such as reading the image’s metadata or using image content analysis techniques to estimate the capture time.

[0119] 3) In response to obtaining a selection operation of a question option for a target image displayed by a target application, performing recognition processing corresponding to the selection operation on the target image.

[0120] In this embodiment, preset question options are set for users to choose from. The application provides a series of preset question options that cover different recognition requirements, such as image authenticity, building recognition, location recognition, etc. Users select one or more questions from these preset options based on their query requirements for the target image. In response to the user's selection, the application performs recognition processing on the target image corresponding to the question selected by the user.

[0121] The recognition processing corresponding to the selection operation includes but is not limited to image authenticity recognition, building recognition, location recognition, device recognition, and lens recognition.

[0122] For example, an application using the technical solution of the present application provides a series of preset question options for users to choose from to identify and answer questions about images. Preset question options: "Image authenticity identification", "Building identification", "Location identification", "Filming equipment identification", "Lens type identification".

[0123] The user sees a photo of a city landscape in the app and wants to know where the photo was taken. The user selects "Location Identification" from the preset question options. In response to the user's "Location Identification" selection, the app begins geo-tagging the target image, possibly using image recognition technology, geographic information system (GIS) data, and online databases to determine where the photo was taken. After the app completes the analysis, it displays the identification result: "This photo was taken near the Eiffel Tower in Paris."

[0124] 4) In response to obtaining the target user's operation of inputting a target image into a target application and target question information for the target image, performing recognition processing corresponding to the target question information on the target image.

[0125] In this embodiment, the user inputs an image into the target application. This operation may be uploading, shooting or selecting an image. The user asks specific questions about the input image. These questions may be about the content, source, authenticity, etc. of the image. The system performs corresponding recognition processing on the target image based on the user's operation and question information to answer the user's question. That is, the user inputs the image to be recognized and the question information at the same time.

[0126] For example, a user is using an application that adopts the technical solution of the present application, uploads a photo of an ancient statue taken in a museum, and asks: "Is this statue real?" The application responds to the user's question and uses the image processing method of the present application to determine the authenticity of the statue.

[0127] 5) In response to detecting a change in the usage information of the electronic device, identifying and processing a target image displayed by the target application.

[0128] In this embodiment, the usage information can represent the triggering operation that triggers the recognition process, and the change in the usage information includes a change in the usage scenario or a change in the usage object. For example, the usage information can be the state of the device, the environment, or the user's operation. When this information changes, the system automatically performs image processing to recognize and analyze the currently displayed image.

[0129] For example, a smart home security system equipped with cameras to monitor the residence can detect changes in the scene and automatically initiate the image recognition process when specific events are detected.

[0130] When the camera detects someone approaching your home, this is a scene change because the area previously monitored by the camera was empty. The camera automatically captures this scene and starts the image recognition process.

[0131] In one embodiment, if Figure 2 As shown, the target image is recognized and processed, including:

[0132] S101. Acquire image feature data of the target image, where the image feature data includes overall information and local information of the target image.

[0133] In this embodiment, the overall information refers to the overall features of the image, such as global features such as color distribution, texture, shape, etc. Local information refers to the features of a specific area or object in the image, such as detailed features such as the edge, corner point, local texture, etc. of a specific object.

[0134] For example, the target image is a natural landscape photo containing flowers. The overall information may include the overall color tendency of the image (such as green and red as the main colors), the overall texture (such as the texture of the grass and the texture of the flowers). The local information may include the edge contour of the flower, the local texture of the petals, and the detailed features of the stamens.

[0135] S102: In the case where target question information is not obtained, inferring the image feature data to obtain a recognition result of the target image.

[0136] In this embodiment, when only the target image is input, the classification model is directly used for reasoning, the content of the image is inferred based on the extracted image feature data, and a recognition result is generated. For example, continuing with the above example of the flower photo, if there is no specific question information, the system may use a pre-trained classification model to identify the authenticity of the image and output the recognition result: "The image is forged." Or, according to the recognition result output specification preset by the pre-trained classification model, it is output whether the target image is a photographed image or an image generated based on a certain generation model.

[0137] In this embodiment, it is also possible to consider using an autoregressive decoding model to output source information and corresponding reasons.

[0138] S103, or, in the case of obtaining target question information, obtaining text feature data of the target question information, fusing the image feature data and the text feature data to obtain fused feature data, and inferring the fused feature data to obtain a recognition result for the target image.

[0139] In this embodiment, if Figure 8 As shown, when the user inputs the target image and question information at the same time, the image feature data is obtained through image encoding, and the text feature data is obtained through word embedding representation and word embedding encoding module.

[0140] The image feature data and text feature data are fused by the image-text feature fusion module to generate fusion features. The image-text feature fusion module associates the semantic information in the text question with the overall and key local information of the target image to support text questions about the overall and local target image.

[0141] The autoregressive decoding module performs reasoning based on the fused feature data, generates reasons corresponding to the image category judgment in the form of text sentences, and outputs a text answer.

[0142] In other embodiments, the question information may also include speech-to-text processing or image-to-text processing or semantic recognition to obtain text input.

[0143] For example, a user uploads a photo of multiple flowers and asks, "Is this red flower real?" The system first extracts the text feature data of the question and then fuses it with the local information of the image (such as the red flower). Through this fusion process, the system can more accurately identify the source information of the red flower and answer the user's question: "This red flower is real." Or, "This red flower comes from Lenovo's Creator zone."

[0144] This application can handle the case where there are only images (directly reasoning through the classification model), and can also handle the case where there are both images and text question information (by fusing image and text feature data for reasoning). This flexibility enables the system to understand and respond to user needs more comprehensively.

[0145] In one embodiment, if Figure 3 As shown, obtaining the image feature data of the target image includes:

[0146] S1011, dividing the target image into a plurality of image blocks, and processing the pixel points of the image blocks into image pixel sequences.

[0147] In this embodiment, the entire image is usually divided into 9 small blocks, each of which is called an image block. Then, the pixels in each image block are processed into a sequence, i.e., an image pixel sequence. This can provide more fine-grained data for subsequent feature extraction and analysis.

[0148] S1012: Concatenate the image pixel sequence and the position coding information corresponding to each image block to obtain a first image block.

[0149] In this embodiment, the image pixel sequence of each image block is combined with its corresponding position coding information (i.e., the position of the image block in the original image) to form a new data structure, called the first image block. The position coding information helps to maintain the spatial relationship of the image blocks in subsequent processing.

[0150] S1013: Obtain a second image block representing source information.

[0151] In this embodiment, a learnable category embedding is used to represent the image classification task, and the category embedding is concatenated with the index number 0 to generate an embedded image block representing the category information.

[0152] Category embedding is a technique that converts discrete category labels into continuous vectors. In image classification tasks, this means mapping each category (such as different objects or scenes) to a point in a high-dimensional space that captures the characteristics of the category. By concatenating the category embedding with the index number 0, we can create an embedded image patch that represents the category information, which can be used to train a model to recognize the category in an image.

[0153] S1014: Utilize a first encoder to encode the first image block and the second image block to obtain the image feature data.

[0154] In this embodiment, the first encoder is used to encode the first image block (including the pixel sequence and position encoding information of the image block) and the second image block (including the source information). This process extracts the feature data of the image, which can be used for subsequent image recognition, classification or other analysis tasks.

[0155] This application obtains image feature data through image encoding, such as Fig. 9 As shown, the target image X generates image feature I through the image coding module, and the target image X is evenly divided into N image blocks, each image block corresponds to a position index number, and all the pixels in each image block are tiled and linearly projected to generate a corresponding image pixel sequence. The position index number of each image block is positionally encoded to generate position coding information, and the image pixel sequence and the position coding information are cascaded to form an embedded image block containing position information.

[0156] In order to introduce image category information into the image encoding process, a learnable category embedding is used to represent the image classification task. The category embedding is concatenated with the index number 0 to generate an embedded image block representing the category information.

[0157] Specifically, the target image is first divided into 9 image blocks, and the RGB values ​​of all pixels in each image block are tiled and stacked to form a vector. The length of each vector is L. The vectors of all N image blocks are combined to form a matrix with N rows and L columns. The matrix is ​​transformed by linear projection to generate a new matrix. The number of rows of the new matrix is ​​still N, but the number of columns becomes D, and the value of D is usually less than L. In the new matrix, each image block corresponds to a numerical vector of length D. In the "embedded image block is connected with position coding" stage, the position indexes of serial numbers 0, 1, 2, ..., N are respectively encoded as position vectors of length D, among which the position vectors corresponding to serial numbers 1, 2, ..., N are respectively added to the numerical vectors of the corresponding image blocks to generate the "embedded position image block" vector, which combines image information and position information.

[0158] During the network model training stage, the category of the target image is encoded to generate a "category embedding" vector of length D. The "category embedding" vector is added to the position vector corresponding to serial number 0 to generate an "embedded category image block" vector. The "embedded position image block" vector and the "embedded category image block" vector both belong to the "embedded image block" vector.

[0159] The embedded image block containing position information (first image block) and the embedded image block representing category information (second image block) are used together as embedded image blocks, and the embedded image blocks are passed through the Transformer encoder to generate image features I.

[0160] Specifically, all “embedded image block” vectors are combined to form a numerical matrix, which is passed through the Transformer encoder to produce image features.

[0161] The Transformer encoder is composed of M Transformer modules stacked in sequence. When the value of M is 12, 24 or 32, the image encoding effect is best, and the output data of the previous Transformer module is used as the input data of the next Transformer module.

[0162] In each Transformer module, the input data is first normalized by the layer, and then passes through the self-attention module. The generated data is added to the input data to generate the intermediate feature D. The intermediate feature D is then normalized by the layer and multi-layer perceptron, and then added to the intermediate feature D to generate the output data of the current Transformer module. The image feature I generated by the last Transformer module serves as the output of the Transformer encoder and also as the output of the image encoding module.

[0163] Before the embedded image blocks are input into the Transformer encoder, the data of all embedded image blocks need to be combined to form a numerical matrix. Each row of the numerical matrix corresponds to a vector of an embedded image block. The numerical matrix is ​​input into the Transformer encoder as a whole for encoding processing.

[0164] The calculation process of the "layer normalization", "self-attention" and "multi-layer perceptron" modules in the Transformer encoder will associate the data in different rows of the above numerical matrix. Therefore, after being processed by the Transformer encoder, the data of different embedded image blocks are actually fused, that is, the image information of different blocks is associated, so that the image features finally extracted can reflect the global context information of the image.

[0165] In one embodiment, reasoning the image feature data to obtain a recognition result of the target image includes at least one of the following:

[0166] 1) inputting the image features into a classification model composed of a multi-layer perceptron and a first activation function to obtain source information of at least part of the image content of the target image;

[0167] In this embodiment, if Fig.10 As shown in the figure, the multilayer perceptron is a feedforward neural network consisting of multiple layers, each layer contains multiple neurons, which can learn complex patterns and relationships of input data. The first activation function is the Softmax function.

[0168] The Softmax function is used in the output layer of the multilayer perceptron. The Softmax function can convert the raw scores (also called logits) output by the neural network into a probability distribution, so that the output value of each category is between 0 and 1, and the sum of the output values ​​of all categories is 1. In this way, each output value can be interpreted as the probability that the model predicts that the input image belongs to a certain category.

[0169] For example, an image recognition task, the goal is to identify whether the scene in the image is indoors or outdoors. First, features are extracted from the image, and then a multi-layer perceptron model with two hidden layers is used to process these features. The output dimension of the last fully connected layer of the model is equal to the number of categories (in this case, 2, i.e. indoors and outdoors). At the output layer, the Softmax activation function is applied to obtain the probability of each category. For example, if the output of the Softmax layer is [0.4, 0.6], it means that the model predicts that the image is 40% likely to be indoors and 60% likely to be outdoors.

[0170] Alternatively, to identify whether an image is generated by AI (such as MidJourney) or real, first extract features from the image, including but not limited to: Color features: AI-generated images may show some unnatural color combinations, or the color distribution in different areas is relatively consistent; Texture features: The generated image may lack the complexity of certain details, especially on edges or small objects (such as details of people, hair, eyes, etc. may appear abnormal); Local inconsistency: AI-generated images may show unnatural symmetry, artifacts or unreal details in some local areas (such as backgrounds or distant views).

[0171] These features are then processed using a multi-layer perceptron model with two hidden layers. The output dimension of the last fully connected layer of the model is equal to the number of categories (in this case, 2, i.e., “real” vs. “AI generated”). At the output layer, a Softmax activation function is applied to obtain the probability of each category.

[0172] When the multi-layer perceptron (MLP) model is used to process these features, it includes the following: Input layer: The extracted features of the image are used as input, such as color distribution, texture information, etc. Hidden layer: It contains two hidden layers, and the number of neurons in each layer is designed according to the complexity of the task. Each layer performs nonlinear transformation of features to extract higher-level image features. Output layer: The output dimension of the last fully connected layer of the model is 2, corresponding to two categories: "real" and "AI generated". Category 0: indicates that the image is "real"; Category 1: indicates that the image is "AI generated".

[0173] At the output layer, the Softmax activation function is applied to obtain the probability of each category. The Softmax function transforms the output of the model into a probability distribution for each category. Suppose the output of the model is [0.3, 0.7], which means that the model predicts that the probability of the image being "real" is 30% and the probability of being "AI generated" is 70%.

[0174] The output dimension can also be set to 4, corresponding to four categories: "Real", "MidJourney Model", "Stable Diffusion Model" and "StyleGAN Model". This can not only determine the authenticity of the image, but also give the specific models that generated it.

[0175] 2) Inputting the image features into a multi-layer perceptron and an output layer composed of a second activation function to obtain source information of at least part of the image content of the target image and an explanation of the source information.

[0176] In this embodiment, the multilayer perceptron is a neural network composed of multiple fully connected layers (also called dense layers), each layer contains multiple neurons, and can learn the complex mapping relationship between image features and category labels. At the top of the multilayer perceptron, an output layer is formed using a second activation function, which is a Softmax function.

[0177] The model not only outputs the category label of the target image (i.e., source information), but also provides an explanation, i.e., the reason for reaching that conclusion. This is usually achieved through additional network structures or algorithms that can explain the model's prediction results.

[0178] For example, Fig.18 As shown in (b), an image is analyzed to determine whether a specific part of it is real, and a detailed explanation of the authenticity of the local area is given. The foreground of the target image is the face of a long-haired woman, and the background is gray. The user asked two questions about the authenticity of the image: 1) the authenticity of the entire face, and 2) the authenticity of the local area of ​​the face (i.e., the nose).

[0179] When processing an image, key features are first extracted, which may include texture features (such as skin texture and details), shape features (such as facial contours and structures), and color features (such as skin color, shadows, and lighting). These features help determine whether the objects in the image are natural and whether they conform to the physical laws of the real world.

[0180] The extracted features are input into a multi-layer perceptron model to determine whether the image is generated by a generative adversarial network (GAN) model such as StyleGAN or comes from the real world.

[0181] The multi-layer perceptron model contains two hidden layers, each of which uses the ReLU activation function for nonlinear transformation to improve the model's expressiveness. In the output layer, the model uses the Softmax activation function to give the probability that the image is real or AI-generated (StyleGAN-generated). Assuming that the model outputs [0.2, 0.8], this means that the model believes that the probability that the image is an AI-generated (StyleGAN) image is 80%, and the probability that it is a real image is 20%.

[0182] In addition to making an overall true or false judgment, the model also uses an attention mechanism to identify key areas in the image that influence the decision. For example, in this image of a face, the attention mechanism highlights areas that influence the judgment of the authenticity of the nose.

[0183] If the model determines that the face is generated by StyleGAN, it will further analyze and provide specific explanations about the nose area. For example, the model may detect that the appearance of the nose looks natural, but due to the lack of details (such as skin texture, pores, blemishes, etc.) and the overly smooth skin color, it infers that the face is generated rather than a real face.

[0184] The text answer generated by the model will provide a detailed explanation of the user's question. For example, for the question "Is the face in the photo real?", the model will answer: "The face in the photo is fake." and further explain: "The nose of the face looks real, but the face lacks texture details and the skin color is too smooth."

[0185] In addition, in response to the question “Is the nose real?”, the model gave specific details through local explanations: “The shape and appearance of the nose look natural, but the overall lack of details may be AI-generated.”

[0186] The entire image processing structure involves not only an overall judgment of the authenticity of the image, but also a detailed explanation of the local areas in the image to enhance the transparency and credibility of the model. By combining technologies such as the attention mechanism, the model can provide specific and targeted explanations to help users understand why the model makes a certain prediction, especially when dealing with complex image generation technologies (such as images generated by StyleGAN), it can provide more informed decision support.

[0187] In one embodiment, if Figure 4 As shown, extracting text feature data of the target question information includes:

[0188] S1031, segmenting the target question information into a text character sequence based on semantic information of the target question information, wherein a mark separator is inserted into the text character sequence;

[0189] In this embodiment, the user's natural language question is decomposed into a series of text characters, and specific token separators (such as spaces, special symbols, etc.) are inserted into these character sequences to help the model understand the structure of words and sentences.

[0190] For example, a user asks: "Is this picture real?" The system splits the question into a sequence of characters: "Is this picture real?" and inserts spaces between each word as token separators.

[0191] S1032, representing the text character sequence and the tag separator as word vector data;

[0192] In this embodiment, word vectors are vectors that map each word or character in a text to a high-dimensional space. These vectors can capture the semantic information of the word or character. For example, the system converts each word and space into a corresponding word vector. For example, each word such as "this", "a piece", "a picture", etc. corresponds to a word vector, and a space may also correspond to a special word vector to indicate the separation between words.

[0193] S1033. Perform word embedding encoding processing on the word vector data to output the text feature data.

[0194] In this embodiment, the word vector data is encoded, usually using word embedding technology (such as Word2Vec, GloVe or BERT, etc.). This process generates text feature data, which can be used to represent the semantic information of the original text and used for subsequent machine learning tasks.

[0195] When this application obtains text feature data, such as Figure 8 As shown in the figure, the text question Q is processed by the word embedding representation module, the text sentence is divided into a text character sequence, and special characters such as separators are inserted into the text character sequence. Each character is processed by the word embedding encoding module to generate a corresponding word embedding vector, and the word embedding vectors of all characters are combined to form the text feature data of the text question.

[0196] Specifically, Fig.11 As shown in the figure, the input data of the word embedding representation module is the text question Q, and the output data is the word embedding S of the text. The word embedding representation module includes two steps: word tagging and word vector representation.

[0197] In the word tagging stage, the text question Q is segmented into text character sequences according to semantic information, and corresponding tag delimiters are inserted at the beginning, end and semantic segmentation positions of the text question sentence.

[0198] In the word vector representation stage, all text characters and token separators in the text character sequence are represented as numerical word vectors. Each text character and token separator corresponds to a word vector, and all word vectors are stacked up to form the word embedding S of the text.

[0199] like Fig.12 As shown, the word embedding encoding module encodes the word embedding S of the text into a text feature T. The word embedding encoding module consists of N encoding layers connected in sequence, and the output data of each encoding layer becomes the input data of the next encoding layer.

[0200] In the i-th (1≤i≤N) encoding layer, the input data of the encoding layer passes through the self-attention module and is added to the input data of the encoding layer to generate feature data R i 1. Characteristic data R i 1 After layer normalization, feature data R is generated i 2,

[0201] Characteristic data R i 2 After being processed by the multi-layer perceptron and the feature data R i 2 Addition generates characteristic data R i 3. Characteristic data R i 3 After layer normalization, the output data of the i-th encoding layer is generated, and the output data of the last encoding layer is used as the output of the word embedding encoding module, that is, the text feature T.

[0202] For example, the system uses the pre-trained BERT model to encode the above word vectors. The BERT model considers the contextual information of each word and outputs the context-related word embedding of each word. These embedding vectors capture the complex relationship between words and can be used as text feature data to answer user questions, such as determining the color of an image.

[0203] This application can convert users' natural language questions into machine-understandable text feature data, which can be combined with image feature data to answer users' questions about images. This approach enables the model to process and understand complex natural language input and provide more accurate answers in multimodal learning tasks.

[0204] In one embodiment, reasoning the fused feature data to obtain a recognition result for the target image includes:

[0205] The fused feature data is decoded using an autoregressive decoding module to obtain response information for question information regarding at least a portion of the image content of the target image.

[0206] In this embodiment, the fused feature data generally refers to data that combines image features and text features (ie, question features). These feature data can represent the semantic information of image content and related questions.

[0207] The autoregressive decoding module is a commonly used technique in the fields of natural language processing and image processing, especially in multimodal learning scenarios such as Visual Question Answering (VQA).

[0208] The autoregressive decoding module uses an autoregressive model (such as a Transformer-based model) to gradually generate a response message. At each step, the model predicts the next word or phrase based on the currently generated response content and the fused feature data. Ultimately, the response information generated by the autoregressive decoding module is an answer to the question information about at least part of the content of the target image, and may include an explanation of the source information.

[0209] Specifically, the autoregressive decoding module performs reasoning based on the fused feature data, generates reasons corresponding to the image category judgment in the form of text sentences, and outputs a text answer.

[0210] For example, in an image question answering system, a user uploads an image and asks: "How many birds are there in this picture?"

[0211] The system first extracts the visual features of the image and converts the user's question into text features. These features are then fused into a unified data representation for the subsequent decoding process.

[0212] The autoregressive decoding module starts working, which uses the fused feature data to gradually build the answer. At each step, the model considers the generated response content (e.g., "two") and the fused feature data to predict the next word (e.g., "bird").

[0213] Finally, the decoding module generates a complete response: “There are two birds in this image.” This response not only answers the question but also provides an explanation, such as: “The response is based on the morphological and quantitative characteristics of the birds detected in the image.”

[0214] This application uses an autoregressive decoding module to generate accurate and interpretable response information using fused feature data, thereby enhancing the interactivity and transparency of the system.

[0215] Specifically, the fusion feature data is decoded using an autoregressive decoding module, including:

[0216] Using a first decoder to decode the fused feature data to obtain a first predicted text character, wherein the first decoder updates a word embedding sequence according to the first predicted text character;

[0217] In this embodiment, the first decoder is used to decode the fused feature data (data combining image features and question text features) to generate the first predicted text character, which is the beginning part of the response information.

[0218] The first decoder updates its internal word embedding sequence based on the generated first predicted text character. This means that the internal state of the decoder is adjusted based on the newly generated character in order to better predict the next character.

[0219] The updated first decoder decodes the fused feature data and the first predicted text character to obtain a second predicted text character;

[0220] In this embodiment, the updated first decoder decodes the fused feature data and the newly added first predicted text character again to generate a second predicted text character.

[0221] This process continues as the decoder generates new predicted text characters and updates its internal state until the end symbol is generated.

[0222] In the case where it is determined that the second predicted text character is a terminator, response information is determined, wherein the response information includes the first predicted text character and the second predicted text character.

[0223] In this embodiment, once the second predicted text character is determined to be the terminator, the decoding process stops, and the response information at this time includes all the first predicted text characters and the second predicted text characters generated previously.

[0224] Specifically, Fig.14 As shown, the input data of the autoregressive decoding module is the fusion feature F, and the output data is the text answer A. This module uses an autoregressive method to predict and output text characters one by one. The process of predicting the next text character depends not only on the input fusion feature, but also on the text characters that have been predicted and output.

[0225] When new text characters are output, the autoregressive decoding module updates the word embedding and generates a new text word embedding sequence W. The word embedding sequence W is processed through causal self-attention, residual connection layer and layer normalization to generate text features E corresponding to the text answer that has been output. The input fusion feature F and text feature E are cross-attended to generate feature C1.

[0226] In the cross-attention process, the text feature E is flattened as the query vector, and the fused feature F is flattened as the key vector and value vector.

[0227] Feature C1 is added to text feature E to generate feature C2. Feature C2 is normalized by layers to generate feature C3. Feature C3 passes through a multi-layer perceptron and is added to feature C3 to generate feature C4. Feature C4 passes through layer normalization and a linear layer to generate a feature vector V. Feature vector V is predicted by a Softmax activation layer to generate a word embedding. The word embedding generates the current text character through tag mapping. The text character is placed at the end of the current text answer.

[0228] The generated word embedding generates a new word embedding sequence by updating the word embedding, and loops in the above autoregressive manner until the text character generated by the token mapping is the terminator, generating a text answer A as the output of the autoregressive decoding module.

[0229] For example, a user uploaded a picture and asked, "What is the person in this picture doing?"

[0230] The system first uses the first decoder to decode the fused feature data and generate the first predicted text character: "read". The first decoder updates its internal word embedding sequence based on "read" and prepares to generate the next character.

[0231] The updated first decoder decodes the fused feature data and "read" again to generate the second predicted text character: "book". This process continues, and the decoder generates subsequent characters, such as ".", until the end character is encountered.

[0232] When the decoder generates the end character ".", the decoding process stops. The response information at this time is "Reading.", which is the answer to the user's question.

[0233] In this way, the system can gradually build a complete response information while ensuring that each step of the prediction is based on the latest context information. This iterative decoding process enables the model to generate coherent and relevant answers, improving the accuracy and naturalness of the answers.

[0234] In one embodiment, outputting a recognition result for the target image includes at least one of the following:

[0235] 1) outputting source information of at least part of the image content of the target image and an explanation of the source information;

[0236] In this embodiment, the source information of the image content and the related reasons / basis are directly output, and the explanation of the source information is the reason for reaching the conclusion. The source information refers to the specific content that can be identified in the image, such as the category of the object, the description of the scene, the type of action, etc.

[0237] For the recognized image content, the system also provides an explanation of why the model reached such a recognition result, which is usually based on visual clues and contextual information in the image.

[0238] For example, the system identifies the Eiffel Tower and uses it as key information to determine that the photo was taken in Paris. The system provides an explanation: "The Eiffel Tower in the photo is a landmark building in Paris, so it can be inferred that the photo was taken in Paris."

[0239] 2) outputting source information of at least a portion of the image content of the target image and a first control, wherein the first control can be triggered to display an explanation of the source information;

[0240] In this embodiment, the explanation is not displayed directly, but through a control or a thumbnail display control (such as view more), and the entire explanation is displayed after the control is triggered.

[0241] The first control is a user interface control provided by the system, such as a "see more" button or link, and the user can trigger the display of more explanations by clicking on this control. When the user triggers the first control, the system will display detailed explanations for the source information, which explain why the system has reached such a recognition result.

[0242] For example, a user uploaded a photo of the Eiffel Tower and asked, "Where was this picture taken?"

[0243] The system identified the Eiffel Tower in the image and determined that the picture was taken in Paris. The system displayed a button on the user interface that read "See location explanation." After the user clicked the "See location explanation" button, the system displayed a detailed explanation: "The Eiffel Tower in the picture is a landmark building in Paris. Based on its unique structural features and other landmark buildings in the background, we determined that this picture was taken in Paris."

[0244] 3) outputting source information of at least a portion of the image content of the target image, and, when the source information is triggered, outputting an explanation of the source information;

[0245] In this embodiment, the source information is not directly displayed on the interface, but is output in the form of a control (such as a button or a link). The user can trigger more operations through this control.

[0246] After the user further clicks on the source information, the action of outputting the explanation is triggered. That is, when the user interacts with the control (such as clicking a button), the system will display a detailed explanation of the source information. These explanations provide the reasons for obtaining the source information.

[0247] For example, a user uploaded a landscape photo that included the Eiffel Tower and asked, “Where was this picture taken?”

[0248] After analyzing the image, the system determines that the photo was taken in Paris and displays a button on the interface that says "Show location information". The source information (Paris) is not directly displayed on the interface, but is triggered by clicking the "Show location information" button.

[0249] When the user clicks the "Show location information" button, the system displays a pop-up window or sidebar with a detailed explanation: "The Eiffel Tower in the picture is a landmark building in Paris. Based on its unique structural features and other landmark buildings in the background, we determined that this picture was taken in Paris."

[0250] 4) outputting source information for at least a portion of the image content of the target image and first response information for first question information for at least a portion of the image content;

[0251] In this embodiment, the source information of at least part of the image content includes the category of the object in the image, the description of the scene, the background of the event, etc.

[0252] In addition to displaying the source information, other answer information for the question can also be displayed. The first question information is a question raised by the user for at least part of the content of the image. The first answer information is a targeted answer provided by the system based on the user's question and the image content.

[0253] The user's question is not about the authenticity of the image, but about the time when the image was generated or acquired. The corresponding response information can be timestamp information, or other information, such as photographer information, generation location information, etc.

[0254] 5) Outputting source information of at least a portion of the image content of the target image and a second control, wherein the second control can be triggered to display associated information of the source information.

[0255] In this embodiment, the system provides a user interface control (second control), such as a "more information" button or link. This control is designed as a recommendation control for source information, and is used to trigger the display of more content related to the source information.

[0256] When the user interacts with the second control (such as clicking a button), the system will display recommended information related to the source information. This information may include purchase links, configuration interfaces, sharing interfaces, etc.

[0257] For example, a user uploaded a photo of the Eiffel Tower in Paris and asked, "What is the background of this image?"

[0258] After analyzing the image, the system determines that the background of the picture is Paris and displays a button on the interface that says "Explore Paris". Users can click this button to get more information about Paris.

[0259] After the user clicks the "Explore Paris" button, the system displays a pop-up window or sidebar containing the following recommended information: Purchase link: provides a purchase link for the Eiffel Tower model; travel destination configuration interface: allows users to set Paris as a travel destination and displays travel package information; sharing interface: provides the option to share this picture to social media.

[0260] In one embodiment, if Figure 5 As shown, the recognition result for the target image is output, including:

[0261] S201, outputting source information of at least a portion of the image content of the target image;

[0262] In this embodiment, the system analyzes the target image, identifies and outputs the source information of at least part of the content in the image. The source information includes the authenticity result of the image, the source model, the shooting device, the shooting location (such as a scenic spot), the camera position, the shooter, etc.

[0263] S202: Output target content data associated with the source information based on the category information of the source information.

[0264] In this embodiment, the output of explanations or response information associated with the source information, or recommendation information for scenic spots, or purchase or sharing links for equipment, etc. is determined based on the category information of the source information.

[0265] Based on the identified source information, the system decides what type of information or link to output, whether to output an explanation based on the identification result, and what kind of explanation or associated response information to output based on the identification result.

[0266] In this embodiment, the system determines whether to output an explanation based on the certainty of the recognition result or the needs of the user. If the recognition result is very clear and confident, or the user explicitly requests an explanation, the system may provide an explanation. If an explanation is required, the system will decide whether to provide an explanation or associated response information. This depends on the nature of the recognition result and the specific needs of the user.

[0267] Specifically, if the recognition result is an image authenticity result or a source model, the user needs to understand the reasons behind the recognition result, and an explanation is output at this time. If the recognition result is the shooting time, the answer information of the user's question based on the image content is provided. If the recognition result is a certain scenic spot or product, relevant recommendation information is provided. If the recognition result involves purchasable goods or equipment, a purchase link is provided. If the user may want to share the image, a sharing link is provided.

[0268] For example, a user uploaded a photo taken in front of the Eiffel Tower in Paris and asked, "What is the background of this image?"

[0269] After analyzing the image, the system determines that the background of the picture is the Eiffel Tower in Paris, and outputs the source information of this part of the image content: "The background of this picture is the Eiffel Tower in Paris."

[0270] The system recognizes that the source information belongs to the "travel destination" category, so it provides a "More Information" button as the second control. After the user clicks this button, the system displays additional data related to the Eiffel Tower in Paris: Purchase link: provides a purchase link for Eiffel Tower souvenirs; Travel destination configuration interface: allows users to set Paris as a travel destination and displays travel package information; Share interface: provides the option to share this picture to social media.

[0271] In this way, the present application not only provides direct information about the image content, but also provides related additional services and data based on the categories of this information, thereby enhancing the user experience and the practicality of the information.

[0272] In summary, if Figure 7 and Figure 8 As shown, the working process of the image processing method is as follows:

[0273] The input data includes the target image X and the text question Q, and the output data includes the image category L and the text answer A.

[0274] The target image X generates image feature data through image encoding. The image feature data contains both the overall information of the target image and the key local information of the target image to support the authenticity judgment of the overall and local areas of the image. The image feature data is passed through the classification module to generate the image category L as the first output data.

[0275] The text question Q is processed by the word embedding representation module, which divides the text sentence into a text character sequence and inserts special characters such as separators into the text character sequence. Each character is processed by the word embedding encoding module to generate a corresponding word embedding vector. The word embedding vectors of all characters are combined to form the text feature data of the text question.

[0276] The image feature data and text feature data are fused by the image-text feature fusion module to generate fusion features. The image-text feature fusion module associates the semantic information in the text question with the overall and key local information of the target image to support text questions about the overall and local target image.

[0277] Finally, the autoregressive decoding module performs reasoning based on the fused feature data, generates reasons corresponding to the image category judgment in the form of text sentences, and outputs the text answer A as the second output data.

[0278] Specifically, Fig. 9 As shown in the figure, when image encoding is performed, the target image X generates image features I through the image encoding module, and the target image X is evenly divided into nine image blocks, each image block corresponds to a position index number, and all the pixels in each image block are tiled and linearly projected to generate a corresponding image pixel sequence, and the position index number of each image block is positionally encoded to generate position coding information, and the image pixel sequence and the position coding information are cascaded to form an embedded image block containing position information.

[0279] In order to introduce image category information into the image encoding process, a learnable category embedding is used to represent the image classification task. The category embedding is concatenated with the index number 0 to generate an embedded image block representing the category information.

[0280] The embedded image block containing position information and the embedded image block representing category information are used as the embedded image block, and the embedded image block is passed through the Transformer encoder to generate image features I.

[0281] The Transformer encoder is composed of M Transformer modules stacked in sequence. When the value of M is 12, 24 or 32, the image encoding effect is best, and the output data of the previous Transformer module is used as the input data of the next Transformer module.

[0282] In each Transformer module, the input data is first normalized by the layer, and then passes through the self-attention module. The generated data is added to the input data to generate the intermediate feature D. The intermediate feature D is then normalized by the layer and multi-layer perceptron, and then added to the intermediate feature D to generate the output data of the current Transformer module. The image feature I generated by the last Transformer module serves as the output of the Transformer encoder and also as the output of the image encoding module.

[0283] like Fig.10As shown in the figure, when the classification module is used, the input of the classification module is the image feature I and the output is the image category L. The classification module includes a multi-layer perceptron and a Softmax activation layer in sequence. The Softmax activation layer outputs a vector composed of probability values. The sum of all probability values ​​is 1, and the category corresponding to the maximum probability value is used as the image category.

[0284] like Fig.11 As shown in the figure, when word embedding is used, the input data of the word embedding representation module is the text question Q, and the output data is the word embedding S of the text. The word embedding representation module includes two steps: word tagging and word vector representation.

[0285] In the word tagging stage, the text question Q is segmented into text character sequences according to semantic information, and corresponding tag delimiters are inserted at the beginning, end and semantic segmentation positions of the text question sentence.

[0286] In the word vector representation stage, all text characters and token separators in the text character sequence are represented as numerical word vectors. Each text character and token separator corresponds to a word vector, and all word vectors are stacked up to form the word embedding S of the text.

[0287] like Fig.12 As shown in the figure, when word embedding encoding is performed, the word embedding encoding module encodes the word embedding S of the text into a text feature T. The word embedding encoding module is composed of N encoding layers connected in sequence, and the output data of each encoding layer becomes the input data of the next encoding layer.

[0288] In the i-th (1≤i≤N) encoding layer, the input data of the encoding layer passes through the self-attention module and is added to the input data of the encoding layer to generate feature data R i 1. Characteristic data R i 1 After layer normalization, feature data R is generated i 2,

[0289] Characteristic data R i 2 After being processed by the multi-layer perceptron and the feature data R i 2 Addition generates characteristic data R i 3. Characteristic data R i 3 After layer normalization, the output data of the i-th encoding layer is generated, and the output data of the last encoding layer is used as the output of the word embedding encoding module, that is, the text feature T.

[0290] like Fig.13 As shown in the figure, when the image-text features are fused, the input text feature T is sequentially normalized and self-attentioned to generate feature T1, feature T1 is added to the text feature T to generate feature T2, feature T2 is normalized, and then fused with the image feature I through cross-attention to generate feature G1.

[0291] Feature G1 is added to feature T2 to generate feature G2, and feature G2 is sequentially normalized and multi-layer perceptron to generate feature G3. Finally, feature G3 is added to feature G2 to generate fused feature F, which is used as the output of the image-text feature fusion module.

[0292] like Fig.14 As shown in the figure, during autoregressive decoding, the input data of the autoregressive decoding module is the fusion feature F, and the output data is the text answer A. This module uses autoregressive method to predict and output text characters one by one. The process of predicting the next text character depends not only on the input fusion feature, but also on the text characters that have been predicted and output.

[0293] When new text characters are output, the autoregressive decoding module updates the word embedding and generates a new text word embedding sequence W. The word embedding sequence W is processed through causal self-attention, residual connection layer and layer normalization to generate text features E corresponding to the text answer that has been output. The input fusion feature F and text feature E are cross-attended to generate feature C1.

[0294] In the cross-attention process, the text feature E is flattened as the query vector, and the fused feature F is flattened as the key vector and value vector.

[0295] Feature C1 is added to text feature E to generate feature C2. Feature C2 is normalized by layers to generate feature C3. Feature C3 passes through a multi-layer perceptron and is added to feature C3 to generate feature C4. Feature C4 passes through layer normalization and a linear layer to generate a feature vector V. Feature vector V is predicted by a Softmax activation layer to generate a word embedding. The word embedding generates the current text character through tag mapping. The text character is placed at the end of the current text answer.

[0296] The generated word embedding generates a new word embedding sequence by updating the word embedding, and loops in the above autoregressive manner until the text character generated by the token mapping is the terminator, generating a text answer A as the output of the autoregressive decoding module.

[0297] For example, in Fig.18 In (a), the foreground of the target image is a man's face, and the background is a wall with a colorful pattern. The user asks "Is this picture real?" After receiving the two input data of the text question and the target image, the target image is analyzed and processed.

[0298] Then the image source is determined to be the "Stable Diffusion model", indicating that the target image is generated by the Stable Diffusion model, and a text answer is given: "This picture is fake. The eyebrows of the face in the picture look unnatural and overlap. The face part is generated by the Stable Diffusion model, but the background part is real." This application not only determines that the image is fake as a whole, but also gives an analytical judgment on the authenticity of the local area of ​​the image.

[0299] exist Fig.18 In (b), the foreground of the target image is the face of a long-haired woman, and the background is gray. The user asks, "Is the face in the photo real? Is the nose real?" The user not only asks about the authenticity of the entire face, but also about the authenticity of the local area of ​​the face, namely the nose.

[0300] After analyzing and processing the target image, it determines that the image source is the "StyleGAN model", indicating that the image is generated by the StyleGAN model, and gives a text answer: "The face in the photo is fake. The nose looks real, but the face lacks texture details and the skin color is too smooth." It not only gives the overall judgment that "the face in the photo is fake", but also gives the local judgment that "the nose of the face looks real", directly answering the user's question about the authenticity of the nose.

[0301] exist Fig.18 In (c), the foreground of the target image is the face of a man in formal attire, and the background is blue and partially white. The user asks, "Is the face in the photo real?" After analyzing and processing the target image, it is determined that the image source is a "real image", indicating that the image was taken in the real physical world, and a text answer is given, "The face in the photo is real. This is a male face, and the face looks natural and coordinated overall." This describes the authenticity of the male face in the picture.

[0302] This application supports users to ask questions about the authenticity of images through text sentences. The scope of text questions not only supports the entire image, but also supports local areas, which can meet the personalized needs of users. This solution provides a technical basis for further developing more intelligent multi-modal (for example, users ask questions through voice) deep fake image recognition and attribution methods.

[0303] This application can not only determine whether the target image is a deep fake image, but also, when it is determined that the target image is a deep fake image, this solution can further identify which type of model the target image is produced by and locate the source of the deep fake image. In practical applications, this helps to manage technical risks and ensure user safety.

[0304] This application elaborates on the authenticity of the image in the form of text answers (or other forms), not only making an overall judgment on the authenticity of the image, but also making analytical judgments on the authenticity of local areas of the image based on questions raised by users, making it easier for users to understand the logic behind the authenticity judgment and improving the user experience.

[0305] Throughout the image processing process, three attention mechanisms are used: self-attention, causal self-attention, and cross-attention.

[0306] like Fig.15 As shown in the figure, the self-attention mechanism is used for both image encoding and text encoding. There is only one input feature for self-attention, which can be either an image feature or a text feature. The input feature undergoes three different linear projection transformations to generate a query matrix, a key matrix, and a value matrix respectively. The query matrix is ​​matrix-multiplied with the key matrix, scaled and activated by the Softmax layer, and then matrix-multiplied with the value matrix to generate the output feature.

[0307] like Fig.16 As shown in the figure, the causal self-attention mechanism is used in the autoregressive decoding process. The input data is text features and there is only one input. The input text features are transformed through three different linear projections to generate query matrix, key matrix and value matrix respectively. Since the process of predicting the current text characters only depends on the predicted text characters and the current text features, but not on the text characters or text features predicted in the future, the query matrix is ​​multiplied by the key matrix by masking matrix multiplication. That is, the values ​​corresponding to the future text features in the query matrix and the key matrix are masked and not allowed to participate in the matrix multiplication process. Only the values ​​corresponding to the predicted text features are involved in the matrix operation process. The result of the masking matrix multiplication is scaled and Softmax layer, and then multiplied with the value matrix by the masking matrix to generate the output feature.

[0308] like Fig.17 As shown in the figure, the cross attention mechanism is used in the image-text feature fusion module and the autoregressive decoding module. The fusion feature input to the autoregressive decoding module can be regarded as a special image feature. There are two input data of the cross attention mechanism: text features and image features. The text features generate a query matrix through linear projection, and the image features generate a key matrix and a value matrix through two different linear projections. The query matrix and the key matrix are matrix-multiplied. The result of the multiplication is scaled and the Softmax activation layer, and then matrix-multiplied with the value matrix to generate fusion features.

[0309] In one embodiment, before the actual application of this solution, it is necessary to construct training samples and use the training samples to train the network model.

[0310] When constructing training samples, mainstream image generation models are used to generate image samples, which are then added to the training set. The category of each image is the image generation model used to generate the image. At the same time, the training set also contains an appropriate number of real images, which are labeled as "real images". Several text questions about the authenticity of each image are provided for each image. Volunteers write text answers based on the text questions, focusing on the authenticity of the image as a whole and in parts. Each image, image category, text question and text answer are combined to form a sample of the training set.

[0311] In order to improve the generalization ability of this scheme and be able to identify various types of deep fake images in practical applications, this scheme collects a large number of multi-category and diversified training samples, and uses various types of image generation models to generate fake images, including generative adversarial network (GAN) series methods and diffusion model series methods. The generative adversarial network series methods include AttGAN, CycleGAN, GDWCT, IMLE, ProGAN, StarGAN, StarGAN-v2, StyleGAN, StyleGAN2, etc. The diffusion model series methods include DALL-E-2, GLIDE, Latent Diffusion, Stable Diffusion, etc.

[0312] During the training phase, the network model is trained using training samples. In order to improve the robustness of the model in practical applications, data enhancement methods are used in the process of constructing the training set to simulate common operations in actual image applications, generate new sample images, and add the new sample images to the training set to increase the diversity of training samples. The data enhancement methods include adding Gaussian noise to the training sample images, and performing Gaussian blur, JPEG compression, rotation, and mirroring on the training sample images.

[0313] During the model training phase, the word embeddings generated by the autoregressive decoding module and the word embeddings of the training samples are compared through the loss function. The loss is then used to update the model parameters through the back-propagation mechanism and the optimizer. This optimization process is only used for model training and is not required for actual reasoning.

[0314] The present application also provides an image processing device, such as Figure 6 As shown, including:

[0315] a recognition module configured to perform recognition processing on a target image in response to a target trigger operation, wherein the target image is an image input to a target application or an image currently displayed by the target application, and the target application is an application capable of performing the recognition processing or calling a target program file to perform the recognition processing;

[0316] An output module is configured to output a recognition result for the target image, wherein the recognition result can at least indicate source information of at least a portion of image content of the target image.

[0317] It should be noted that the principle of solving the technical problem by the image processing device provided in the embodiment of the present application is similar to the image processing method provided in the embodiment of the present application. Therefore, the implementation of the image processing device provided in the embodiment of the present application can refer to the implementation of the image processing method provided in the embodiment of the present application, and the repeated parts will not be repeated.

[0318] An embodiment of the present application provides an electronic device, including: a memory and a processor, wherein an executable program is stored in the memory, and the processor executes the executable program to implement the steps of any method provided in the embodiment of the present application.

[0319] The processor may be a general purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The PLD may be a complex programmable logic device (PLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. The general purpose processor may be a microprocessor or any conventional processor.

[0320] Since the electronic device introduced in the embodiments of the present application is an electronic device provided with a memory for implementing the method disclosed in the embodiments of the present application, based on the method introduced in the embodiments of the present application, technical personnel in this field can understand the structure and deformation of the electronic device introduced in the embodiments of the present application, so it is not repeated here.

[0321] The embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the image processing methods provided in the embodiment of the present application are implemented.

[0322] The storage medium in this embodiment may be included in the electronic device, or may exist independently without being installed in the electronic device. The storage medium carries one or more computer programs, and when the one or more computer programs are executed, the steps of any method of the task processing method provided in the embodiment of the present application are implemented.

[0323] It should be understood that each scheme in this embodiment has the corresponding technical effects in the above method embodiment, which will not be repeated here.

[0324] According to an embodiment of the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. Optionally, the specific examples in this embodiment can refer to the examples described in any embodiment of the present application, and this embodiment is not repeated here. Obviously, it should be understood by those skilled in the art that the above-mentioned modules or steps of the present application can be implemented with a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented with a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from here, or they can be made into individual integrated circuit modules respectively, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the present application is not limited to any specific combination of hardware and software.

[0325] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present application.

Claims

1. An image processing method, comprising: In response to a target trigger operation, performing recognition processing on a target image, the target image being an image input to a target application or an image currently displayed by the target application, the target application being an application capable of executing the recognition processing or calling a target program file to execute the recognition processing; A recognition result for the target image is output, where the recognition result can at least indicate source information of at least a portion of image content of the target image.

2. The method according to claim 1, wherein: In response to the target trigger operation, the target image is identified and processed, including one of the following: In response to obtaining an operation of a target user inputting a target image into a target application, performing recognition processing on the target image; In response to obtaining target question information for a target image displayed by a target application, performing recognition processing corresponding to the target question information on the target image; In response to obtaining a selection operation of a question option for a target image displayed by a target application, performing recognition processing corresponding to the selection operation on the target image; In response to obtaining an operation of a target user inputting a target image into a target application and target question information for the target image, performing recognition processing corresponding to the target question information on the target image; In response to detecting a change in usage information of the electronic device, a target image displayed by a target application is identified.

3. The method according to claim 1 or 2, wherein: Perform recognition processing on the target image, including: Acquire image feature data of the target image, wherein the image feature data includes overall information and local information of the target image; In the case where the target question information is not obtained, the image feature data is inferred to obtain the recognition result of the target image; or, When target question information is obtained, text feature data of the target question information is obtained, the image feature data and the text feature data are fused to obtain fused feature data, and the fused feature data is inferred to obtain a recognition result for the target image.

4. The method according to claim 3, wherein: Acquiring image feature data of the target image, including: Dividing the target image into a plurality of image blocks, and processing the pixel points of the image blocks into image pixel sequences; Concatenating the image pixel sequence with the position coding information corresponding to each image block to obtain a first image block; Acquire a second image block representing source information; The first image block and the second image block are encoded using a first encoder to obtain the image feature data.

5. The method according to claim 3, wherein: Reasoning the image feature data to obtain a recognition result of the target image includes at least one of the following: Inputting the image features into a classification model composed of a multi-layer perceptron and a first activation function to obtain source information of at least part of the image content of the target image; The image features are input into a multi-layer perceptron and an output layer formed by a second activation function to obtain source information of at least a portion of the image content of the target image and an explanation of the source information.

6. The method according to claim 3, wherein: Extracting text feature data of the target question information includes: Segmenting the target question information into a text character sequence based on semantic information of the target question information, wherein a mark separator is inserted into the text character sequence; Representing the text character sequence and the token separator as word vector data; The word vector data is subjected to word embedding encoding processing to output the text feature data.

7. The method according to claim 3, wherein: Reasoning the fused feature data to obtain a recognition result for the target image includes: The fused feature data is decoded using an autoregressive decoding module to obtain response information for question information regarding at least a portion of the image content of the target image.

8. The method according to claim 1, wherein: Outputting a recognition result for the target image includes at least one of the following: Outputting source information of at least a portion of the image content of the target image and an explanation of the source information; Outputting source information for at least a portion of the image content of the target image and a first control, wherein the first control can be triggered to display an explanation for the source information; Outputting source information of at least a portion of the image content of the target image, and, when the source information is triggered, outputting an explanation of the source information; Outputting source information for at least a portion of image content of the target image and first response information for first question information for at least a portion of the image content; The source information of at least a part of the image content of the target image and a second control are output, and the second control can be triggered to display the associated information of the source information.

9. The method according to claim 1 or 8, wherein: Outputting the recognition result for the target image, including: Outputting source information of at least a portion of image content of the target image; Target content data associated with the source information is output based on the category information of the source information.

10. An image processing device, comprising: a recognition module configured to perform recognition processing on a target image in response to a target trigger operation, wherein the target image is an image input to a target application or an image currently displayed by the target application, and the target application is an application capable of performing the recognition processing or calling a target program file to perform the recognition processing; An output module is configured to output a recognition result for the target image, wherein the recognition result can at least indicate source information of at least a portion of image content of the target image.

Images