Text generation method, device, equipment and readable storage medium

Abstract

The present application discloses a text generation method, apparatus, device and readable storage medium, which belongs to the field of artificial intelligence technology. The method includes: obtaining a target image; classifying the target image to obtain a classification result of the target image, the classification result characterizing whether the target image is an adversarial image, and the adversarial image is an image disturbed by noise; when it is determined based on the classification result that the target image is not an adversarial image, obtaining a first target text related to the target image; generating a second target text based on the target image and the first target text through a visual language model, the second target text is related to the target image and has a contextual relationship with the first target text. When it is determined based on the classification result that the target image is not an adversarial image, it means that the target image has not been disturbed by noise, so that the visual language model can generate a second target text with a higher accuracy based on the target image and the first target text, thereby improving the security and reliability of the visual language model.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a text generation method, apparatus, device, and readable storage medium. Background Art

[0002] Visual language models (VLMs) are widely used in the field of artificial intelligence. They combine image processing and natural language processing. They can generate contextual information related to an image and previous text, where the context is related to the image.

[0003] Generally, if an image is not perturbed by noise, the visual language model outputs the correct context based on the image and the preceding text. In some cases, an adversarial image is created by carefully designing the image so that it is perturbed by noise. When the adversarial image and the preceding text are fed into the visual language model, the model outputs incorrect context, compromising its security and reliability. Summary of the Invention

[0004] The present application provides a text generation method, apparatus, device, and readable storage medium, which can improve the security and reliability of visual language models. The technical solution includes the following contents.

[0005] In one aspect, a text generation method is provided, the method comprising:

[0006] Acquire the target image;

[0007] Classifying the target image to obtain a classification result of the target image, wherein the classification result indicates whether the target image is an adversarial image, wherein the adversarial image is an image perturbed by noise;

[0008] When it is determined based on the classification result that the target image is the adversarial image, obtaining a first target text related to the target image;

[0009] A second target text is generated based on the target image and the first target text by using a visual language model, where the second target text is related to the target image and has a contextual relationship with the first target text.

[0010] In another aspect, a text generation device is provided, comprising:

[0011] An acquisition module, used to acquire a target image;

[0012] a classification module, configured to classify the target image and obtain a classification result of the target image, wherein the classification result indicates whether the target image is an adversarial image, wherein the adversarial image is an image perturbed by noise;

[0013] The acquisition module is further configured to acquire a first target text related to the target image when it is determined based on the classification result that the target image is not the adversarial image;

[0014] A generating module is configured to generate a second target text based on the target image and the first target text by using a visual language model, wherein the second target text is related to the target image and has a contextual relationship with the first target text.

[0015] In a possible implementation, the target image is related to a third target text;

[0016] The classification module is used to determine target attention information based on the target image, where the target attention information represents the influence of at least one image block in the target image on at least one character in the third target text; and classify the target image according to the target attention information to obtain the classification result.

[0017] In one possible implementation, the classification module is used to determine multiple visual tokens based on the target image, any one of the visual tokens represents the visual information of an image block in the target image; determine multiple first attention scores based on the multiple visual tokens, any one of the first attention scores represents the influence information of the image block corresponding to a visual token on the first character of the at least one character; and determine the target attention information based on the multiple first attention scores.

[0018] In one possible implementation, the classification module is used to determine multiple second attention scores based on the multiple visual tokens and each character before the non-first character in the at least one character, and any second attention score represents the impact information of the image block corresponding to a visual token on the non-first character; and determine the target attention information based on the multiple first attention scores and the multiple second attention scores.

[0019] In a possible implementation, the classification module is used to obtain a fourth target text, which has a contextual relationship with the third target text; and determine target attention information based on the fourth target text and the target image.

[0020] In a possible implementation, the fourth target text is a question text with an answer of yes or no, and the third target text is the answer.

[0021] In a possible implementation, the target attention information is multiple;

[0022] The classification module is used to classify the target image according to any target attention information to obtain a sub-result, and the sub-result represents whether the target image is the adversarial image; if the number of target sub-results is not less than a quantity threshold, it is determined that the classification result represents that the target image is not the adversarial image, and the target sub-result is a sub-result representing that the target image is not the adversarial image; if the number of target sub-results is less than the quantity threshold, it is determined that the classification result represents that the target image is the adversarial image.

[0023] In a possible implementation, the classification result is obtained by classifying the target image according to the target attention information by a target classifier;

[0024] The acquisition module is further configured to acquire a first sample image, where the first sample image is not the adversarial image;

[0025] The device further comprises:

[0026] a determining module, configured to determine first attention information based on the first sample image, where the first attention information represents information about an influence of at least one image block in the first sample image on at least one character in a first sample text, where the first sample text is related to the first sample image;

[0027] The classification module is further configured to classify the first sample image according to the first attention information using an initial classifier to obtain a first prediction result, where the first prediction result indicates whether the first sample image is the adversarial image;

[0028] A training module, configured to train the initial classifier based on the first prediction result to obtain a first classifier;

[0029] The determination module is further configured to determine the target classifier based on the first classifier.

[0030] In a possible implementation, the classification result is obtained by classifying the target image according to the target attention information by a target classifier;

[0031] The acquisition module is further configured to acquire a second sample image, where the second sample image is the adversarial image;

[0032] The device further comprises:

[0033] a determining module, configured to determine second attention information based on the second sample image, where the second attention information represents information about an influence of at least one image block in the second sample image on at least one character in a second sample text, where the second sample text is related to the second sample image;

[0034] The classification module is further configured to classify the second sample image according to the second attention information using the initial classifier to obtain a second prediction result, where the second prediction result indicates whether the second sample image is the adversarial image;

[0035] A training module, configured to train the initial classifier based on the second prediction result to obtain a second classifier;

[0036] The determination module is further configured to determine the target classifier based on the second classifier.

[0037] In one possible implementation, the acquisition module is used to generate a fourth sample text based on a first sample image and a third sample text related to the first sample image through the visual language model, where the first sample image is not the adversarial image, and the fourth sample text is related to the first sample image and has a contextual relationship with the third sample text; based on the fourth sample text, noise perturbation is performed on the first sample image to obtain the second sample image.

[0038] In a possible implementation, the acquisition module is configured to acquire noise text that is semantically different from the fourth sample text; and perform noise perturbation on the first sample image based on the noise text and the fourth sample text to obtain the second sample image.

[0039] In one possible implementation, the generation module is used to perform feature extraction on the target image through a visual language model to obtain target image features, where the target image features represent the visual information of the target image; obtain a first text feature through the visual language model, where the first text feature represents the language information of the first target text; and generate a second target text based on the target image features and the first text features through the visual language model.

[0040] In one possible implementation, the generation module is used to generate the first character in the second target text based on the target image feature and the first text feature through the visual language model; generate the non-first character based on the target image feature, the first text feature and each character before the non-first character in the second target text through the visual language model; and determine the second target text based on the first character and the non-first character.

[0041] In a possible implementation, the generation module is further configured to generate prompt information when the target image is determined to be the adversarial image based on the classification result, wherein the prompt information is configured to indicate that the second target text cannot be generated through the visual language model.

[0042] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements any of the above-mentioned text generation methods.

[0043] On the other hand, a computer-readable storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned text generation methods.

[0044] On the other hand, a computer program is also provided. The computer program is at least one, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned text generation methods.

[0045] On the other hand, a computer program product is also provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned text generation methods.

[0046] The technical solution provided by this application brings at least the following beneficial effects:

[0047] The technical solution provided by this application is to classify a target image and obtain a classification result indicating whether the target image is an adversarial image. Since adversarial images are noise-perturbed images, if the target image is determined not to be an adversarial image based on the classification result, it indicates that the target image has not been perturbed by noise. This allows the visual language model to generate a highly accurate second target text based on the target image and the first target text, thereby improving the security and reliability of the visual language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] Figure 1 Schematic diagram of a computer system for a text generation method provided in an embodiment of the present application;

[0050] Figure 2 This is a flowchart of a text generation method provided by an embodiment of the present application;

[0051] Figure 3is a schematic diagram of attention information determined based on different question texts provided by an embodiment of the present application;

[0052] Figure 4 is a schematic diagram of attention information determined based on different images provided by an embodiment of the present application;

[0053] Figure 5 This is a schematic diagram of the effect of whether to deploy a detection model provided by an embodiment of the present application;

[0054] Figure 6 is a flowchart of another text generation method provided in an embodiment of the present application;

[0055] Figure 7 This is a schematic diagram of image classification provided by an embodiment of the present application;

[0056] Figure 8 This is a structural diagram of a text generation device provided in an embodiment of the present application;

[0057] Figure 9 This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0058] Figure 10 This is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0060] It should be noted that the terms "first," "second," and the like in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0061] The various optional embodiments of this application can be implemented based on artificial intelligence technology. Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI is the study of the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0062] Artificial intelligence technology is an interdisciplinary subject that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. Among them, the pre-trained model is also called a large model or a basic model. After fine-tuning, it can be widely used in downstream tasks in various major directions of artificial intelligence. For example, after fine-tuning the pre-trained model, the visual language model in each optional embodiment of the present application can be obtained. This model is a technology that combines image processing and natural language processing and can process images and text.

[0063] Generally, if an image is not perturbed by noise, a visual language model can output the correct context based on the image and the preceding text. However, if an image is carefully designed to be perturbed by noise, an adversarial image is generated. When the adversarial image and the preceding text are fed into the visual language model, the model will output an incorrect context, reducing its security and reliability.

[0064] Based on this, embodiments of the present application provide a text generation method that can determine whether a target image is an adversarial image. Only when the target image is not an adversarial image is the target image and the first target text input into the visual language model, allowing the visual language model to generate a second target text with higher accuracy, thereby improving the security and reliability of the visual language model.

[0065] like Figure 1 As shown, Figure 11 is a schematic diagram of a computer system for a text generation method provided in an embodiment of the present application. The computer system includes a terminal device 101 and a server 102. An instant messaging client is installed and running on the terminal device 101. The text generation method provided in the embodiment of the present application can be executed by the terminal device 101, by the server 102, or by both the terminal device 101 and the server 102, and this is not limited in the embodiment of the present application.

[0066] Server 102 provides backend services for the client installed on terminal device 101. In one possible implementation, server 102 performs primary computing tasks, while terminal device 101 performs secondary computing tasks. Alternatively, server 102 performs secondary computing tasks, while terminal device 101 performs primary computing tasks. Alternatively, terminal device 101 and server 102 utilize a distributed computing architecture for collaborative computing.

[0067] Optionally, the terminal device 101 may be any electronic device capable of human-computer interaction with a user through one or more methods such as a keyboard, touchpad, remote control, voice interaction, or handwriting device. For example, the terminal device 101 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, PC (Personal Computer), mobile phone, PDA (Personal Digital Assistant), wearable device, PPC (Pocket PC), smart car computer, smart TV, etc.

[0068] Terminal device 101 may generally refer to one of multiple terminal devices. This embodiment uses terminal device 101 as an example. Those skilled in the art will appreciate that the number of terminal devices 101 may be greater or lesser. For example, there may be only one terminal device 101, or there may be dozens, hundreds, or even more terminal devices 101. This embodiment of the application does not limit the number or type of terminal devices 101.

[0069] The server 102 is a single server, or a server cluster consisting of multiple servers, or any one of a cloud computing platform and a virtualization center, which is not limited in the embodiments of the present application. The server 102 is directly or indirectly connected to the terminal device 101 via a wired network or a wireless network. The server 102 has a data receiving function, a data processing function, and a data sending function. Of course, the server 102 may also have other functions, which are not limited in the embodiments of the present application.

[0070] In an exemplary embodiment of the present application, the target object 103 may use the terminal device 101. For example, the target object may input the target image 104 and the first target text 105 into the terminal device 101. The terminal device 101 sends the target image 104 and the first target text 105 to the server 102 via a wireless network or a wired network.

[0071] Server 102 can pre-train a target classifier using clean images and adversarial images. When server 102 receives target image 104, it inputs target image 104 into the target classifier, and the target classifier determines the classification result of target image 104. If the target image 104 is determined to be a clean image based on the classification result, server 102 inputs target image 104 and first target text 105 into a visual language model, and outputs a second target text through the visual language model. If the target image 104 is determined to be an adversarial image based on the classification result, server 102 generates a prompt message to indicate that the second target text cannot be generated through the visual language model, for example, the prompt message is "Reject Answer."

[0072] The server 102 may send the second target text or prompt information to the terminal device 101 via a wireless network or a wired network. The terminal device 101 may display the target image 104, the first target text 105, and the second target text 106. Alternatively, the terminal device 101 may display the target image 104, the first target text 105, and the prompt information.

[0073] Those skilled in the art should understand that the above-mentioned terminal device 101 and server 102 are merely examples, and other existing or future terminal devices or servers, if applicable to the present application, should also be included in the scope of protection of the present application and are incorporated herein by reference.

[0074] The embodiment of the present application provides a text generation method, which can be applied to the above-mentioned computer system. In this example, the terminal device 101 or the server 102 can be collectively referred to as an electronic device, and the electronic device is used to execute the method of the embodiment of the present application, that is, the text generation method of the embodiment of the present application is executed by the electronic device. Figure 2 As shown, the method includes the following steps.

[0075] Step 201: Acquire a target image.

[0076] The target image can be an image of any content, style, or type. For example, the target image can be an image in a sketch style, a realistic style, or a comic style. The target image can also be an image of scenery, a person, a building, or the like. The method for acquiring the target image is not limited herein. For example, the target image can be an image captured by an electronic device using an image acquisition device, or an image input by a target object, or an image acquired via the Internet.

[0077] In step 202 , the target image is classified to obtain a classification result of the target image. The classification result indicates whether the target image is an adversarial image. The adversarial image is an image disturbed by noise.

[0078] The electronic device can obtain a trained target classifier and use the target classifier to classify the target image. The embodiment of the present application does not limit the structure of the target classifier. For example, the target classifier can be a network such as a support vector machine (SVM), a decision tree, or a multilayer perceptron (MLP), and can also include at least one of a linear layer, an activation layer, a normalization layer, etc. The embodiment of the present application also does not limit the input of the target classifier. For example, the input of the target classifier can be a target image, or it can be information determined based on the target image. The information can be the target attention information mentioned below, or it can be a feature used to characterize the visual information of the target image (i.e., a target image feature).

[0079] It's understandable that the target classifier's data processing methods vary depending on its input and structure. For example, a target classifier includes a linear layer and an activation layer. After the target image features are input into the target classifier, they are first linearly mapped using the linear layer. The activation layer then activates the linear mapping result to produce the classification result.

[0080] The classification result is a numerical value that can indicate whether the target image is an adversarial image or not. For example, a classification result of 1 indicates that the target image is an adversarial image, and a classification result of 0 indicates that the target image is not an adversarial image.

[0081] Alternatively, the numerical value of the classification result may be the probability that the target image is perturbed by noise, where the larger the numerical value, the higher the probability that the target image is perturbed by noise. Optionally, the classification result is a probability value between 0 and 1. If the classification result is greater than a probability threshold, it indicates that the target image is highly likely to be perturbed by noise, and the classification result indicates that the target image is an adversarial image; if the classification result is not greater than the probability threshold, it indicates that the target image is less likely to be perturbed by noise, and the classification result indicates that the target image is not an adversarial image.

[0082] In a possible implementation, the target image is related to the third target text, and step 202 includes steps A1 and A2 (not shown in the figure).

[0083] Step A1: determining target attention information based on a target image, where the target attention information represents information about the influence of at least one image block in the target image on at least one character in a third target text.

[0084] The target image includes multiple image blocks, and the shape of any image block is not limited here. For example, the image block is rectangular or irregular in shape, and the size and shape of any two image blocks are the same or different. The electronic device can determine a semantically related third target text based on the target image. The content of the third target text is not limited here. For example, if the target image is an image containing multiple animals, the third target text can be the type of animal in the target image, for example, the third target text is "cow, zebra, deer", or the third target text can be the number of animals in the target image, for example, the third target text is "7 animals", etc.

[0085] In the process of determining the third target text based on the target image, an attention mechanism can be used to determine target attention information, and the third target text can be determined based on the target attention information. It should be noted that the electronic device may not determine the third target text, but only determine the target attention information. The attention mechanism is a technology derived from the study of human vision. It enables the model to focus on important parts of the input data, improving the model's performance and efficiency. In the field of deep learning, especially in sequence models such as natural language processing and computer vision, the attention mechanism plays an important role. It mimics the human attention process, enabling the model to focus on key information in the input data and ignore less relevant information. By assigning different attention scores or weights to different parts of the input data, the model can identify the most important information.

[0086] In this example, the input data includes a target image, the information recognized by the model (i.e., the output data) includes a third target text, and the target attention information includes an attention score or weight assigned to at least one image block in the target image. The attention score or weight of any image block represents the influence of the image block on a character in the third target text. It is understood that the larger the value of the influence information, the greater the degree of influence of the image block on the character.

[0087] In a possible implementation, step A1 includes steps A11 to A13 (not shown in the figure).

[0088] Step A11 : determining a plurality of visual tokens based on the target image, wherein any visual token represents visual information of an image block in the target image.

[0089] In an embodiment of the present application, the target image can be divided into multiple image blocks, and each image block is input into a visual encoder. The visual encoder is a module for performing feature extraction on an image, and its structure is not limited here. For example, the visual encoder can be any one of a Transformer module, an Auto-Encoder (AE), a Variational Auto-Encoder (AE), etc. The structure of the visual encoder is different, and the way of performing feature extraction on the image block by the visual encoder is also different, which will not be repeated here.

[0090] The visual encoder performs feature extraction on any image block to obtain image block features, which represent the visual information of the image block in terms of color, texture, structure, content, etc. The image block features can be used as visual tokens, or the image block features can be converted into visual tokens through a visual language alignment module. Among them, the visual language alignment module is a module that formally aligns visual features with language features, and its structure is not limited here. For example, the visual language alignment module includes at least one of a linear layer, a normalization layer, a feedforward network, etc. Assuming that the language feature is an N-dimensional vector and the image block feature is an M-dimensional vector, and M is different from N, then: after the image block feature is input into the visual language alignment module, the M-dimensional vector can be converted into an N-dimensional vector through the visual language alignment module to obtain a visual token, so that the visual token conforms to the form of the language feature and can represent the visual information of the image block.

[0091] Step A12: determining a plurality of first attention scores based on the plurality of visual tokens, wherein any first attention score represents information about the influence of an image block corresponding to a visual token on a first character of at least one character.

[0092] In an embodiment of the present application, the first character in the third target text is generated based on multiple visual tokens through a large language model. The large language model includes a text encoder, which can be any one of an embedding layer, a Transformer module, an autoencoder, a variational autoencoder, etc. The start tag can be input into the text encoder, and the first language token used to represent the start tag is determined by the text encoder. The start tag is a special tag, for example, the start tag is [CLS] or other arbitrary symbols. The large language model also includes a text generation module, which is a Transformer module or other network containing an attention module. The first language token and multiple visual tokens can be input into the text generation module, and the first character is generated by the text generation module.

[0093] Assume that the text generation module is a Transformer module, which includes at least one network layer, and each network layer includes at least one attention network. In this example, for the first network layer, the first attention network performs attention processing on multiple visual tokens and the first language token to obtain the processing result of the first attention network; the non-first attention network performs attention processing on multiple visual tokens and the processing result of the previous attention network to obtain the processing result of the non-first attention network; and the processing result of the last attention network is determined based on the processing result of the first network layer. For the non-first network layer, the first attention network performs attention processing on multiple visual tokens and the processing result of the previous network layer to obtain the processing result of the first attention network; the non-first attention network performs attention processing on multiple visual tokens and the processing result of the previous attention network to obtain the processing result of the non-first attention network; and the processing result of the non-first network layer is determined based on the processing result of the last attention network. The processing result of the last network layer represents the language information of the target image. Based on the processing result of the last network layer, the probabilities of multiple characters in the dictionary are determined, and the character with the highest probability is used as the first character in the third target text.

[0094] Taking the first attention network in the first network layer as an example, in this example, on the one hand, the query vector can be determined based on the first language token, and on the other hand, for any visual token, the key vector and value vector can be determined based on the visual token. Then, the first attention score is determined based on the query vector and the key vector. , , where q represents the query vector, T is the transpose symbol, The key vector representing the i-th visual token, and the dimension of the query vector and key vector are both d. The first attention score is then normalized to obtain a first attention weight. Based on the first attention weight, the value vector is weighted to obtain an attention processing result. In this way, the attention processing result corresponding to each visual token can be determined based on the query vector and the key and value vectors of each visual token, and the processing result of the first attention network can be determined based on each attention processing result.

[0095] It is understandable that each attention network in the text generation module can determine the processing result according to the implementation principle of the first attention network described above. Since the text generation module is used to determine the first character in the third target text, based on this, the larger the first attention score, the greater the influence of the image block corresponding to the first attention score on the first character.

[0096] Step A13: Determine target attention information based on multiple first attention scores.

[0097] All or part of the first attention scores can be used to form target attention information in matrix form. The matrix includes elements of m rows and n columns, where m and n are both positive integers, and one element is a first attention score.

[0098] Since a first attention score represents the influence information of an image block on the first character, the target attention information is determined by combining multiple first attention scores, so that the target attention information can represent the influence information of at least one image block on the first character, thereby improving the accuracy of the target attention information.

[0099] In an exemplary embodiment, the at least one character in the third target text further includes a non-first character in the third target text. Step A13 includes: determining a plurality of second attention scores based on the plurality of visual tokens and characters preceding the non-first character in the at least one character, wherein each second attention score represents information about the impact of an image block corresponding to a visual token on the non-first character; and determining target attention information based on the plurality of first attention scores and the plurality of second attention scores.

[0100] The method for determining the first character based on multiple first attention scores has been described above and will not be repeated here. The first character and multiple visual tokens can be input into the large language model, and the second character of the third target text can be generated using the large language model. The first character, the second character, and multiple visual tokens can be input into the large language model, and the third character of the third target text can be generated using the large language model. And so on, until the last character of the third target text is generated.

[0101] For any character from the second character to the last character, the character is a non-first character, and the electronic device can input the characters before the non-first character and multiple visual tokens into the large language model. The text encoder is used to determine the second language token, and the second language token is used to represent any character before the non-first character. The second language token and the multiple visual tokens are input into the text generation module, and the non-first character is generated by the text generation module. Optionally, the first language token can also be input into the text generation module. The generation principle of the non-first character is similar to that of the first character, and will not be repeated here.

[0102] In the process of generating non-first characters, multiple second attention scores need to be generated. The principle of generating the second attention scores is similar to that of generating the first attention scores and will not be repeated here. All or part of the first attention scores and all or part of the second attention scores can be used to form the target attention information in the form of a matrix, where any element in the matrix is ​​a first attention score or a second attention score.

[0103] According to the above description, in the process of determining a character, it is necessary to first determine the first attention score or the second attention score, and then determine the character based on the attention score. In actual applications, the electronic device can only determine the first attention score or the second attention score without determining the character.

[0104] Since a first attention score represents the influence information of an image block on the first character, and a second attention score represents the influence information of an image block on non-first characters, the target attention information is determined by combining multiple first attention scores and multiple second attention scores, so that the target attention information can represent the influence information of at least one image block on multiple characters, thereby improving the accuracy of the target attention information.

[0105] In a possible implementation, step A1 includes: obtaining a fourth target text, where the fourth target text has a contextual relationship with the third target text; and determining target attention information based on the fourth target text and the target image.

[0106] In the embodiment of the present application, the text used to determine attention information and classify the image based on the attention information is called probe text, that is, the fourth target text is the probe text. The fourth target text can be text entered into the electronic device by the target subject, or pre-set text, or text obtained from the Internet, etc.

[0107] The fourth target text can be a question text or a prompt text. The question text can be a text with a quantity as the answer, for example, the fourth target text is "What animals are in the image?", or a text with a yes or no answer, for example, the fourth target text is "Is there a clock in the image?", or other text besides the above two types of text, for example, the fourth target text is "What is in the image?". The prompt text is used to prompt the model to generate the third target text, for example, the prompt text is "Please use one sentence to describe the style of the image."

[0108] The fourth target text is in a contextual relationship with the third target text. A contextual relationship is a relationship in which the preceding and following texts are coherent in terms of semantics, logic, grammar, and so on. Optionally, the fourth target text is the preceding text of the third target text. For example, if the third target text is "cow, zebra, deer," the fourth target text could be "What animals are in the image?" For another example, if the third target text is "7 animals," the fourth target text could be "How many animals are in the image?" Alternatively, the fourth target text is the following text of the third target text. For example, if the fourth target text is "cow, zebra, deer," the third target text could be "What animals are in the image?"

[0109] As mentioned above, the electronic device can determine multiple visual tokens based on the target image through a visual encoder. In addition, the electronic device can also determine multiple third language tokens based on a fourth target text through a text encoder, where any third language token represents a character in the fourth target text. Multiple visual tokens, first language tokens, and multiple third language tokens can be input into a text generation module, and the first character can be generated by the text generation module. Multiple visual tokens, first language tokens, at least one second language token, and multiple third language tokens can be input into a text generation module, and non-first characters can be generated by the text generation module.

[0110] Multiple first attention scores can be determined based on multiple visual tokens and multiple third language tokens. Target attention information can be determined based on multiple first attention scores, or the first character can be determined based on multiple first attention scores; multiple second attention scores can be determined based on multiple visual tokens, multiple third language tokens, and characters before non-first characters; and target attention information can be determined based on multiple first attention scores and multiple second attention scores. The processing methods of third language tokens and visual tokens are similar. Based on this, the generation principles of the first character and non-first character can be found in the description of step A12 and will not be repeated here.

[0111] It should be noted that the electronic device can determine a query vector based on the first language token or any second language token, and determine a key vector and a value vector based on any visual token or any third language token. Then, an attention score is determined based on the query vector and the key vector, and an attention processing result is determined based on the attention score and the value vector. In this way, attention processing is performed on the first language token, each second language token, each visual token, and each third language token, but the target attention information only includes the attention score generated based on the visual token.

[0112] In the embodiment of the present application, the generation of the third target text is constrained by the fourth target text. In the process of generating the third target text from the fourth target text and the target image, target attention information is generated, so that the target attention information represents the influence of the image block on the characters in the specific text. This constrains the generation of the target attention information, facilitates the subsequent classification of the target image based on the target attention information, and improves the accuracy of the classification results.

[0113] Optionally, the fourth target text is a question with a yes or no answer, and the third target text is the answer. The fourth target text is arbitrary text and may or may not be related to the target image. For example, the fourth target text could be "Is there a clock in the image?" or "Is there an animal in the image?" Since the third target text is determined based on the target image and the fourth target text, it can be a context of the fourth target text, and based on this, the third target text can be "yes" or "no."

[0114] Research has shown that there are differences in the patterns of attention information determined based on different question texts. Figure 3 As shown, Figure 3 (1) in FIG shows a schematic diagram of attention information determined based on a question text with a yes or no answer. Figure 3 (2) in the figure shows the attention information determined based on the question text with the answer being the number, Figure 3 (3) in shows the attention information determined based on other question texts, where other question texts are question texts other than question texts with answers of yes or no and question texts with answers of quantity. Figure 3 In FIG, the horizontal axis 301 represents the number of visual tokens, the vertical axis 302 represents the number of images, and the area indicated by the number 303 represents the correspondence between the value and the grayscale.

[0115] from Figure 3 It can be seen that compared with Figure 3 (2) and (3) in the above, Figure 3(1) in is more regular. That is, the attention information generated based on the question text with the answer being yes or no is more consistent in pattern, making the target attention information more conducive to the classification of the target image and improving the accuracy of the classification results.

[0116] Step A2: classify the target image according to the target attention information to obtain the classification result.

[0117] In the embodiments of this application, images that are not perturbed by noise are called clean images, and images that are perturbed by noise are called adversarial images. Research has shown that when the probe text is fixed, there are differences in the patterns of attention information determined based on clean images and adversarial images.

[0118] like Figure 4 As shown, Figure 4 (1) is the attention information determined based on the clean image. Figure 4 (2) is based on attention information determined by an adversarial image. Figure 4 (3) in is the attention information determined based on another adversarial image. Figure 4 (1) to (3) all use the same probe text "Is there a clock in the image?". In addition, Figure 4 The adversarial image in (2) is generated by the Contrastive Language-Image Pre-Training (CLIP) model. Figure 4 The adversarial image in (3) is generated by the Visual Language Model (VLM). Figure 4 In FIG, the horizontal axis 401 represents the number of visual tokens, the vertical axis 402 represents the number of images, and the area indicated by the label 403 represents the correspondence between the value and the grayscale.

[0119] from Figure 4 It can be seen that Figure 4 (1) and Figure 4 There are significant differences between (2) and (3) in terms of attention information, while Figure 4 (2) and (3) in the above are similar in terms of attention information. Based on this, the target image is classified according to the target attention information, and a classification result that can characterize whether the target image is disturbed by noise is obtained, which can ensure the accuracy of the classification result.

[0120] In an embodiment of the present application, the classification result is obtained by classifying the target image according to the target attention information through the target classifier. That is to say, the target attention information is input into the target classifier, and the classification result of the target image is determined by the target classifier. The structure of the target classifier is not limited here. For example, the target classifier can be any network such as a support vector machine (SVM), a decision tree or a multilayer perceptron (MLP), or the target classifier is a network composed of a linear layer and an activation layer. The structure of the target classifier is different, and the way of determining the classification result by the target classifier is also different, which will not be repeated here.

[0121] In a possible implementation, there are multiple target attention information. Step A2 includes steps A21 to A23 (not shown in the figure).

[0122] Step A21: for any target attention information, classify the target image according to any target attention information to obtain a sub-result, and the sub-result represents whether the target image is an adversarial image.

[0123] In an embodiment of the present application, multiple target attention information can be determined based on the target image according to the implementation principle of step A1. Optionally, the electronic device can obtain multiple different fourth target texts and determine corresponding target attention information based on each fourth target text and the target image, thereby obtaining multiple target attention information. The target image can be classified according to each target attention information by a target classifier to obtain a corresponding sub-result.

[0124] The input and structure of the target classifier vary, and the method for determining sub-results also varies. For example, the target classifier includes a linear layer and an activation layer. The target attention information can be input into the target classifier, linearly mapped to the target attention information through the linear layer, and activated through the activation layer to obtain the sub-result.

[0125] The sub-result is a numerical value that can indicate whether the target image is an adversarial image or not. Alternatively, the sub-result represents the probability that the target image is perturbed by noise; the larger the sub-result, the higher the probability that the target image is perturbed by noise. Alternatively, the sub-result is a probability value between 0 and 1. If the sub-result is greater than a probability threshold, the sub-result indicates that the target image is an adversarial image; if the sub-result is not greater than the probability threshold, the sub-result indicates that the target image is not an adversarial image, that is, the target image is a clean image.

[0126] Step A22: If the number of target sub-results is not less than the number threshold, it is determined that the classification result indicates that the target image is not an adversarial image, and the target sub-result is a sub-result indicating that the target image is not an adversarial image.

[0127] In this example, the quantity thresholds are all pre-set values ​​based on manual experience. The quantity threshold is no greater than the number of target attention information, and the number of target attention information and the fourth target text are the same. For example, if there are three fourth target texts, then the number of target attention information is also three, and the quantity threshold can be 2 or 3.

[0128] For any sub-result, if the sub-result indicates that the target image is not an adversarial image, the sub-result is considered a target sub-result. If the number of target sub-results is at least a threshold, this indicates that the target image was judged to be a clean image with a high probability multiple times, thereby indicating that the target image has a higher probability of being a clean image. Based on this, the electronic device determines that the classification result indicates that the target image is not an adversarial image.

[0129] Step A23: If the number of target sub-results is less than the number threshold, it is determined that the classification result characterizes the target image as an adversarial image.

[0130] If the number of target sub-results is less than the threshold, it indicates that the target image is judged to be a clean image less frequently, thereby indicating that the target image is more likely to be an adversarial image. Based on this, the electronic device determines that the classification result indicates that the target image is an adversarial image.

[0131] The method used in steps A22 and A23 to determine the classification result is a voting method. When multiple sub-results indicate that the target image is not an adversarial image, the classification result is determined to indicate that the target image is not an adversarial image. When only a few sub-results indicate that the target image is not an adversarial image, the classification result is determined to indicate that the target image is an adversarial image. Determining the classification result by voting can reduce the probability of error in the classification result and improve the accuracy of the classification result. It should be noted that the target classifier used for classification is a trained network, and the training method is as follows.

[0132] In this example, the method of the embodiment of the present application further includes steps B1 to B4 (not shown in the figure), wherein steps B1 to B4 are performed before step A2.

[0133] Step B1: Obtain a first sample image, where the first sample image is not an adversarial image.

[0134] The method for acquiring the first sample image is similar to that for acquiring the target image and will not be further described here. The first sample image is a clean image, unperturbed by noise. In this example, a clean image is one that produces correct results using a trained neural network model. For example, for a clean image containing seven zebras, a trained neural network model can be used to obtain a result indicating that the clean image contains zebras or seven animals.

[0135] Step B2: determining first attention information based on the first sample image, where the first attention information represents information about the influence of at least one image block in the first sample image on at least one character in a first sample text, and the first sample text is related to the first sample image.

[0136] The first sample image includes multiple image blocks. The electronic device can determine semantically related first sample text based on the first sample image. During this determination, an attention mechanism can be used to determine first attention information, and the first sample text can be determined based on the first attention information. The implementation principle of step B2 is similar to that of step A1 and will not be repeated here.

[0137] In step B3, the first sample image is classified according to the first attention information by an initial classifier to obtain a first prediction result, where the first prediction result indicates whether the first sample image is an adversarial image.

[0138] In this example, the initial classifier is used to be trained into the target classifier. Therefore, the initial classifier and the target classifier are similar in structure and function. That is, the implementation principle of step B3 is similar to that of step A2 and will not be repeated here.

[0139] Step B4: training an initial classifier based on the first prediction result to obtain a first classifier; and determining a target classifier based on the first classifier.

[0140] The electronic device obtains a first annotation result, and the first annotation result characterizes that the first sample image is not disturbed by noise. The first loss between the first annotation result and the first prediction result can be calculated, and the first loss is used as the loss of the initial classifier. The parameters of the initial classifier are adjusted based on the loss of the initial classifier, and the initial classifier is trained once to obtain a trained initial classifier. If the trained initial classifier meets the training end condition, the trained initial classifier is used as the target classifier. If the trained initial classifier does not meet the training end condition, the trained initial classifier is used as the initial classifier for the next training, and the next training is performed on the initial classifier according to the implementation principle of step B3 to step B4. And so on, through multiple training, a first classifier is obtained. The first classifier can be used as the target classifier, or the first classifier can be trained again to obtain the target classifier.

[0141] The embodiments of the present application do not limit the requirement that the trained initial classifier satisfies the training termination condition. For example, the trained initial classifier satisfies the training termination condition including: the trained initial classifier has been trained a number of times set based on manual experience, for example, the trained initial classifier has been trained 500 times, or the performance index of the trained initial classifier has reached an index set based on manual experience.

[0142] The initial classifier is trained using the attention information of the clean image, so that the classifier is optimized in the direction of making the prediction results determined based on the attention information close to the annotation results of the clean image, thereby improving the classifier's ability to distinguish the attention information of the clean image, so that the classifier can accurately determine the classification results based on the attention information.

[0143] In another possible implementation, the method of the embodiment of the present application further includes steps B5 to B8 (not shown in the figure), wherein steps B5 to B8 are performed before step A2.

[0144] Step B5: Obtain a second sample image, where the second sample image is an adversarial image.

[0145] The second sample image is acquired in a similar manner to the target image and will not be further described here. The second sample image has been perturbed by noise and is therefore an adversarial image. In this example, an adversarial image is one that produces an incorrect result using a trained neural network model. For example, a clean image containing seven zebras can be trained using a trained neural network model to produce a result that indicates the clean image contains a deer or ten animals.

[0146] In an exemplary embodiment, step B5 includes steps B51 and B52 (not shown in the figure).

[0147] Step B51: Generate a fourth sample text based on the first sample image and the third sample text related to the first sample image through the visual language model. The first sample image is not an adversarial image, and the fourth sample text is related to the first sample image and has a contextual relationship with the third sample text.

[0148] There are multiple first sample images, each of which corresponds to a third sample text. The embodiments of this application do not limit the method for obtaining the first sample images and the corresponding third sample text. For example, the first sample images and the corresponding third sample text are data pairs obtained via the Internet, or the first sample images are images captured by an image capture device, and the third sample text is text entered by a user, etc.

[0149] The first sample image and the third sample text are input into the visual language model, and the fourth sample text is generated by the visual language model. The generation process can be seen in the description of step 204, which will not be repeated here. The third sample text and the fourth sample text are both related to the first sample image, and the fourth sample text is the context of the third sample text, or the fourth sample text is the context of the third sample text. For example, if the first sample image is an image containing a train station, then when the third sample text is "Where was the picture taken?", the fourth sample text is "train station". For another example, if the first sample image is an image containing a cow, then when the third sample text is "What animals are in the image?", the fourth sample text is "cows".

[0150] Step B52: Based on the fourth sample text, perform noise perturbation on the first sample image to obtain a second sample image.

[0151] In an embodiment of the present application, the loss can be determined based on the encoding of the fourth sample text according to the calculation formula of the loss function, and noise can be added to the first sample image according to the loss, so that the semantics of the text determined based on the image after adding noise deviates from the semantics of the fourth sample text, thereby achieving noise perturbation of the first sample image.

[0152] By performing noise perturbation on the first sample image through the fourth sample text, the possibility of the model determining the fourth sample text based on the first sample image is reduced, so that the first sample image after noise perturbation approaches the adversarial image.

[0153] Exemplarily, step B52 includes: acquiring noise text that is semantically different from the fourth sample text; and performing noise perturbation on the first sample image based on the noise text and the fourth sample text to obtain a second sample image.

[0154] The noise text is any text that is semantically different from the fourth sample text. For example, if the fourth sample text is "train station," the noise text could be a bus station, an airport, etc. Another example is if the fourth sample text is "cow," the noise text could be a zebra, a deer, a bush, etc. The method for obtaining the noise text is not limited herein. For example, the noise text could be text read from the internet, or could be text entered by a user.

[0155] In an embodiment of the present application, the noise text and the fourth sample text can be encoded separately, and the loss can be determined based on the two encodings according to the calculation formula of the loss function. Noise is added to the first sample image based on the loss, so that the semantics of the text determined based on the image after adding noise deviates from the semantics of the fourth sample text and approaches the semantics of the noise text, thereby achieving noise perturbation of the first sample image.

[0156] By performing noise perturbation on the first sample image through the noise text and the fourth sample text, the possibility of the model determining the fourth sample text based on the first sample image is reduced, and the possibility of the model determining the noise text based on the first sample image is increased, so that the first sample image after noise perturbation approaches the adversarial image.

[0157] According to the implementation principle of step B51 to step B52, noise perturbation can be performed on the first sample image multiple times. After each noise perturbation is performed on the first sample image, it is determined whether the first sample image after noise perturbation meets the conditions. If the conditions are met, the first sample image after noise perturbation is used as the second sample image. Otherwise, noise perturbation is performed again on the first sample image after noise perturbation. And so on, until the second sample image is obtained. The first sample image after noise perturbation meets the conditions including: the visual language model determines the noise text based on the third sample text and the first sample image after noise perturbation, or the confidence level characterizing the first sample image after noise perturbation as an adversarial image is greater than a value set based on manual experience.

[0158] In this example, an adversarial attack involves adding noise to the input data (e.g., the first sample image) to cause the model to misjudge the input data. The added noise is called an adversarial perturbation, and the resulting sample (e.g., the second sample image) is called an adversarial example. Various adversarial attack methods can be used. Following the implementation principles of steps B51 and B52, multiple second sample images are generated and used to train a target classifier. This makes the target classifier robust and improves its accuracy. Different adversarial attack methods can correspond to different loss functions and / or noise intensities.

[0159] Step B6: determining second attention information based on the second sample image, where the second attention information represents information about the influence of at least one image block in the second sample image on at least one character in the second sample text, and the second sample text is related to the second sample image.

[0160] The second sample image includes multiple image blocks. The electronic device can determine semantically related second sample text based on the second sample image. During this determination, an attention mechanism can be used to determine second attention information, and the second sample text can be determined based on the second attention information. The implementation principle of step B6 is similar to that of step A1 and will not be repeated here.

[0161] In step B7, the initial classifier classifies the second sample image based on the second attention information to obtain a second prediction result, which indicates whether the second sample image is an adversarial image. The implementation principle of step B7 is similar to that of step A2 and will not be repeated here.

[0162] Step B8: training an initial classifier based on the second prediction result to obtain a second classifier; and determining a target classifier based on the second classifier.

[0163] The electronic device obtains a second annotation result, and the second annotation result characterizes that the second sample image has been disturbed by noise. The second loss between the second annotation result and the second prediction result can be calculated, and the second loss is used as the loss of the initial classifier. The parameters of the initial classifier are adjusted based on the loss of the initial classifier, and the initial classifier is trained once to obtain a trained initial classifier. If the trained initial classifier meets the training end condition, the trained initial classifier is used as the target classifier. If the trained initial classifier does not meet the training end condition, the next training is performed on the trained initial classifier. After multiple trainings, a second classifier is obtained. The second classifier can be used as the target classifier, or the second classifier can be trained again to obtain the target classifier.

[0164] The initial classifier is trained by the attention information of the adversarial image, so that the classifier is optimized in the direction of making the prediction result determined based on the attention information close to the annotation result of the adversarial image, thereby improving the classifier's ability to distinguish the attention information of the adversarial image, so that the classifier can accurately determine the classification result based on the attention information.

[0165] In practical applications, clean images and adversarial images can be used to train an initial classifier, improving the classifier's ability to discern attention information in clean images and in adversarial images, allowing the classifier to accurately determine classification results based on attention information. In other words, the initial classifier can be trained based on the first and second prediction results to obtain a target classifier. The training method is described in steps B4 and B8 and will not be further described here.

[0166] Alternatively, a first classifier may be trained according to steps B1 to B4, and then the first classifier may be trained according to the implementation principles of steps B5 to B8 to obtain a target classifier. Alternatively, a second classifier may be trained according to steps B5 to B8, and then the second classifier may be trained according to the implementation principles of steps B1 to B4 to obtain a target classifier.

[0167] Afterwards, a target classifier is used to determine the classification result based on the target attention information of the target image.

[0168] Step 203: When it is determined based on the classification result that the target image is not an adversarial image, a first target text related to the target image is obtained.

[0169] The first target text corresponds to the target image, and its content is as shown in step B51, which will not be repeated here.

[0170] Step 204 : Generate a second target text based on the target image and the first target text using a visual language model. The second target text is related to the target image and has a contextual relationship with the first target text.

[0171] In an embodiment of the present application, the target image and the first target text are input into a visual language model, and a second target text is generated by the visual language model. The second target text is similar to the fourth sample text, as can be seen in the description of step B51, which will not be repeated here. The embodiment of the present application does not limit the structure of the visual language model. Exemplarily, the visual language model is any one of a CLIP model, a VLM model, a BLIP (Bootstrapping Language-Image Pre-training, a unified pre-training of visual language understanding and generation) model, an InstructBLIP (Instruct-Based Language-Image Pretraining, an instruction-based cross-modal pre-training) model, or an InstructBLIP model (for example, the InstructBLIP Vicuna-7B model, X-InstructBLIP), etc. The structure of the visual language model is different, and the implementation principle of step 204 is also different, which will not be repeated here.

[0172] In a possible implementation, step 204 includes steps C1 to C3 (not shown in the figure).

[0173] In step C1, feature extraction is performed on the target image through a visual language model to obtain target image features, where the target image features represent visual information of the target image.

[0174] In this example, the visual language model includes the visual encoder mentioned above. The target image includes multiple image blocks, and the features of each image block are determined according to the implementation principle of step A11. The features of each image block can be used as visual tokens, or the visual language model includes the visual language alignment module mentioned above, and the features of each image block are converted into visual tokens through the visual language alignment module. The target image features include various visual tokens. For example, according to the positional relationship of each image block in the target image, each visual token is spliced ​​to obtain the target image features. Since the visual token represents the visual information of the image block, the target image features containing each visual token can represent the visual information of the target image.

[0175] Step C2: obtaining a first text feature through a visual language model, where the first text feature represents language information of the first target text.

[0176] In this example, the visual language model includes the large language model mentioned above. Since the large language model includes a text encoder, each character in the first target text can be encoded by the text encoder to obtain each fourth language token, and any fourth language token represents a character in the first target text. The first text feature contains each fourth language token. For example, according to the positional relationship of each character in the first target text, each fourth language token is spliced ​​to obtain the first text feature. Since the fourth language token represents a character, and the language information includes the combination of multiple characters, the semantics of the characters themselves, and the semantics of the text composed of multiple characters, etc., the first text feature containing each fourth language token can represent the language information of the first target text.

[0177] Step C3: Generate a second target text based on the target image features and the first text features through a visual language model.

[0178] The visual language model also includes the aforementioned text generation module. The target image features and the first text features are input into the text generation module, which then generates a second target text. This generates the second target text based on the target image features and the first text features. This allows for feature-level analysis of the visual information of the target image features and the linguistic information of the first target text, ensuring that the semantics of the second target text match those of the fusion of the target image and the first target text, thus ensuring the accuracy of the second target text.

[0179] In an exemplary embodiment, step C3 includes: generating a first character of the second target text based on the target image features and the first text features using a visual language model; generating non-first characters based on the target image features, the first text features, and each character preceding the non-first character in the second target text using a visual language model; and determining the second target text based on the first character and the non-first characters.

[0180] In an embodiment of the present application, the electronic device can determine the probabilities of multiple characters in the dictionary based on the target image features and the first text features through a text generation module, and use the character with the highest probability as the first character in the second target text. Then, the second text feature is determined through a text encoder, and the second text feature represents each character before the non-first character in the second target text. Through the text generation module, based on the target image features, the first text features and the second text features, the probabilities of multiple characters in the dictionary are determined, and the character with the highest probability is used as a non-first character. The characters are spliced ​​in the order of generation to obtain the second target text. The generation method of the second target text can be seen in the description of step A12. The implementation principles of the two are similar and will not be repeated here.

[0181] The first character is generated using the target image features and the first text features, making it semantically related to the target image and the first target text. Non-first characters are generated using the target image features, the first text features, and each character preceding the non-first character, making them semantically related to both the target image and the first target text, as well as each of the preceding characters. This allows the non-first character and each of the preceding characters to form a semantically coherent text, thereby improving the accuracy of the second target text.

[0182] In a possible implementation, the method of the embodiment of the present application further includes step D (not shown in the figure), which is performed after step 202.

[0183] In step D, when the target image is determined to be an adversarial image based on the classification result, prompt information is generated, where the prompt information is used to indicate that the second target text cannot be generated through the visual language model.

[0184] The embodiments of the present application do not limit the content of the prompt message. For example, the prompt message is "Refuse to answer." or "Please enter a clean image." In this example, if the target image is an adversarial image. Since the visual language model will output an incorrect result for adversarial images, if the target image is an adversarial image, by executing step D instead of executing steps 203 and 204, the probability of the model processing adversarial images can be reduced, thereby improving the reliability and security of the model.

[0185] Optionally, the algorithm that implements the method of the embodiment of the present application, that is, the algorithm that implements steps 201 to 204 and step D, is referred to as a detection model. Generally, a visual language model cannot know whether it is under an adversarial attack, so that the visual language model will determine the correct result for a clean image and the wrong result for an adversarial image. If a detection model is deployed, it can be used to determine whether the visual language model is under an adversarial attack, so that the visual language model can determine the correct result for a clean image and reject the answer for an adversarial image.

[0186] like Figure 5 As shown, Figure 5 This is a schematic diagram showing the effect of deploying a detection model, as provided in an embodiment of the present application. If the electronic device does not deploy a detection model, the visual language model will determine the correct answer for clean images and the incorrect answer for adversarial images. If the electronic device deploys a detection model, the visual language model will refuse to answer for adversarial images.

[0187] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, storage, and display, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant region. For example, the images and text involved in this application were obtained with full authorization.

[0188] The above method classifies a target image and obtains a classification result indicating whether the target image is an adversarial image. Since adversarial images are noise-perturbed images, if the classification result indicates that the target image is not an adversarial image, it indicates that the target image has not been perturbed by noise. This allows the visual language model to generate a highly accurate second target text based on the target image and the first target text, thereby improving the security and reliability of the visual language model.

[0189] The above describes the text generation method from the perspective of method steps, and the following is a systematic and comprehensive description. In an embodiment of the present application, a target classifier for distinguishing clean images from adversarial images can be trained first, and the target classifier can be used to classify the target image. If the target image is determined to be a clean image based on the classification result, the target image is processed by the visual language model. If the target image is determined to be an adversarial image based on the classification result, the target image is refused to be processed using the visual language model. The embodiment of the present application is applicable to visual language models of arbitrary structures and classifiers of arbitrary structures. For example, the visual language model includes any one of the CLIP model, VLM model, BLIP model, etc., and the classifier includes any one of the SVM, decision tree, MLP, etc. Specifically, Figure 6As shown, the text generation method includes the following steps.

[0190] Step 601: Obtain a clean image that is not disturbed by noise and a reference question.

[0191] Step 602: Determine an adversarial image perturbed by noise based on the clean image and the reference problem.

[0192] The clean image corresponds to the first sample image mentioned above, the reference question corresponds to the third sample text mentioned above, and the adversarial image corresponds to the second sample image mentioned above. The implementation principles of steps 601 and 602 can be found in the description of steps B51 and B52, and will not be repeated here.

[0193] Step 603: Based on the clean image, the adversarial image and the probe question, a target classifier is trained.

[0194] The probe question may be any question text with a yes or no answer, for example, the probe question is "Is there a clock in the image?" The implementation principle of step 603 can be found in the description of steps B1 to B8 and will not be repeated here.

[0195] like Figure 7 As shown in the training process in . The electronic device can obtain a clean image (corresponding to the first sample image mentioned above) and a reference question, wherein the model will output the correct answer to the reference question for the clean image. For example, for the clean image and the reference question "Where was the photo taken?", the model will output the correct answer "train station." (corresponding to the fourth sample text mentioned above). By performing an adversarial attack on the clean image, an adversarial image is obtained, and the model will output an incorrect answer to the reference question for the adversarial image. For example, for the adversarial image and the reference question "Where was the photo taken?", the model will output an incorrect answer "park." (corresponding to the noisy text mentioned above). It should be noted that the implementation method of the adversarial attack is as described in steps B51 to B52, and will not be repeated here.

[0196] Assuming the visual language model is a Large Visual Language Model (LVLM), then: A clean image and a probe question can be fed into the LVLM to obtain attention information for the clean image (the first attention information mentioned above). This attention information is then fed into a classifier to obtain a prediction for the clean image (the first prediction result mentioned above). This prediction indicates the likelihood that the image has been perturbed by noise. Since clean images are guaranteed to be free of noise, their label can be set to 0, and a loss is calculated based on the prediction result and 0.

[0197] In addition, the adversarial image and the probe question can be input into the LVLM to obtain the attention information of the adversarial image (the second attention information mentioned above). This attention information is then input into the classifier to obtain the prediction result of the adversarial image (the second prediction result mentioned above). This prediction result represents the likelihood that the image is perturbed by noise. Since the adversarial image is definitely perturbed by noise, the label of the adversarial image can be set to 1, and another loss is calculated based on the prediction result and 1.

[0198] The classifier is trained once using the two losses mentioned above to obtain the trained classifier. The classifier can be trained multiple times to finally obtain the target classifier.

[0199] Step 604 : Determine a classification result based on the target image and the probe question by using a target classifier.

[0200] The implementation principle of step 604 can be found in the description of step 202 and will not be repeated here.

[0201] Step 605: Determine whether the classification result indicates that the target image is a clean image.

[0202] In this example, the classification result represents the likelihood that the target image is perturbed by noise. If the classification result is below the likelihood threshold, the target image is less likely to be perturbed by noise, indicating that the target image is likely a clean image. In other words, the classification result indicates that the target image is a clean image. If the classification result is not below the likelihood threshold, the target image is more likely to be perturbed by noise, indicating that the target image is likely an adversarial image. In other words, the classification result indicates that the target image is not a clean image.

[0203] If the classification result indicates that the target image is a clean image, the electronic device executes step 606 to determine the answer based on the target image and the reference question using a visual language model. If the classification result indicates that the target image is not a clean image, the electronic device executes step 607 to display a prompt message.

[0204] Among them, the reference image corresponds to the first target text mentioned above. Based on this, the implementation principle of step 606 can be seen in the description of steps 203 to 204, and the implementation principle of step 607 can be seen in the description of step D, which will not be repeated here.

[0205] like Figure 7The application process is shown in . The electronic device can obtain a target image and a reference question (corresponding to the first target text mentioned above), wherein the target image can be a clean image or an adversarial image. The target image and the probe question (corresponding to the fourth target text mentioned above) are input into the LVLM to obtain the attention information of the target image (i.e., the target attention information mentioned above), and the attention information is input into the classifier to obtain the prediction result of the target image (i.e., the classification result mentioned above). The classification result represents the possibility that the image is disturbed by noise. Based on the classification result, it can be determined whether the target image is a clean image or an adversarial image.

[0206] If the target image is a clean image, the target image and the reference question are input into the LVLM model to obtain the correct answer. For example, the clean image and the reference question "What are these people looking at?" are input into the LVLM model to obtain the correct answer "Ocean." (corresponding to the second target text mentioned above).

[0207] If the target image is an adversarial image, generally, if the target image and the reference question are input into the LVLM model, an incorrect answer will be obtained. For example, if the adversarial image and the reference question "What are these people looking at?" are input into the LVLM model, the incorrect answer "People" will be obtained, which reduces the reliability of the LVLM model. Based on this, in an embodiment of the present application, if the target image is an adversarial image, a prompt message is generated instead of using the LVLM model to process the target image, thereby improving the reliability of the LVLM model.

[0208] Alternatively, the electronic device can obtain a dataset consisting of clean images and adversarial images. For example, if the electronic device obtains 5,000 clean images, it can generate 5,000 adversarial images based on these clean images, resulting in a dataset of 10,000 images. A target classifier is trained using this data, and its performance is tested.

[0209] In one possible implementation, a target classifier is trained using part of the images in a certain dataset, and the performance of the target classifier is tested using another part of the images in the dataset, obtaining the results shown in Table 1 below.

[0210] Among them, the adversarial image The representation is obtained by performing adversarial attacks on clean images using the CLIP model. The representation is obtained by performing adversarial attacks on clean images using the LLM model. Represents the number of clean images and adversarial images, respectively. Precision, Recall, Accuracy, and F1-score are four metrics used to measure model performance. Larger values ​​indicate better model performance.

[0211]

[0212] Table 1

[0213] In Table 1, different images from the same dataset are used for training and testing. As can be seen from Table 1, the target classifier maintains relatively high values ​​in most indicators, indicating that the target classifier has good performance.

[0214] In another possible implementation, a target classifier is trained using some images from a certain dataset, and a performance test of the target classifier is performed using some images from another dataset, resulting in the following Table 2.

[0215]

[0216] Table 2

[0217] In Table 2, images from different datasets are used for training and testing. As can be seen from Table 2, the target classifier maintains relatively high values ​​in most indicators, indicating that the target classifier has good performance.

[0218] In another possible implementation, one adversarial attack method is used to obtain adversarial images, and these adversarial images are used to train the target classifier. Another adversarial attack method is used to obtain adversarial images, and these adversarial images are used to perform a performance test on the target classifier, resulting in the following Table 3.

[0219]

[0220] Table 3

[0221] In Table 3, the adversarial images used in training and testing are images generated based on different adversarial attack methods. As can be seen from Table 3, the target classifier maintains relatively high values ​​in most indicators, indicating that the target classifier performs well.

[0222] The embodiment of the present application also performs a performance test on the target classifier for the case of acquiring adversarial images based on different visual language models, and obtains the following Table 4. Among them, models 1 to 6 are 6 different visual language models.

[0223]

[0224] Table 4

[0225] As can be seen from Table 4, when different visual language models are used to obtain adversarial images, the target classifier mostly maintains relatively high values ​​in various indicators, indicating that the target classifier has better performance.

[0226] Optionally, the embodiment of the present application also performs a performance test on the target classifier for the case where adversarial attacks of different intensities are performed on clean images to obtain adversarial images, and obtains the following Table 5.

[0227]

[0228] Table 5

[0229] In Table 5, "255" refers to the grayscale pixel value range between 0 and 255, where 0 represents black and 255 represents white. "n / 255" means the difference between the values ​​of the same pixel in the adversarial image and the clean image is between ±n. For example, "4 / 255" means the value of a pixel in the clean image plus ±4 is the value of that pixel in the adversarial image. As can be seen from Table 5, when adversarial images are acquired at different attack intensities, the target classifier generally maintains relatively high values ​​across all metrics, indicating good performance.

[0230] Optionally, an embodiment of the present application sets three probe questions, and two target classifiers are trained using these three probe questions. Among them, the target image and the three probe questions are input into a target classifier. When the sub-results of at least two probe questions indicate that the target image is a clean image, the electronic device determines that the target image is a clean image. Otherwise, the electronic device determines that the target image is an adversarial image. The performance of this target classifier corresponds to the data in the row "2 / 3" in Table 6. The target image and the three probe questions are input into another target classifier. When the sub-results of all probe questions indicate that the target image is a clean image, the electronic device determines that the target image is a clean image. Otherwise, the electronic device determines that the target image is an adversarial image. The performance of this target classifier corresponds to the data in the row "3 / 3" in Table 6. Based on this, the two target classifiers are performance tested to obtain the following Table 6.

[0231]

[0232] Table 6

[0233] As can be seen from Table 6, for different data sets, different numbers of probe problems, and different ways of acquiring adversarial images, the target classifier mostly maintains high values ​​in various indicators, indicating that the target classifier has good performance.

[0234] In addition, the embodiment of the present application also performs a performance test on the target classifier for the black box attack scenario, and obtains the following Table 7. Among them, the black box attack is a technology that determines the adversarial image through the model without knowing the structure, parameters, training algorithm, etc. of the model.

[0235]

[0236] Table 7

[0237] As can be seen from Table 7, for the black-box attack scenarios of different models, the target classifier mostly maintains a high value in various indicators, indicating that the target classifier has good performance.

[0238] Figure 8 The figure shows a schematic diagram of the structure of a text generation device provided by an embodiment of the present application. Figure 8 As shown, the device includes:

[0239] An acquisition module 801 is used to acquire a target image;

[0240] A classification module 802 is configured to classify a target image and obtain a classification result of the target image. The classification result indicates whether the target image is an adversarial image. An adversarial image is an image perturbed by noise.

[0241] The acquisition module 801 is further configured to acquire a first target text related to the target image when it is determined based on the classification result that the target image is not a confrontation image;

[0242] The generating module 803 is configured to generate a second target text based on the target image and the first target text by using a visual language model, where the second target text is related to the target image and has a contextual relationship with the first target text.

[0243] In a possible implementation, the target image is associated with a third target text;

[0244] The classification module 802 is used to determine target attention information based on the target image, where the target attention information represents the influence of at least one image block in the target image on at least one character in the third target text; classify the target image according to the target attention information to obtain a classification result.

[0245] In one possible implementation, the classification module 802 is used to determine multiple visual tokens based on the target image, any one of the visual tokens represents the visual information of an image block in the target image; determine multiple first attention scores based on the multiple visual tokens, any one of the first attention scores represents the influence information of the image block corresponding to a visual token on the first character of at least one character; and determine the target attention information based on the multiple first attention scores.

[0246] In one possible implementation, the classification module 802 is used to determine multiple second attention scores based on multiple visual tokens and each character before the non-first character in at least one character, where any second attention score represents the impact information of an image block corresponding to a visual token on the non-first character; and determine target attention information based on multiple first attention scores and multiple second attention scores.

[0247] In a possible implementation, the classification module 802 is configured to obtain a fourth target text, which has a contextual relationship with the third target text; and determine target attention information based on the fourth target text and the target image.

[0248] In a possible implementation, the fourth target text is a question text with an answer of yes or no, and the third target text is the answer.

[0249] In a possible implementation, the target attention information is multiple;

[0250] The classification module 802 is used to classify the target image according to any target attention information to obtain a sub-result, and the sub-result represents whether the target image is an adversarial image; if the number of target sub-results is not less than the quantity threshold, it is determined that the classification result represents that the target image is not an adversarial image, and the target sub-result is a sub-result representing that the target image is not an adversarial image; if the number of target sub-results is less than the quantity threshold, it is determined that the classification result represents that the target image is an adversarial image.

[0251] In one possible implementation, the classification result is obtained by classifying the target image according to the target attention information by the target classifier;

[0252] The acquisition module 801 is further configured to acquire a first sample image, where the first sample image is not an adversarial image.

[0253] The device also includes:

[0254] a determining module, configured to determine first attention information based on the first sample image, where the first attention information represents information about an influence of at least one image block in the first sample image on at least one character in a first sample text, where the first sample text is related to the first sample image;

[0255] The classification module 802 is further configured to classify the first sample image according to the first attention information using an initial classifier to obtain a first prediction result, where the first prediction result indicates whether the first sample image is an adversarial image;

[0256] A training module, configured to train an initial classifier based on the first prediction result to obtain a first classifier;

[0257] The determination module is further configured to determine a target classifier based on the first classifier.

[0258] In one possible implementation, the classification result is obtained by classifying the target image according to the target attention information by the target classifier;

[0259] The acquisition module 801 is further configured to acquire a second sample image, where the second sample image is an adversarial image.

[0260] The device also includes:

[0261] a determination module, configured to determine second attention information based on the second sample image, where the second attention information represents information about an influence of at least one image block in the second sample image on at least one character in a second sample text, where the second sample text is related to the second sample image;

[0262] The classification module 802 is further configured to classify the second sample image according to the second attention information using the initial classifier to obtain a second prediction result, where the second prediction result indicates whether the second sample image is an adversarial image;

[0263] A training module, configured to train the initial classifier based on the second prediction result to obtain a second classifier;

[0264] The determination module is further configured to determine a target classifier based on the second classifier.

[0265] In one possible implementation, the acquisition module 801 is used to generate a fourth sample text based on a first sample image and a third sample text related to the first sample image through a visual language model, where the first sample image is not an adversarial image, and the fourth sample text is related to the first sample image and has a contextual relationship with the third sample text; based on the fourth sample text, noise perturbation is performed on the first sample image to obtain a second sample image.

[0266] In a possible implementation, the acquisition module 801 is configured to acquire a noise text that is semantically different from the fourth sample text; and perform noise perturbation on the first sample image based on the noise text and the fourth sample text to obtain a second sample image.

[0267] In one possible implementation, the generation module 803 is used to perform feature extraction on the target image through a visual language model to obtain target image features, where the target image features represent the visual information of the target image; obtain first text features through the visual language model, where the first text features represent the language information of the first target text; and generate a second target text based on the target image features and the first text features through the visual language model.

[0268] In one possible implementation, the generation module 803 is used to generate the first character in the second target text based on the target image features and the first text features through a visual language model; generate non-first characters based on the target image features, the first text features, and each character before the non-first character in the second target text through a visual language model; and determine the second target text based on the first character and the non-first characters.

[0269] In a possible implementation, the generation module 803 is further configured to generate prompt information when the target image is determined to be an adversarial image based on the classification result, where the prompt information is configured to indicate that the second target text cannot be generated through the visual language model.

[0270] The above-mentioned device classifies a target image and obtains a classification result indicating whether the target image is an adversarial image. Since adversarial images are noise-perturbed images, if the target image is determined not to be an adversarial image based on the classification result, it indicates that the target image has not been perturbed by noise. This allows the visual language model to generate a highly accurate second target text based on the target image and the first target text, thereby improving the security and reliability of the visual language model.

[0271] It should be understood that the above Figure 8 The provided device is illustrated only by the division of the above-mentioned functional modules when implementing its functions. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0272] Figure 9 FIG2 shows a block diagram of a terminal device 900 provided by an exemplary embodiment of the present application. The terminal device 900 includes a processor 901 and a memory 902 .

[0273] Processor 901 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 901 may be implemented in hardware using at least one of the following: a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), or a PLA (Programmable Logic Array). Processor 901 may also include a main processor and a coprocessor. The main processor is used to process data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 901 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing content displayed on the display screen. In some embodiments, processor 901 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0274] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one computer program, which is executed by the processor 901 to implement the text generation method provided in the method embodiment of the present application.

[0275] In some embodiments, the terminal device 900 may optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 may be connected via a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 903 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, and a power supply 908.

[0276] The peripheral device interface 903 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 901 and the memory 902. In some embodiments, the processor 901, the memory 902, and the peripheral device interface 903 are integrated on the same chip or circuit board. In other embodiments, any one or two of the processor 901, the memory 902, and the peripheral device interface 903 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0277] The RF circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 904 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF circuit 904 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The RF circuit 904 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 904 may also include circuitry related to Near Field Communication (NFC), although this application does not limit this.

[0278] The display screen 905 is used to display a user interface (UI). This UI may include graphics, text, icons, videos, or any combination thereof. If the display screen 905 is a touchscreen display, it is also capable of detecting touch signals on or above the surface of the display screen 905. These touch signals can be input as control signals to the processor 901 for processing. In this case, the display screen 905 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be a single display screen 905, located on the front panel of the terminal device 900. In other embodiments, there can be at least two display screens 905, located on different surfaces of the terminal device 900 or in a foldable design. In still other embodiments, the display screen 905 can be a flexible display, located on a curved or foldable surface of the terminal device 900. Furthermore, the display screen 905 can be configured as a non-rectangular, irregular shape, also known as a special-shaped screen. The display screen 905 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0279] The camera component 906 is used to capture images or videos. Optionally, the camera component 906 includes a front camera and a rear camera. Typically, the front camera is set on the front panel of the terminal, and the rear camera is set on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera component 906 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0280] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals to be input into the processor 901 for processing, or input into the radio frequency circuit 904 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there can be multiple microphones, which are respectively arranged in different parts of the terminal device 900. The microphone can also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker can be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signals into sound waves audible to humans, but also convert the electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0281] The power supply 908 is used to power the various components in the terminal device 900. The power supply 908 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When the power supply 908 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0282] In some embodiments, the terminal device 900 further includes one or more sensors 909 , including but not limited to: an acceleration sensor 911 , a gyroscope sensor 912 , a pressure sensor 913 , an optical sensor 914 , and a proximity sensor 915 .

[0283] The accelerometer 911 can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the terminal device 900. For example, the accelerometer 911 can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 901 can control the display screen 905 to display the user interface in a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 911. The accelerometer 911 can also be used to collect game or user motion data.

[0284] The gyroscope sensor 912 can detect the orientation and rotation angle of the terminal device 900. It can work in conjunction with the accelerometer 911 to collect the user's 3D movements of the terminal device 900. Based on the data collected by the gyroscope sensor 912, the processor 901 can implement the following functions: motion sensing (for example, changing the UI based on the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0285] The pressure sensor 913 can be set on the side frame of the terminal device 900 and / or the lower layer of the display screen 905. When the pressure sensor 913 is set on the side frame of the terminal device 900, it can detect the user's grip signal of the terminal device 900, and the processor 901 performs left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor 913. When the pressure sensor 913 is set on the lower layer of the display screen 905, the processor 901 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 905. The operable controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0286] Optical sensor 914 is used to detect ambient light intensity. In one embodiment, processor 901 can control the display brightness of display screen 905 based on the ambient light intensity detected by optical sensor 914. Specifically, when the ambient light intensity is high, the display brightness of display screen 905 is increased; when the ambient light intensity is low, the display brightness of display screen 905 is decreased. In another embodiment, processor 901 can also dynamically adjust the shooting parameters of camera assembly 906 based on the ambient light intensity detected by optical sensor 914.

[0287] The proximity sensor 915, also known as a distance sensor, is typically located on the front panel of the terminal device 900. The proximity sensor 915 is used to detect the distance between the user and the front of the terminal device 900. In one embodiment, when the proximity sensor 915 detects that the distance between the user and the front of the terminal device 900 is gradually decreasing, the processor 901 controls the display screen 905 to switch from the screen-on state to the screen-off state. When the proximity sensor 915 detects that the distance between the user and the front of the terminal device 900 is gradually increasing, the processor 901 controls the display screen 905 to switch from the screen-off state to the screen-on state.

[0288] Those skilled in the art will understand that Figure 9 The structure shown in the figure does not constitute a limitation on the terminal device 900, and the terminal device 900 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0289] Figure 10This is a schematic diagram of the structure of the server provided in the embodiment of the present application. The server 1000 may have relatively large differences due to different configurations or performances, and may include one or more processors 1001 and one or more memories 1002, wherein the one or more memories 1002 store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 1001 to implement the text generation method provided in the above-mentioned various method embodiments. Exemplarily, the processor 1001 is a CPU. Of course, the server 1000 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 1000 may also include other components for implementing device functions, which will not be described in detail here.

[0290] In an exemplary embodiment, a computer-readable storage medium is further provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned text generation methods.

[0291] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0292] In an exemplary embodiment, a computer program is further provided. The computer program is at least one, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned text generation methods.

[0293] In an exemplary embodiment, a computer program product is further provided. The computer program product stores at least one computer program, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned text generation methods.

[0294] It should be understood that the term "plurality" used herein refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0295] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0296] The above description is merely an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A text generation method, characterized in that: The method comprises: Acquire a target image and a fourth target text; determining target attention information based on the fourth target text and the target image, wherein the target image is related to the third target text, the fourth target text and the third target text have a contextual relationship, and the target attention information represents information about the influence of at least one image block in the target image on at least one character in the third target text; Classifying the target image according to the target attention information to obtain a classification result, wherein the classification result indicates whether the target image is an adversarial image, wherein the adversarial image is an image perturbed by noise and is used to cause a visual language model to output an incorrect following text based on the preceding text; When it is determined based on the classification result that the target image is not the adversarial image, obtaining a first target text related to the target image; A second target text is generated based on the target image and the first target text by using the visual language model, where the second target text is related to the target image and has a contextual relationship with the first target text.

2. The method according to claim 1, characterized in that The determining target attention information based on the fourth target text and the target image includes: determining a plurality of visual tokens based on the target image, wherein any visual token represents visual information of an image block in the target image; Determining a plurality of first attention scores based on the fourth target text and the plurality of visual tokens, wherein each first attention score represents information about an influence of an image block corresponding to a visual token on a first character of at least one character in the third target text; Based on the multiple first attention scores, the target attention information is determined.

3. The method according to claim 2, characterized in that The determining the target attention information based on the multiple first attention scores includes: Determining a plurality of second attention scores based on the plurality of visual tokens and respective characters preceding a non-first character of at least one character in the third target text, wherein each second attention score represents information about an influence of an image block corresponding to a visual token on the non-first character; The target attention information is determined based on the multiple first attention scores and the multiple second attention scores.

4. The method according to claim 1, wherein The fourth target text is a question text whose answer is yes or no, and the third target text is the answer.

5. The method according to claim 1, wherein The target attention information is multiple; and the target image is classified according to the target attention information to obtain a classification result, including: For any target attention information, classify the target image according to the any target attention information to obtain a sub-result, wherein the sub-result represents whether the target image is the adversarial image; If the number of target sub-results is not less than the number threshold, determining that the classification result indicates that the target image is not the adversarial image, and the target sub-result is a sub-result indicating that the target image is not the adversarial image; If the number of the target sub-results is less than the number threshold, it is determined that the classification result represents that the target image is the adversarial image.

6. The method according to claim 1, characterized in that The classification result is obtained by classifying the target image according to the target attention information by a target classifier; the method further includes: Acquire a first sample image, where the first sample image is not the adversarial image; determining first attention information based on the first sample image, where the first attention information represents information about the influence of at least one image block in the first sample image on at least one character in a first sample text, where the first sample text is related to the first sample image; classifying the first sample image according to the first attention information using an initial classifier to obtain a first prediction result, where the first prediction result indicates whether the first sample image is the adversarial image; Training the initial classifier based on the first prediction result to obtain a first classifier; The target classifier is determined based on the first classifier.

7. The method according to claim 1, characterized in that The classification result is obtained by classifying the target image according to the target attention information by a target classifier; the method further includes: Acquire a second sample image, where the second sample image is the adversarial image; determining second attention information based on the second sample image, where the second attention information represents information about the influence of at least one image block in the second sample image on at least one character in a second sample text, where the second sample text is related to the second sample image; classifying the second sample image according to the second attention information using an initial classifier to obtain a second prediction result, where the second prediction result indicates whether the second sample image is the adversarial image; Training the initial classifier based on the second prediction result to obtain a second classifier; The target classifier is determined based on the second classifier.

8. The method according to claim 7, characterized in that The acquiring of the second sample image includes: generating, by the visual language model, a fourth sample text based on a first sample image and a third sample text related to the first sample image, wherein the first sample image is not the adversarial image, and the fourth sample text is related to the first sample image and has a contextual relationship with the third sample text; Based on the fourth sample text, noise perturbation is performed on the first sample image to obtain the second sample image.

9. The method according to claim 8, characterized in that The performing noise perturbation on the first sample image based on the fourth sample text to obtain the second sample image includes: Acquire a noise text that is semantically different from the fourth sample text; Based on the noise text and the fourth sample text, noise perturbation is performed on the first sample image to obtain the second sample image.

10. The method according to any one of claims 1 to 9, characterized in that Generating a second target text based on the target image and the first target text by using the visual language model includes: Performing feature extraction on the target image using a visual language model to obtain target image features, wherein the target image features represent visual information of the target image; Acquire a first text feature through the visual language model, where the first text feature represents language information of the first target text; A second target text is generated by the visual language model based on the target image features and the first text features.

11. The method according to claim 10, characterized in that Generating a second target text based on the target image feature and the first text feature by using the visual language model includes: generating a first character in the second target text based on the target image feature and the first text feature by the visual language model; generating the non-first character by the visual language model based on the target image feature, the first text feature, and each character preceding the non-first character in the second target text; The second target text is determined based on the first character and the non-first characters.

12. The method according to any one of claims 1 to 9, characterized in that The method further comprises: When it is determined based on the classification result that the target image is the adversarial image, prompt information is generated, where the prompt information is used to prompt that the second target text cannot be generated through the visual language model.

13. A text generation device, characterized in that: The device comprises: An acquisition module, used for acquiring a target image and a fourth target text; a classification module for determining target attention information based on the fourth target text and the target image, wherein the target image is related to the third target text, the fourth target text and the third target text are in a contextual relationship, and the target attention information represents information about the influence of at least one image block in the target image on at least one character in the third target text; classifying the target image according to the target attention information to obtain a classification result, wherein the classification result represents whether the target image is an adversarial image, wherein the adversarial image is an image perturbed by noise and is used to cause the visual language model to output an incorrect following text based on the preceding text; The acquisition module is further configured to acquire a first target text related to the target image when it is determined based on the classification result that the target image is not the adversarial image; A generating module is configured to generate a second target text based on the target image and the first target text by using the visual language model, wherein the second target text is related to the target image and has a contextual relationship with the first target text.

14. The device according to claim 13, characterized in that The classification module is configured to determine a plurality of visual tokens based on the target image, wherein any visual token represents visual information of an image block in the target image; Determining a plurality of first attention scores based on the fourth target text and the plurality of visual tokens, wherein each first attention score represents information about an influence of an image block corresponding to a visual token on a first character of at least one character in the third target text; Based on the multiple first attention scores, the target attention information is determined.

15. The device according to claim 14, characterized in that The classification module is configured to determine a plurality of second attention scores based on the plurality of visual tokens and respective characters preceding a non-first character of at least one character in the third target text, wherein each second attention score represents information about an influence of an image block corresponding to a visual token on the non-first character; The target attention information is determined based on the multiple first attention scores and the multiple second attention scores.

16. The device according to claim 13, characterized in that The fourth target text is a question text whose answer is yes or no, and the third target text is the answer.

17. The device according to claim 13, characterized in that The target attention information is multiple; The classification module is configured to classify the target image according to any target attention information to obtain a sub-result, wherein the sub-result indicates whether the target image is the adversarial image; if the number of target sub-results is not less than a quantity threshold, determine that the classification result indicates that the target image is not the adversarial image, and the target sub-result is a sub-result indicating that the target image is not the adversarial image; If the number of the target sub-results is less than the number threshold, it is determined that the classification result represents that the target image is the adversarial image.

18. The device according to claim 13, characterized in that The classification result is obtained by classifying the target image according to the target attention information by a target classifier; The acquisition module is further configured to acquire a first sample image, where the first sample image is not the adversarial image; The device further comprises: a determining module, configured to determine first attention information based on the first sample image, where the first attention information represents information about an influence of at least one image block in the first sample image on at least one character in a first sample text, where the first sample text is related to the first sample image; The classification module is further configured to classify the first sample image according to the first attention information using an initial classifier to obtain a first prediction result, where the first prediction result indicates whether the first sample image is the adversarial image; A training module, configured to train the initial classifier based on the first prediction result to obtain a first classifier; The determination module is further configured to determine the target classifier based on the first classifier.

19. The device according to claim 13, characterized in that The classification result is obtained by classifying the target image according to the target attention information by a target classifier; The acquisition module is further configured to acquire a second sample image, where the second sample image is the adversarial image; The device further comprises: a determining module, configured to determine second attention information based on the second sample image, where the second attention information represents information about an influence of at least one image block in the second sample image on at least one character in a second sample text, where the second sample text is related to the second sample image; The classification module is further configured to classify the second sample image according to the second attention information using an initial classifier to obtain a second prediction result, where the second prediction result indicates whether the second sample image is the adversarial image; A training module, configured to train the initial classifier based on the second prediction result to obtain a second classifier; The determination module is further configured to determine the target classifier based on the second classifier.

20. The device according to claim 19, characterized in that The acquisition module is configured to generate a fourth sample text based on a first sample image and a third sample text related to the first sample image using the visual language model, wherein the first sample image is not the adversarial image, and the fourth sample text is related to the first sample image and has a contextual relationship with the third sample text; Based on the fourth sample text, noise perturbation is performed on the first sample image to obtain the second sample image.

21. The device according to claim 20, characterized in that The acquisition module is configured to acquire a noise text that is semantically different from the fourth sample text; and perform noise perturbation on the first sample image based on the noise text and the fourth sample text to obtain the second sample image.

22. The device according to any one of claims 13 to 21, characterized in that The generation module is configured to perform feature extraction on the target image through a visual language model to obtain target image features, where the target image features represent visual information of the target image; obtain first text features through the visual language model, where the first text features represent language information of the first target text; and generate a second target text based on the target image features and the first text features through the visual language model.

23. The device according to claim 22, characterized in that The generation module is used to generate the first character in the second target text based on the target image feature and the first text feature through the visual language model; generate the non-first character based on the target image feature, the first text feature and each character before the non-first character in the second target text through the visual language model; and determine the second target text based on the first character and the non-first character.

24. The device according to any one of claims 13 to 21, characterized in that The generation module is further configured to generate prompt information when it is determined based on the classification result that the target image is the adversarial image, wherein the prompt information is configured to indicate that the second target text cannot be generated through the visual language model.

25. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the text generation method according to any one of claims 1 to 12.

26. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor to enable the electronic device to implement the text generation method according to any one of claims 1 to 12.

27. A computer program product, characterized in that The computer program product stores at least one computer program, and the at least one computer program is loaded and executed by a processor to enable the electronic device to implement the text generation method according to any one of claims 1 to 12.

Images