Image processing method and device, electronic equipment and readable storage medium

Abstract

The invention relates to the technical field of image processing, and provides an image processing method and device, electronic equipment and a readable storage medium. The method comprises the following steps: determining at least one pixel block embedding vector according to image data to be processed; performing convolution processing on the to-be-processed image data to obtain a feature vector; processing the feature vector and the embedded vector of each pixel block through an attention mechanism to obtain a cross attention vector; performing parameterization processing on the cross attention vector and a preset query vector to obtain an interactive extraction vector; processing the interactive extraction vector and the embedded vector of each pixel block based on visual transformation to obtain a visual transformation vector; performing information extraction processing on the visual transformation vector and the interactive extraction vector to obtain a re-extracted orientation vector, and performing nonlinear transformation processing on the re-extracted orientation vector to obtain a target feature vector; and determining whether the to-be-processed image data contains the to-be-processed image data based on the target feature vector, thereby improving the semantic richness of the target feature vector.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and readable storage medium. Background Art

[0002] Dense prediction tasks are now increasingly widely used in various application scenarios such as object detection or semantic segmentation. In the dense prediction tasks of the existing technology, the conventional method is to perform multimodal pre-training to learn semantic information through vision transformer (ViT), obtain the feature representation of the image through visual transformation processing, and then use the feature representation of the image to perform downstream dense prediction tasks. However, due to the lack of vision-specific inductive bias and specific prior knowledge of image data, ViT's training process converges slowly, and the feature representation generated by the processing has low semantic richness and low information dimension.

[0003] It can be seen that the existing technology has problems such as incomplete semantics of feature representation generated by processing due to lack of prior knowledge, inability to highlight and capture global and local features, and low information dimension. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure provide an image processing method, device, electronic device and readable storage medium to solve the problems in the prior art that the feature representation generated by processing is incomplete in semantics due to lack of prior knowledge, global and local features cannot be captured prominently, and the information dimension is low.

[0005] According to a first aspect of an embodiment of the present disclosure, an image processing method is provided, including: determining at least one pixel block embedding vector corresponding to the image data to be processed according to image data to be processed; performing convolution processing on the image data to be processed to obtain a feature vector corresponding to the image data to be processed; processing the feature vector corresponding to the image data to be processed and each pixel block embedding vector through an attention mechanism to obtain a cross-attention vector corresponding to the image data to be processed; performing parameterized processing on the cross-attention vector corresponding to the image data to be processed and a preset query vector to obtain an interaction extraction vector corresponding to the image data to be processed; processing the interaction extraction vector corresponding to the image data to be processed and each pixel block embedding vector based on visual transformation to obtain a visual transformation vector corresponding to the image data to be processed; performing information extraction processing on the visual transformation vector corresponding to the image data to be processed and the interaction extraction vector corresponding to the image data to be processed to obtain a re-extraction vector corresponding to the image data to be processed; performing nonlinear transformation processing on the re-extraction vector corresponding to the image data to be processed to obtain a target feature vector corresponding to the image data to be processed; and determining whether the image data to be processed contains the image data to be processed based on the target feature vector corresponding to the image data to be processed.

[0006] According to a second aspect of the embodiments of the present disclosure, an image processing device is provided, comprising: a first determining module, configured to determine, based on the image data to be processed, at least one pixel block embedding vector corresponding to the image data to be processed; a first processing module, configured to perform convolution processing on the image data to be processed to obtain a feature vector corresponding to the image data to be processed; a second processing module, configured to process the feature vector corresponding to the image data to be processed and each pixel block embedding vector through an attention mechanism to obtain a cross-attention vector corresponding to the image data to be processed; and a third processing module, configured to perform parameterized processing on the cross-attention vector corresponding to the image data to be processed and a preset query vector to obtain an interactive extraction vector corresponding to the image data to be processed. The fourth processing module is used to process the interactive extraction vector and each pixel block embedding vector corresponding to the image data to be processed based on the visual transformation to obtain the visual transformation vector corresponding to the image data to be processed; the fifth processing module is used to perform information extraction processing on the visual transformation vector corresponding to the image data to be processed and the interactive extraction vector corresponding to the image data to be processed to obtain the re-extraction vector corresponding to the image data to be processed; the sixth processing module is used to perform nonlinear transformation processing on the re-extraction vector corresponding to the image data to be processed to obtain the target feature vector corresponding to the image data to be processed; the second determination module is used to determine whether the image data to be processed is contained in the image data to be processed based on the target feature vector corresponding to the image data to be processed.

[0007] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0008] According to a fourth aspect of an embodiment of the present disclosure, a readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0009] Compared with the prior art, the disclosed embodiments have the following beneficial effects: by dividing the image data to be processed into pixel blocks, pixel block data corresponding to at least one image data to be processed is obtained, and each pixel block data is embedded to obtain an embedding vector corresponding to each pixel block data; the image data to be processed is processed by a convolution layer to obtain a feature vector corresponding to the image data to be processed; then, the feature vector corresponding to the image data to be processed and each pixel block embedding vector can be processed by an attention mechanism to obtain a cross-attention vector corresponding to the image data to be processed; the cross-attention vector corresponding to the image data to be processed and a preset query vector are processed by parameterization to obtain an interactive extraction vector corresponding to the image data to be processed; the interactive extraction vector and the query vector corresponding to the image data to be processed can be processed by a visual transformer Each pixel block embedding vector is processed to obtain a visual transformation vector corresponding to the image data to be processed, and then the visual transformation vector corresponding to the image data to be processed and the interaction extraction vector corresponding to the image data to be processed can be processed again to obtain a re-extraction vector corresponding to the image data to be processed. The re-extraction vector corresponding to the image data to be processed can be processed by a feedforward network to perform nonlinear transformation to obtain a target feature vector corresponding to the image data to be processed. Based on the target feature vector corresponding to the image data to be processed, it can be determined whether the image data to be processed in the intensive prediction task contains the image data to be processed, thereby improving the accuracy of image recognition, enhancing the generalization ability of the image processing model, improving the semantic richness of the target feature vector, improving the information dimension of the target feature vector, and improving the refinement of feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 is a schematic diagram of an application scenario of an embodiment of the present disclosure;

[0012] Figure 2 is a flowchart of an image processing method provided by an embodiment of the present disclosure;

[0013] Figure 3 is a processing flow diagram of an image processing model provided by an embodiment of the present disclosure;

[0014] Figure 4 is a structural schematic diagram of an image processing device provided by an embodiment of the present disclosure;

[0015] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0017] It should be noted that the user information (including but not limited to terminal device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0018] An image processing method and device according to an embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0019] Figure 1 Schematic diagram of an application scenario of an embodiment of the present disclosure. The application scenario may include terminal devices 1, 2 and 3, a server 4 and a network 5.

[0020] Terminal devices 1, 2 and 3 can be hardware or software. When terminal devices 1, 2 and 3 are hardware, they can be various electronic devices with display screens and supporting communication with server 4, including but not limited to smart phones, tablet computers, laptop portable computers and desktop computers, etc.; when terminal devices 1, 2 and 3 are software, they can be installed in the above electronic devices. Terminal devices 1, 2 and 3 can be implemented as multiple software or software modules, or as a single software or software module, and the embodiments of the present disclosure are not limited to this. Furthermore, various applications can be installed on terminal devices 1, 2 and 3, such as data processing applications, instant messaging tools, social platform software, search applications, shopping applications, etc.

[0021] The server 4 may be a server that provides various services, for example, a background server that receives a request sent by a terminal device that establishes a communication connection with the server, and the background server may receive and analyze the request sent by the terminal device, and generate a processing result. The server 4 may be a single server, or a server cluster composed of several servers, or a cloud computing service center, which is not limited in the embodiments of the present disclosure.

[0022] It should be noted that the server 4 can be hardware or software. When the server 4 is hardware, it can be various electronic devices that provide various services for the terminal devices 1, 2, and 3. When the server 4 is software, it can be multiple software or software modules that provide various services for the terminal devices 1, 2, and 3, or it can be a single software or software module that provides various services for the terminal devices 1, 2, and 3, and the embodiments of the present disclosure are not limited to this.

[0023] The network 5 can be a wired network connected by coaxial cable, twisted pair and optical fiber, or it can be a wireless network that can interconnect various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), infrared, etc., which is not limited in the embodiments of the present disclosure.

[0024] The user can establish a communication connection with the server 4 via the network 5 through the terminal devices 1, 2 and 3 to receive or send information, etc. Specifically, the server 4 can divide the image data to be processed into pixel blocks to obtain pixel block data corresponding to at least one image data to be processed, and embed each pixel block data to obtain an embedding vector corresponding to each pixel block data, process the image data to be processed through a convolution layer to obtain a feature vector corresponding to the image data to be processed, and then perform attention mechanism processing on the feature vector corresponding to the image data to be processed and the embedding vectors of each pixel block to obtain a cross-attention vector corresponding to the image data to be processed, process the cross-attention vector corresponding to the image data to be processed and a preset query vector through parameterized processing to obtain an interactive extraction vector corresponding to the image data to be processed, and process the interactive extraction vector corresponding to the image data to be processed and each pixel block embedding vector through a visual transformer to obtain a visual transformation vector corresponding to the image data to be processed, and then perform information extraction processing on the visual transformation vector corresponding to the image data to be processed and the interactive extraction vector corresponding to the image data to be processed again to obtain a re-extraction vector corresponding to the image data to be processed, and perform nonlinear transformation processing on the re-extraction vector corresponding to the image data to be processed through a feedforward network to obtain a target feature vector corresponding to the image data to be processed, and based on the target feature vector corresponding to the image data to be processed, it can be determined whether the image data to be processed in the dense prediction task contains the image data to be processed.

[0025] It should be noted that the specific types, quantities and combinations of the terminal devices 1, 2 and 3, the server 4 and the network 5 can be adjusted according to the actual needs of the application scenario, and the embodiments of the present disclosure are not limited to this.

[0026] Figure 2 It is a flowchart of an image processing method provided by an embodiment of the present disclosure. Figure 2 The image processing method can be Figure 1 The server executes. Figure 2 As shown, the image processing method includes:

[0027] Step 201: Determine at least one pixel block embedding vector corresponding to the image data to be processed according to the image data to be processed.

[0028] Specifically, the image data to be processed can be analyzed by the image processing model deployed in the server, and the image data to be processed can be divided into at least one pixel block, and then at least one pixel block embedding vector can be obtained for each pixel block through embedding processing, deep learning model, hash function, or feature extraction processing, which is not limited here. This enhances the representation capability of the image data to be processed, improves the efficiency and accuracy of image processing, and improves the performance of subsequent image analysis, recognition or compression tasks.

[0029] Among them, the image data to be processed can be the original image data that needs to be processed. The image data to be processed can be a digital photo, a video frame, or a scanned image, etc., which is not limited here. The image data to be processed can be stored in the form of a pixel matrix, including visual information such as image brightness and color, which is not limited here.

[0030] The pixel block embedding vector corresponding to the image data to be processed can be a feature vector obtained by dividing the image data to be processed into at least one pixel block and extracting each pixel block through vectorization processing. The pixel block embedding vector corresponding to the image data to be processed can represent the key information of the pixel block in numerical form, and the key information includes but is not limited to texture, color distribution, and / or edge features, etc., and the pixel block embedding vector can be used for various image processing tasks such as image classification, retrieval, recognition, and compression, which are not limited here. The pixel block embedding vector can be obtained by a deep learning network, a hash function, or a scale-invariant feature transformation, etc., which are not limited here, and can be used to characterize the characteristics of a local area of ​​the image data to be processed.

[0031] For example, in the field of image retrieval technology, when it is necessary to perform image retrieval in a database, the image processing model can divide each image in the database and the query image into pixel blocks and extract pixel block embedding vectors, and then measure the similarity between images by the distance between pixel block embedding vectors, so that images similar to the query image can be retrieved. In this embodiment, the distance between pixel block embedding vectors can be calculated by methods such as Euclidean distance, cosine similarity, etc.

[0032] Step 202: Perform convolution processing on the image data to be processed to obtain a feature vector corresponding to the image data to be processed.

[0033] Specifically, the convolutional neural network (CNN) model pre-configured in the image processing model can be used to perform multi-level convolution and pooling processing on the image data to be processed. Multiple convolution kernels can be slid on the image to be processed to extract local features in the image to be processed. The local features include but are not limited to edges, textures, and / or shapes, etc., and the pooling layer can be used to reduce the dimension and computational complexity of the image data to be processed. The obtained local features can then be integrated and mapped into a high-dimensional feature vector space to obtain a feature vector corresponding to the image data to be processed, thereby enhancing the ability to extract image features, so that the feature vector can more comprehensively characterize the content and structure of the image to be processed; improving the robustness of the image processing model, and improving the performance of subsequent image classification, recognition or retrieval tasks.

[0034] The feature vector corresponding to the image data to be processed may be a high-dimensional feature representation extracted from the image data to be processed by processing the image data layer by layer through a convolutional neural network. The feature vector corresponding to the image data to be processed may contain multiple elements, each of which may correspond to a feature or attribute in the image to be processed. The feature vector may be obtained by processing structures such as a convolutional layer, a pooling layer, and a fully connected layer in a convolutional neural network, which is not limited here, and the global and local features in the image to be processed may be captured by a convolutional neural network.

[0035] For example, in a face recognition system, the object processing model can use the face image as the image data to be processed, perform convolution processing on the face image through a convolutional neural network, and obtain a feature vector corresponding to the face image.

[0036] Step 203, the feature vector corresponding to the image data to be processed and the embedding vectors of each pixel block are processed by the attention mechanism to obtain the cross attention vector corresponding to the image data to be processed.

[0037] Specifically, the feature vector corresponding to the image data to be processed can be subjected to self-attention processing to obtain the self-attention vector corresponding to the image data to be processed, and then each pixel block embedding vector and the self-attention vector corresponding to the image data to be processed can be subjected to information exchange through cross-attention to obtain the cross-attention vector corresponding to the image data to be processed, thereby enhancing the global and local correlation of image features, so that the cross-attention vector can capture the overall structure and detail features of the image, improve the refinement of the image processing model, improve the semantic richness of the target feature vector, and enhance the information dimension of the target feature vector.

[0038] Among them, the attention mechanism is a computational method that simulates the human visual attention process. It can select and focus on the most important part of the current task in the process of processing a large amount of information, thereby improving the efficiency and accuracy of information processing. In image processing, the attention mechanism can be used to enhance feature extraction, target detection, and image generation tasks.

[0039] The cross-attention vector corresponding to the image data to be processed can be a fusion representation obtained by processing the feature vector and the pixel block embedding vector through the attention mechanism. The cross-attention vector can contain the global feature information of the image and incorporate the key details of each pixel block. The cross-attention vector can be used in a variety of image processing tasks such as image classification, object detection, image segmentation, and image description generation, which are not limited here.

[0040] For example, in an image description generation system, the image processing model can use the base image as the image data to be processed, and can perform self-attention processing on the feature vector corresponding to the base image to obtain the self-attention vector corresponding to the base image, and then perform cross-attention processing on the pixel block embedding vectors corresponding to each base image and the self-attention vector corresponding to the base image to obtain the cross-attention vector corresponding to the base image.

[0041] Step 204 , parameterize the cross-attention vector corresponding to the image data to be processed and the preset query vector to obtain an interactive extraction vector corresponding to the image data to be processed.

[0042] Specifically, the cross-attention vector corresponding to the image data to be processed can be processed by a deep learning module (BI-Mamba) for processing long sequence data to obtain an extraction vector corresponding to the image data to be processed, and then the extraction vector corresponding to the image data to be processed and the preset query vector can be cross-attention processed to obtain an interactive extraction vector corresponding to the image data to be processed, thereby enhancing the interaction ability between image features and query vectors, improving the semantic richness of the target feature vector, improving the information dimension of the target feature vector, and improving the flexibility of the image processing model.

[0043] The preset query vector may be a pre-set query vector, which may be used to characterize the target feature of the query or attention. The preset query vector may be used to guide the image processing process so that the processing process focuses on the image features related to the query vector.

[0044] The interactive extraction vector corresponding to the image data to be processed can be a vector representation obtained by performing cross-attention processing on the obtained extraction vector and the query vector after the cross-attention vector corresponding to the image data to be processed is processed by the BI-Mamba module. The interactive extraction vector can contain global and local feature information of the image data to be processed, and incorporates information on specific features or regions that the query vector focuses on. The interactive extraction vector can be used in image processing tasks such as image retrieval, target recognition, or image analysis.

[0045] For example, in an image retrieval system, we can take the query keywords or images input by the user as input, and then process the cross-attention vector corresponding to the image data to be processed through the BI-Mamba module to obtain the extraction vector corresponding to the image data to be processed, and then perform cross-attention processing on the extraction vector corresponding to the image data to be processed and the preset query vector to obtain the interactive extraction vector corresponding to the image data to be processed.

[0046] Step 205 , based on the visual transformation, the interaction extraction vector corresponding to the image data to be processed and the embedding vectors of each pixel block are processed to obtain the visual transformation vector corresponding to the image data to be processed.

[0047] Specifically, the interaction extraction vectors and each pixel block embedding vector corresponding to the image data to be processed can be serialized through the Vision Transformer (ViT) module, and the extraction vectors and each pixel block embedding vector corresponding to the image data to be processed can be input into the encoder of the encoder-decoder model (Transformer) architecture for self-attention mechanism processing. Vision Transformer can capture the relationship between pixel block embedding vectors and interaction extraction vectors through the Transformer architecture by treating each pixel block embedding vector and interaction extraction vector as serialized input. Then, each vector can be transformed and updated through linear projection to obtain the visual transformation vector corresponding to the image data to be processed, thereby enhancing the global correlation and context understanding ability of image features, and improving the dynamics and adaptability of the image processing model.

[0048] The visual transformation may be a process of processing the interactive extraction vector and each pixel block embedding vector corresponding to the image data to be processed through the Vision Transformer module. The Vision Transformer may be a visual model based on the Transformer architecture, and the Vision Transformer module may realize the effective extraction and transformation of image features.

[0049] The visual transformation vector corresponding to the image data to be processed may be an output vector obtained by processing the interaction extraction vector and the pixel block embedding vector through the Vision Transformer module. The visual transformation vector may contain global and local feature information of the image data to be processed, and incorporates the correlation information between different regions in the image data to be processed. The visual transformation vector may be obtained by concatenating and linearly projecting the input vector through the encoder of the Vision Transformer module. The visual transformation vector may be used in image processing tasks such as image classification, target detection, image segmentation, or image generation, and is not limited here.

[0050] For example, in an image classification task, after obtaining the interaction extraction vector and each pixel block embedding vector corresponding to the image data to be processed, the image processing model can process the interaction extraction vector and each pixel block embedding vector corresponding to the image data to be processed through the Vision Transformer module to obtain the visual transformation vector corresponding to the image data to be processed.

[0051] Step 206 , performing information extraction processing on the visual transformation vector corresponding to the image data to be processed and the interactive extraction vector corresponding to the image data to be processed, to obtain a re-extraction vector corresponding to the image data to be processed.

[0052] Specifically, the visual transformation vector and the interaction extraction vector corresponding to the image data to be processed can be cross-attention processed to obtain the visual transformation interaction vector corresponding to the image data to be processed, and then the visual transformation interaction vector corresponding to the image data to be processed can be processed by the BI-Mamba module to obtain the re-extraction vector corresponding to the image data to be processed, thereby enhancing the integration capability and robustness of image feature information. The re-extraction vector integrates the complementary information of the visual transformation vector and the interaction extraction vector, and can more comprehensively characterize the content of the image data to be processed; improve the accuracy and efficiency of the image processing model, improve the adaptability and scalability of the image processing model, improve the semantic richness of the target feature vector, and improve the information dimension of the target feature vector.

[0053] The re-extracted vector corresponding to the image data to be processed may be a vector representation obtained by fusing the visual transformation vector and the interactive extraction vector through information extraction processing and information extraction processing. The re-extracted vector may contain global and local feature information of the image data to be processed. The re-extracted vector may be obtained by processing the visual transformation vector and the interactive extraction vector through cross-attention and processing the processing result through the BI-Mamba module. The re-extracted vector may be used for image processing tasks such as image classification, target detection, image segmentation, and image recognition, which are not limited here.

[0054] For example, in medical image analysis tasks, such as tumor detection or organ segmentation, the image data to be processed may be a medical image. The image processing model may process the medical image to obtain a visual transformation vector and an interactive extraction vector, and then perform cross-attention processing on the visual transformation vector corresponding to the medical image and the interactive extraction vector corresponding to the medical image to obtain a visual transformation interactive vector corresponding to the medical image, and then process the visual transformation interactive vector corresponding to the medical image through the BI-Mamba module to obtain a re-extraction vector corresponding to the medical image.

[0055] Step 207 , performing nonlinear transformation processing on the re-extracted vector corresponding to the image data to be processed, to obtain a target feature vector corresponding to the image data to be processed.

[0056] Specifically, the re-extracted vector corresponding to the image data to be processed can be input into a feedforward network, which can be composed of a multi-layer neural network, an activation function, a normalization layer, and / or a discard layer, etc. The activation function includes but is not limited to ReLU, Sigmoid, or Tanh, etc., which are not limited here. The re-extracted vector can be nonlinearly transformed to extract a deeper feature representation, that is, the target feature vector corresponding to the image data to be processed, thereby enhancing the nonlinear expression ability and abstraction ability of the image feature. The target feature vector is extracted from the re-extracted vector through nonlinear transformation, which improves the accuracy and robustness of the image processing model, reduces the influence of noise and redundant information, and improves the generalization ability of the image processing model for complex image scenes.

[0057] The target feature vector corresponding to the image data to be processed may be a deep feature representation obtained by processing the re-extracted vector corresponding to the image data to be processed through a feedforward network. The target feature vector may contain global and local feature information of the image data to be processed, and incorporates the correlation information, complementary features, and deep abstract features between different regions in the image data to be processed, which are not limited here. The target feature vector may be used for image processing tasks such as image classification, target detection, image segmentation, image recognition, or image retrieval, which are not limited here.

[0058] For example, in an autonomous driving system, the image processing model can process the image data captured by the vehicle camera to obtain a re-extracted vector, and then perform nonlinear transformation on the re-extracted vector through a feedforward network to obtain a target feature vector.

[0059] Step 208 : determining whether the image data to be processed includes the image data to be processed based on the target feature vector corresponding to the image data to be processed.

[0060] Specifically, the target feature vector corresponding to the image data to be processed can be compared with the target feature vector corresponding to the image data to be processed or the similarity can be calculated. The comparison can be performed by calculating the distance metric of the feature vector, deep metric learning, or twin network, etc., which is not limited here. By comparing the similarity of the two feature vectors, it can be determined whether the image data to be processed contains a part or a whole similar to the image data to be processed, that is, whether the two belong to the same category or whether they contain the same object or scene, etc., which is not limited here. The method of calculating the distance degree of the feature vector includes but is not limited to Euclidean distance or cosine similarity, etc., thereby enhancing the accuracy and robustness of image recognition, improving the precision of recognition, improving the efficiency and flexibility of image matching, improving the generalization ability of the image processing model, and improving the refinement of feature extraction.

[0061] The image data to be processed may be image data that needs to be identified or matched. The image data to be processed may be obtained by real-time capture, retrieval from a database, or upload by a user. The image data to be processed may be applied to a variety of technical fields, including but not limited to security monitoring, intelligent retrieval, human-computer interaction, and medical image analysis.

[0062] For example, in an intelligent security monitoring system, the real-time image data captured by the surveillance camera needs to be processed to identify potential threats or abnormal events. The image processing model can use known image data that needs to be monitored, such as photos of specific people, images of prohibited items, etc., as image data to be processed, and then can use the real-time image data captured by the surveillance camera as image data to be processed. Two feature vectors can be compared. If the similarity is greater than or equal to the set threshold, an alarm is issued to characterize that the surveillance image may contain known threats or abnormal events.

[0063] According to the technical solution provided by the embodiment of the present disclosure, pixel blocks are divided into pixel blocks to obtain pixel block data corresponding to at least one image data to be processed, and each pixel block data is embedded to obtain an embedding vector corresponding to each pixel block data, and the image data to be processed is processed by a convolution layer to obtain a feature vector corresponding to the image data to be processed, and then the feature vector corresponding to the image data to be processed and the embedding vector of each pixel block can be processed by an attention mechanism to obtain a cross-attention vector corresponding to the image data to be processed, and the cross-attention vector corresponding to the image data to be processed and a preset query vector are processed by parameterization to obtain an interactive extraction vector corresponding to the image data to be processed, and the interactive extraction vector corresponding to the image data to be processed and each pixel block can be processed by a visual transformer. The pixel block embedding vector is processed to obtain the visual transformation vector corresponding to the image data to be processed, and then the visual transformation vector corresponding to the image data to be processed and the interaction extraction vector corresponding to the image data to be processed can be processed again to obtain the re-extraction vector corresponding to the image data to be processed. The re-extraction vector corresponding to the image data to be processed can be processed by a feedforward network to perform nonlinear transformation processing to obtain the target feature vector corresponding to the image data to be processed. Based on the target feature vector corresponding to the image data to be processed, it can be determined whether the image data to be processed in the intensive prediction task contains the image data to be processed, thereby improving the accuracy of image recognition, enhancing the generalization ability of the image processing model, improving the semantic richness of the target feature vector, improving the information dimension of the target feature vector, and improving the refinement of feature extraction.

[0064] In some embodiments, at least one pixel block embedding vector corresponding to the image data to be processed is determined based on the image data to be processed, including: performing pixel partitioning processing on the image data to be processed to obtain at least one pixel block data corresponding to the image data to be processed; performing embedded coding processing on each pixel block data corresponding to the image data to be processed to obtain each pixel block embedding vector corresponding to the image data to be processed.

[0065] Specifically, the image data to be processed can be subjected to pixel partitioning processing, which can divide the image data to be processed into multiple non-overlapping or partially overlapping rectangular areas, each area being a pixel block, and then at least one pixel block data corresponding to the image data to be processed can be obtained, and each pixel block data can be subjected to embedded coding processing, which can be implemented by a pre-trained neural network or an embedding layer, and each pixel block image data can be mapped to a high-dimensional vector space to obtain a corresponding pixel block embedding vector. The pixel block embedding vector can contain feature information of different areas in the image data to be processed.

[0066] The pixel block data corresponding to the image data to be processed may be a part of the image obtained by pixel partitioning, and the pixel block data may include pixel value information within a rectangular area in the image data to be processed. The pixel block data may be obtained by dividing the image into a plurality of non-overlapping or partially overlapping rectangular areas, and the pixel block data may be used in subsequent embedded coding processing, feature extraction, image recognition and other tasks.

[0067] For example, in a face recognition system, the input face image needs to be processed to extract the facial feature information for subsequent recognition tasks. The image processing model can use the face image as the image data to be processed, and can perform pixel partitioning to divide the face image into multiple pixel blocks, such as the eyes, nose, and mouth areas. Each pixel block can be embedded and encoded to obtain the corresponding pixel block embedding vector.

[0068] According to the technical solution provided by the embodiment of the present disclosure, by performing pixel partitioning processing on the image data to be processed, the pixel partitioning processing can divide the image data to be processed into multiple non-overlapping or partially overlapping rectangular areas, each area is a pixel block, and then at least one pixel block data corresponding to the image data to be processed can be obtained, and each pixel block data can be embedded and encoded, and each pixel block image data can be mapped to a high-dimensional vector space to obtain a corresponding pixel block embedding vector, thereby enhancing the local expression ability and detail capture ability of image features, improving the flexibility and scalability of the image processing model, and improving the efficiency and accuracy of image feature extraction.

[0069] In some embodiments, the feature vector corresponding to the image data to be processed and the embedding vectors of each pixel block are processed through an attention mechanism to obtain a cross-attention vector corresponding to the image data to be processed, including: performing self-attention processing on the feature vector corresponding to the image data to be processed to obtain a self-attention vector corresponding to the image data to be processed; performing cross-attention processing on the embedding vectors of each pixel block and the self-attention vector corresponding to the image data to be processed to obtain a cross-attention vector corresponding to the image data to be processed.

[0070] Specifically, the feature vector corresponding to the image data to be processed can be subjected to self-attention processing. The self-attention processing can calculate the correlation scores between different positions in the feature vector corresponding to the image data to be processed through the self-attention mechanism, and update the feature representation of each position, so as to obtain the self-attention vector corresponding to the image data to be processed. The self-attention vector can be used to characterize the dependency between different positions in the feature vector, and each pixel block embedding vector and the self-attention vector can be subjected to cross-attention processing. By calculating the correlation score between the pixel block embedding vector and the self-attention vector, the representation of each pixel block embedding vector is updated, so as to obtain the cross-attention vector corresponding to the image data to be processed.

[0071] Among them, the self-attention vector corresponding to the image data to be processed can be a feature vector obtained through self-attention processing, the self-attention vector can contain the information in the feature vector corresponding to the image data to be processed, and incorporate the dependency relationship between different positions in the feature vector. The self-attention vector corresponding to the image data to be processed can be obtained by processing the feature vector corresponding to the image data to be processed through the self-attention mechanism. The self-attention vector can be used for subsequent cross-attention processing, feature fusion, and image recognition.

[0072] For example, in medical image analysis, the image processing model can use medical images as image data to be processed, extract feature vectors corresponding to the medical images, and perform self-attention processing to obtain self-attention vectors, and then divide the medical images into multiple pixel blocks, perform embedding coding processing on each pixel block, and obtain pixel block embedding vectors. Then, the pixel block embedding vectors and self-attention vectors can be cross-attention processed to obtain cross-attention vectors.

[0073] According to the technical solution provided by the embodiment of the present disclosure, by performing self-attention processing on the feature vector corresponding to the image data to be processed and updating the feature representation of each position, the self-attention vector corresponding to the image data to be processed can be obtained. Each pixel block embedding vector and the self-attention vector can be cross-attention processed to obtain the cross-attention vector corresponding to the image data to be processed, thereby enhancing the global context perception ability and local detail capture ability of the image features, improving the accuracy and robustness of the image processing model, reducing the impact of noise and redundant information, improving the adaptability of the image processing model to complex image scenes, improving the semantic richness of the target feature vector, improving the information dimension of the target feature vector, and improving the precision of feature extraction.

[0074] In some embodiments, the cross-attention vector corresponding to the image data to be processed and the preset query vector are parameterized to obtain the interactive extraction vector corresponding to the image data to be processed, including: performing information extraction processing on the cross-attention vector corresponding to the image data to be processed to obtain the extraction vector corresponding to the image data to be processed; performing cross-attention processing on the extraction vector corresponding to the image data to be processed and the preset query vector to obtain the interactive extraction vector corresponding to the image data to be processed.

[0075] Specifically, the cross-attention vector corresponding to the image data to be processed can be subjected to information extraction processing, and the information extraction processing can be performed through the BI-Mamba module to extract the cross-attention vector to obtain the extraction vector corresponding to the image data to be processed. The extraction vector can be a compression and representation of the information in the cross-attention vector, and the extraction vector and the preset query vector can be subjected to cross-attention processing, and the correlation score calculation processing can be performed on the extraction vector and the preset query vector, and the representation of the extraction vector can be updated, so that the interactive extraction vector corresponding to the image data to be processed can be obtained.

[0076] The extracted vector corresponding to the image data to be processed may be a vector representation obtained by processing the cross-attention vector through the BI-Mamba module. The extracted vector may be a compression and representation of the information in the cross-attention vector.

[0077] For example, in an intelligent retrieval system, the image processing model can convert the query image or keyword input by the user into a preset query vector, and use the image to be retrieved as the image data to be processed. The cross-attention vector corresponding to the image data to be processed is processed through the BI-Mamba module to obtain an extraction vector, and then the extraction vector and the preset query vector can be cross-attention processed to obtain an interactive extraction vector.

[0078] According to the technical solution provided by the embodiment of the present disclosure, the cross-attention vector corresponding to the image data to be processed is processed by the BI-Mamba module to obtain the extraction vector corresponding to the image data to be processed. The extraction vector and the preset query vector can be cross-attention processed to obtain the interactive extraction vector corresponding to the image data to be processed, thereby enhancing the correlation between the image features and the target task, improving the pertinence and effectiveness of feature extraction, improving the adaptability of the image processing model to complex query conditions, enhancing the generalization ability of the image processing model, and improving the semantic richness of the target feature vector.

[0079] In some embodiments, the interactive extraction vectors and the embedding vectors of each pixel block corresponding to the image data to be processed are processed based on the visual transformation to obtain the visual transformation vector corresponding to the image data to be processed, including: splicing the extraction vectors and the embedding vectors of each pixel block corresponding to the image data to be processed to obtain the splicing vector corresponding to the image data to be processed; performing linear projection processing on the splicing vector corresponding to the image data to be processed to obtain the visual transformation vector corresponding to the image data to be processed.

[0080] Specifically, the extracted vector corresponding to the image data to be processed can be spliced ​​with the embedding vectors of each pixel block, and the extracted vector and the embedding vectors of all pixel blocks can be spliced ​​in dimension to obtain the splicing vector corresponding to the image data to be processed. Then, the splicing vector can be linearly projected through the ViT module to map the splicing vector to a vector space, so that the visual transformation vector corresponding to the image data to be processed can be obtained.

[0081] The splicing vector corresponding to the image data to be processed may be a vector obtained by splicing the extraction vector and each pixel block embedding vector in dimension. The splicing vector may be obtained by vector splicing.

[0082] For example, in an autonomous driving system, the image processing model can use the image taken by the camera in front of the vehicle as the image data to be processed, and can splice the extraction vector corresponding to the image data to be processed with the embedding vectors of all pixel blocks to obtain a splicing vector, and then can perform linear projection processing on the splicing vector through the ViT module to obtain a visual transformation vector.

[0083] According to the technical solution provided by the embodiments of the present disclosure, the extracted vector corresponding to the image data to be processed can be spliced ​​with the embedding vectors of each pixel block, and the extracted vector and the embedding vectors of all pixel blocks can be spliced ​​in dimension to obtain the splicing vector corresponding to the image data to be processed, and then the splicing vector can be linearly projected through the ViT module to map the splicing vector to a vector space, so as to obtain the visual transformation vector corresponding to the image data to be processed, thereby enhancing the expressiveness and robustness of image features, improving the efficiency and accuracy of feature fusion, improving the processing capability of the image processing model for complex image data, improving the semantic richness of the target feature vector, and improving the information dimension of the target feature vector.

[0084] In some embodiments, information extraction processing is performed on the visual transformation vector corresponding to the image data to be processed and the interaction extraction vector corresponding to the image data to be processed to obtain a re-extraction vector corresponding to the image data to be processed, including: cross-attention processing is performed on the visual transformation vector corresponding to the image data to be processed and the interaction extraction vector corresponding to the image data to be processed to obtain a visual transformation interaction vector corresponding to the image data to be processed; information extraction processing is performed on the visual transformation interaction vector corresponding to the image data to be processed to obtain a re-extraction vector corresponding to the image data to be processed.

[0085] Specifically, the visual transformation vector and the interaction extraction vector corresponding to the image data to be processed can be cross-attention processed, for example, the correlation score of the visual transformation vector and the interaction extraction vector can be calculated, so as to obtain the visual transformation interaction vector corresponding to the image data to be processed. Then, the visual transformation interaction vector can be processed by the BI-Mamba module to obtain the re-extraction vector corresponding to the image data to be processed.

[0086] The visual transformation interaction vector corresponding to the image data to be processed can be a vector representation obtained by cross-attention processing of the visual transformation vector and the interaction extraction vector. The visual transformation interaction vector integrates the information of the visual transformation vector and the interaction extraction vector, and the visual transformation interaction vector can be used to characterize the association and interaction between the visual transformation vector and the interaction extraction vector.

[0087] For example, in a face recognition system, the image processing model can use the face image to be recognized as the image data to be processed, perform cross-attention processing on the visual transformation vector and the interaction extraction vector to obtain the visual transformation interaction vector, and then process the visual transformation interaction vector through the BI-Mamba module to obtain the re-extraction vector.

[0088] According to the technical solution provided by the embodiment of the present disclosure, by performing cross-attention processing on the visual transformation vector and the interaction extraction vector corresponding to the image data to be processed, and calculating the correlation score of the visual transformation vector and the interaction extraction vector, the visual transformation interaction vector corresponding to the image data to be processed can be obtained. Then, the visual transformation interaction vector can be processed by the BI-Mamba module to obtain the re-extraction vector corresponding to the image data to be processed, thereby enhancing the interaction and fusion capabilities between image features, improving the accuracy and efficiency of feature extraction, improving the image processing model's understanding and processing capabilities for complex image data, improving the semantic richness of the target feature vector, improving the information dimension of the target feature vector, and improving the precision of feature extraction.

[0089] In some embodiments, a nonlinear transformation is performed on the re-extracted vector corresponding to the image data to be processed to obtain a target feature vector corresponding to the image data to be processed, including: performing a linear transformation on the re-extracted vector corresponding to the image data to be processed to obtain a linear transformation vector corresponding to the image data to be processed; performing a nonlinear activation on the linear transformation vector corresponding to the image data to be processed to obtain a target feature vector corresponding to the image data to be processed.

[0090] Specifically, the re-extracted vector corresponding to the image data to be processed can be linearly transformed through a feedforward network to obtain a linear transformation vector corresponding to the image data to be processed, and then the linear transformation vector can be nonlinearly activated through a nonlinear activation function. The nonlinear activation function includes but is not limited to ReLU, Sigmoid, or Tanh, etc. The linear transformation vector can be nonlinearly transformed element by element, so as to obtain a target feature vector corresponding to the image data to be processed.

[0091] The linear transformation vector corresponding to the image data to be processed may be a vector representation obtained by performing linear transformation processing on the re-extracted vector through a feedforward network. The linear transformation vector may be used for subsequent nonlinear activation processing, feature fusion, or image recognition.

[0092] For example, in a medical image analysis system, the image processing model can use the medical image to be analyzed as the image data to be processed, perform linear transformation processing on the re-extracted vector corresponding to the medical image through a feedforward network to obtain a linear transformation vector, and then perform nonlinear activation processing on the linear transformation vector through a ReLU function to obtain a target feature vector.

[0093] According to the technical solution provided by the embodiment of the present disclosure, a linear transformation processing is performed on the re-extracted vector corresponding to the image data to be processed through a feedforward network to obtain a linear transformation vector corresponding to the image data to be processed, and then a nonlinear activation function can be used to perform nonlinear activation processing on the linear transformation vector, and the linear transformation vector can be nonlinearly transformed element by element, so as to obtain a target feature vector corresponding to the image data to be processed, thereby enhancing the expressiveness and robustness of the feature vector, improving the flexibility and adaptability of feature extraction, improving the recognition ability of the image processing model for complex image data, and improving the recognition accuracy and efficiency.

[0094] Figure 3 FIG. 1 is a schematic diagram of a processing flow of an image processing model provided by an embodiment of the present disclosure. Figure 3 As shown, the processing flow of the image processing model includes:

[0095] A first BI-Mamba module 301 is used to perform information extraction processing on the cross-attention vector corresponding to the image data to be processed, so as to obtain an extraction vector corresponding to the image data to be processed. The processing performed by the above-mentioned BI-Mamba module can be performed by the first BI-Mamba module 301 or the second BI-Mamba module 303;

[0096] The ViT module 302 is used to process the interaction extraction vector and each pixel block embedding vector corresponding to the image data to be processed based on the visual transformation to obtain the visual transformation vector corresponding to the image data to be processed;

[0097] The second BI-Mamba module 303 is used to perform information extraction processing on the visual transformation interaction vector corresponding to the image data to be processed, and obtain a re-extracted vector corresponding to the image data to be processed;

[0098] The feed-forward network 304 is used to perform nonlinear transformation processing on the re-extracted vector corresponding to the image data to be processed to obtain the target feature vector corresponding to the image data to be processed.

[0099] According to the technical solution provided by the embodiment of the present disclosure, the image data to be processed can be divided into pixel blocks of fixed size for input of the ViT module 302, and the image data to be processed can be converted into processable block embedding features through the ViT module 302; the image data to be processed can be processed by a lightweight convolutional network to obtain a preliminary feature representation of the image data to be processed, the image features extracted by convolution can be self-attention processed, the features obtained by self-attention can be used as key-value pairs, and the image embedding features of the block encoding of the image data to be processed can be used as query vectors for cross-attention calculation, and then the first BI-Mamba module 301 can be used to further extract and enhance the feature representation, enhance the model's ability to capture details, and the preset query vector and the features obtained in the previous step can be cross-attention processed. The calculation of cross attention can further extract more important feature information. The features obtained in the previous step and the features of the ViT module 302 can be cross-attended. The features obtained by block division and the features obtained by the first BI-Mamba module can be added and fused and input into the ViT module 302 for processing. The output of the ViT module 302 can be fused through cross attention, and then the features obtained in the previous step can be input into the second BI-Mamba module 303 to further refine the features, and then a nonlinear transformation can be performed through the feedforward network 304 to obtain the target feature vector, thereby improving the accuracy of image recognition, enhancing the generalization ability of the image processing model, improving the semantic richness of the target feature vector, improving the information dimension of the target feature vector, and improving the precision of feature extraction.

[0100] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0101] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0102] Figure 4 is a schematic diagram of an image processing device provided by an embodiment of the present disclosure. Figure 4 As shown, the image processing device comprises:

[0103] A first determination module 401 is used to determine at least one pixel block embedding vector corresponding to the image data to be processed according to the image data to be processed;

[0104] The first processing module 402 is used to perform convolution processing on the image data to be processed to obtain a feature vector corresponding to the image data to be processed;

[0105] The second processing module 403 is used to process the feature vector corresponding to the image data to be processed and the embedding vector of each pixel block through the attention mechanism to obtain the cross attention vector corresponding to the image data to be processed;

[0106] The third processing module 404 is used to perform parameterized processing on the cross attention vector corresponding to the image data to be processed and the preset query vector to obtain an interactive extraction vector corresponding to the image data to be processed;

[0107] A fourth processing module 405 is used to process the interaction extraction vector and each pixel block embedding vector corresponding to the image data to be processed based on the visual transformation to obtain a visual transformation vector corresponding to the image data to be processed;

[0108] A fifth processing module 406 is used to perform information extraction processing on the visual transformation vector corresponding to the image data to be processed and the interactive extraction vector corresponding to the image data to be processed, so as to obtain a re-extraction vector corresponding to the image data to be processed;

[0109] The sixth processing module 407 is used to perform nonlinear transformation processing on the re-extracted vector corresponding to the image data to be processed to obtain a target feature vector corresponding to the image data to be processed;

[0110] The second determination module 408 is used to determine whether the image data to be processed includes the image data to be processed based on the target feature vector corresponding to the image data to be processed.

[0111] According to the technical solution provided by the embodiment of the present disclosure, pixel blocks are divided into pixel blocks to obtain pixel block data corresponding to at least one image data to be processed, and each pixel block data is embedded to obtain an embedding vector corresponding to each pixel block data, and the image data to be processed is processed by a convolution layer to obtain a feature vector corresponding to the image data to be processed, and then the feature vector corresponding to the image data to be processed and the embedding vector of each pixel block can be processed by an attention mechanism to obtain a cross-attention vector corresponding to the image data to be processed, and the cross-attention vector corresponding to the image data to be processed and a preset query vector are processed by parameterization to obtain an interactive extraction vector corresponding to the image data to be processed, and the interactive extraction vector corresponding to the image data to be processed and each pixel block can be processed by a visual transformer. The pixel block embedding vector is processed to obtain the visual transformation vector corresponding to the image data to be processed, and then the visual transformation vector corresponding to the image data to be processed and the interaction extraction vector corresponding to the image data to be processed can be processed again to obtain the re-extraction vector corresponding to the image data to be processed. The re-extraction vector corresponding to the image data to be processed can be processed by a feedforward network to perform nonlinear transformation processing to obtain the target feature vector corresponding to the image data to be processed. Based on the target feature vector corresponding to the image data to be processed, it can be determined whether the image data to be processed in the intensive prediction task contains the image data to be processed, thereby improving the accuracy of image recognition, enhancing the generalization ability of the image processing model, improving the semantic richness of the target feature vector, improving the information dimension of the target feature vector, and improving the refinement of feature extraction.

[0112] In some embodiments, the first determination module 401 is specifically used to perform pixel partitioning processing on the image data to be processed to obtain at least one pixel block data corresponding to the image data to be processed; and perform embedded coding processing on each pixel block data corresponding to the image data to be processed to obtain each pixel block embedding vector corresponding to the image data to be processed.

[0113] In some embodiments, the second processing module 403 is specifically used to perform self-attention processing on the feature vector corresponding to the image data to be processed to obtain the self-attention vector corresponding to the image data to be processed; and perform cross-attention processing on each pixel block embedding vector and the self-attention vector corresponding to the image data to be processed to obtain the cross-attention vector corresponding to the image data to be processed.

[0114] In some embodiments, the third processing module 404 is specifically used to perform information extraction processing on the cross-attention vector corresponding to the image data to be processed to obtain the extraction vector corresponding to the image data to be processed; and perform cross-attention processing on the extraction vector corresponding to the image data to be processed and the preset query vector to obtain the interaction extraction vector corresponding to the image data to be processed.

[0115] In some embodiments, the fourth processing module 405 is specifically used to perform splicing processing on the extraction vector corresponding to the image data to be processed and the embedding vectors of each pixel block to obtain the splicing vector corresponding to the image data to be processed; and perform linear projection processing on the splicing vector corresponding to the image data to be processed to obtain the visual transformation vector corresponding to the image data to be processed.

[0116] In some embodiments, the fifth processing module 406 is specifically used to perform cross-attention processing on the visual transformation vector corresponding to the image data to be processed and the interaction extraction vector corresponding to the image data to be processed to obtain the visual transformation interaction vector corresponding to the image data to be processed; and perform information extraction processing on the visual transformation interaction vector corresponding to the image data to be processed to obtain the re-extraction vector corresponding to the image data to be processed.

[0117] In some embodiments, the sixth processing module 407 is specifically used to perform linear transformation processing on the re-extracted vector corresponding to the image data to be processed to obtain the linear transformation vector corresponding to the image data to be processed; and perform nonlinear activation processing on the linear transformation vector corresponding to the image data to be processed to obtain the target feature vector corresponding to the image data to be processed.

[0118] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

[0119] Figure 5 Schematic diagram of an electronic device 5 provided in an embodiment of the present disclosure. Figure 5 As shown, the electronic device 5 of this embodiment includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 501 executes the computer program 503, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0120] The electronic device 5 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 5 may include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art will appreciate that Figure 5 The electronic device 5 is merely an example and does not limit the electronic device 5 , and may include more or less components than those shown in the figure, or different components.

[0121] The processor 501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0122] The memory 502 may be an internal storage unit of the electronic device 5, for example, a hard disk or memory of the electronic device 5. The memory 502 may also be an external storage device of the electronic device 5, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 5. The memory 502 may also include both an internal storage unit of the electronic device 5 and an external storage device. The memory 502 is used to store computer programs and other programs and data required by the electronic device.

[0123] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0124] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0125] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.

Claims

1. An image processing method, characterized in that: include: Determining, according to the image data to be processed, at least one pixel block embedding vector corresponding to the image data to be processed; Performing convolution processing on the image data to be processed to obtain a feature vector corresponding to the image data to be processed; Processing the feature vector corresponding to the image data to be processed and each pixel block embedding vector through an attention mechanism to obtain a cross attention vector corresponding to the image data to be processed; Performing parameterization processing on the cross attention vector corresponding to the image data to be processed and a preset query vector to obtain an interactive extraction vector corresponding to the image data to be processed; Processing the interactive extraction vector corresponding to the image data to be processed and each pixel block embedding vector based on visual transformation to obtain a visual transformation vector corresponding to the image data to be processed; Performing information extraction processing on the visual transformation vector corresponding to the image data to be processed and the interactive extraction vector corresponding to the image data to be processed to obtain a re-extraction vector corresponding to the image data to be processed; Performing nonlinear transformation processing on the re-extracted vector corresponding to the image data to be processed to obtain a target feature vector corresponding to the image data to be processed; Based on the target feature vector corresponding to the image data to be processed, it is determined whether the image data to be processed includes the image data to be processed.

2. The image processing method according to claim 1, characterized in that: The step of determining at least one pixel block embedding vector corresponding to the image data to be processed according to the image data to be processed comprises: Performing pixel partitioning processing on the image data to be processed to obtain at least one pixel block data corresponding to the image data to be processed; Embedding coding is performed on each pixel block data corresponding to the image data to be processed to obtain an embedding vector for each pixel block corresponding to the image data to be processed.

3. The image processing method according to claim 1, characterized in that: The process of processing the feature vector corresponding to the image data to be processed and each pixel block embedding vector by the attention mechanism to obtain the cross attention vector corresponding to the image data to be processed includes: Performing self-attention processing on the feature vector corresponding to the image data to be processed to obtain a self-attention vector corresponding to the image data to be processed; Cross-attention processing is performed on each pixel block embedding vector and the self-attention vector corresponding to the image data to be processed to obtain a cross-attention vector corresponding to the image data to be processed.

4. The image processing method according to claim 1, characterized in that: The parameterizing of the cross attention vector corresponding to the image data to be processed and the preset query vector to obtain the interaction extraction vector corresponding to the image data to be processed includes: Performing information extraction processing on the cross attention vector corresponding to the image data to be processed to obtain an extraction vector corresponding to the image data to be processed; The extraction vector corresponding to the image data to be processed and the preset query vector are cross-attention processed to obtain an interactive extraction vector corresponding to the image data to be processed.

5. The image processing method according to claim 1, characterized in that: The step of processing the interactive extraction vector corresponding to the image data to be processed and each pixel block embedding vector based on the visual transformation to obtain the visual transformation vector corresponding to the image data to be processed includes: Performing splicing processing on the extraction vector corresponding to the image data to be processed and the embedding vectors of each pixel block to obtain a splicing vector corresponding to the image data to be processed; Linear projection processing is performed on the splicing vector corresponding to the image data to be processed to obtain a visual transformation vector corresponding to the image data to be processed.

6. The image processing method according to claim 1, characterized in that: The performing information extraction processing on the visual transformation vector corresponding to the image data to be processed and the interactive extraction vector corresponding to the image data to be processed to obtain the re-extraction vector corresponding to the image data to be processed includes: Performing cross-attention processing on the visual transformation vector corresponding to the image data to be processed and the interaction extraction vector corresponding to the image data to be processed to obtain the visual transformation interaction vector corresponding to the image data to be processed; Information extraction processing is performed on the visual transformation interaction vector corresponding to the image data to be processed to obtain a re-extracted vector corresponding to the image data to be processed.

7. The image processing method according to claim 1, characterized in that: The performing nonlinear transformation processing on the re-extracted vector corresponding to the image data to be processed to obtain the target feature vector corresponding to the image data to be processed includes: Performing linear transformation processing on the re-extracted vector corresponding to the image data to be processed to obtain a linear transformation vector corresponding to the image data to be processed; Nonlinear activation processing is performed on the linear transformation vector corresponding to the image data to be processed to obtain a target feature vector corresponding to the image data to be processed.

8. An image processing device, characterized in that: include: A first determination module, configured to determine, based on the image data to be processed, at least one pixel block embedding vector corresponding to the image data to be processed; A first processing module, configured to perform convolution processing on the image data to be processed to obtain a feature vector corresponding to the image data to be processed; A second processing module is used to process the feature vector corresponding to the image data to be processed and each pixel block embedding vector through an attention mechanism to obtain a cross attention vector corresponding to the image data to be processed; A third processing module is used to perform parameterized processing on the cross attention vector corresponding to the image data to be processed and a preset query vector to obtain an interactive extraction vector corresponding to the image data to be processed; A fourth processing module, used for processing the interaction extraction vector corresponding to the image data to be processed and each pixel block embedding vector based on visual transformation to obtain a visual transformation vector corresponding to the image data to be processed; a fifth processing module, configured to perform information extraction processing on the visual transformation vector corresponding to the image data to be processed and the interactive extraction vector corresponding to the image data to be processed, so as to obtain a re-extraction vector corresponding to the image data to be processed; A sixth processing module, configured to perform nonlinear transformation processing on the re-extracted vector corresponding to the image data to be processed, so as to obtain a target feature vector corresponding to the image data to be processed; The second determination module is used to determine whether the image data to be processed includes the image data to be processed based on the target feature vector corresponding to the image data to be processed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images