Image processing method, model training method, device, equipment and medium

By dividing the image to be processed into multiple image blocks and calculating the modal correlation, the accuracy problem caused by the randomness of the modal type in image classification is solved, which improves classification accuracy and expands the scope of application of the model.

CN114693977BActive Publication Date: 2024-12-03BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210358363.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-06
Publication Date
2024-12-03
Estimated Expiration
2042-04-06

AI Technical Summary

Technical Problem

In the prior art In the image classification task, the randomness of the modal type of the image to be processed leads to a decrease in the accuracy of the classification results, and the scope of use of the model is limited by the consistency requirements of the modal type during training.

Method used

By dividing the image to be processed into multiple image blocks, the modal correlation of each image block is calculated, and the parameters of the image processing model are adjusted based on the modal correlation, and the feature information of multiple image blocks is used for classification, and even in the mode is Different state types improve classification accuracy at the same time.

Benefits of technology

Improve the correlation between image blocks and modals, improve the accuracy of image classification, expand the scope of use of image processing models, allow image blocks of different modal types to participate in classification, and reduce the classification results of modal information interference.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114693977B_ABST
    Figure CN114693977B_ABST
Patent Text Reader

Abstract

The present disclosure provides an image processing method, a model training method, an apparatus, a device and a medium, which relate to the field of artificial intelligence, and in particular to the field of computer vision. The implementation scheme is: dividing the image to be processed into multiple image blocks; based on the feature information of each of the multiple image blocks, determining the modal correlation of each of the multiple image blocks, wherein the modal correlation of each image block represents the degree of correlation between the feature information of the image block and the modal type of the image to be processed; and based on the modal correlation of each of the multiple image blocks, using the feature information of at least part of the multiple image blocks to perform classification for the image to be processed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of computer vision, and specifically to an image processing method, a model training method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0004] The present disclosure provides an image processing method, a model training method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of the present disclosure, there is provided an image processing method, comprising: dividing an image to be processed into a plurality of image blocks; determining a modal correlation of each of the plurality of image blocks based on feature information of each of the plurality of image blocks, wherein the modal correlation of each image block represents a degree of correlation between the feature information of the image block and a modal type of the image to be processed; and performing classification of the image to be processed using feature information of at least part of the plurality of image blocks based on the modal correlation of each of the plurality of image blocks.

[0006] According to another aspect of the present disclosure, a training method for an image processing model is provided, the image processing model includes at least a modal correlation calculation module, a classification information update module and a classification prediction module, the training method including: dividing a sample image into a plurality of image blocks, wherein the sample image has a classification label; determining a feature sequence corresponding to the sample image, wherein the feature sequence includes feature information corresponding to each of the plurality of image blocks, modal information of the sample image and classification information of the sample image; inputting the feature information and modal information corresponding to each of the plurality of image blocks in the feature sequence into the modal correlation calculation module to obtain the modal correlation of each of the plurality of image blocks; inputting the feature information, classification information corresponding to each of the plurality of image blocks in the feature sequence and the modal correlation of each of the plurality of image blocks into the classification information update module to obtain updated classification information; inputting the classification information into the classification prediction module to obtain a first predicted classification of the sample image; and adjusting the parameters of the image processing model based on the classification label and the first predicted classification of the sample image.

[0007] According to another aspect of the present disclosure, there is provided an image processing apparatus, comprising: a first division unit, configured to divide an image to be processed into a plurality of image blocks; a first determination unit, configured to determine a modal correlation of each of the plurality of image blocks based on feature information of each of the plurality of image blocks, wherein the modal correlation of each image block represents a degree of correlation between the feature information of the image block and a modal type of the image to be processed; and a classification unit, configured to perform classification of the image to be processed based on the modal correlation of each of the plurality of image blocks and using feature information of at least some of the plurality of image blocks.

[0008] According to another aspect of the present disclosure, a training device for an image processing model is provided, wherein the image processing model at least includes a modal correlation calculation module, a classification information update module, and a classification prediction module, and the training device includes: a second division unit, configured to divide a sample image into a plurality of image blocks, wherein the sample image has a classification label; a second determination unit, configured to determine a feature sequence corresponding to the sample image, wherein the feature sequence includes feature information corresponding to each of the plurality of image blocks, modal information of the sample image, and classification information of the sample image; a first acquisition unit, configured to obtain each of the plurality of image blocks in the feature sequence; The feature information and modal information corresponding to each of the multiple image blocks in the feature sequence are input into the modal correlation calculation module to obtain the modal correlation of each of the multiple image blocks; the second acquisition unit is configured to input the feature information, classification information corresponding to each of the multiple image blocks in the feature sequence and the modal correlation of each of the multiple image blocks into the classification information update module to obtain updated classification information; the third acquisition unit is configured to input the classification information into the classification prediction module to obtain a first predicted classification of the sample image; and the first adjustment unit is configured to adjust the parameters of the image processing model based on the classification label of the sample image and the first predicted classification.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the above methods.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute any one of the above methods.

[0011] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements any one of the above methods when executed by a processor.

[0012] According to one or more embodiments of the present disclosure, classification processing of the image to be processed can be performed with reference to the modal relevance of each image block of the image to be processed, thereby improving the ability to perceive the degree of relevance of each image block with the modality and improving the accuracy of classification.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0015] Figure 1 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0016] Figure 2 A flowchart of an image processing method according to an embodiment of the present disclosure is shown;

[0017] Figure 3 A schematic diagram showing an image processing method according to an embodiment of the present disclosure is shown;

[0018] Figure 4 A schematic diagram showing another image processing method according to an embodiment of the present disclosure is shown;

[0019] Figure 5 A flowchart of a method for training an image processing model according to an embodiment of the present disclosure is shown;

[0020] Figure 6 A schematic diagram showing a characteristic sequence according to an embodiment of the present disclosure;

[0021] Fig. 7A and Figure 7B A schematic diagram showing a training method for an image processing device according to an embodiment of the present disclosure is shown;

[0022] Figure 8 A structural block diagram of an image processing device according to an embodiment of the present disclosure is shown;

[0023] Fig. 9 A structural block diagram of a training device for an image processing model according to an embodiment of the present disclosure is shown; and

[0024] Fig.10 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0026] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0027] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0028] In the related art, when performing a classification task on an image to be processed, it is necessary to first extract feature information from the image to be processed, and then input the extracted feature information into a trained classification model to obtain a classification result. For example, in the field of face anti-counterfeiting, feature information is extracted from an image to be processed containing a face, and the feature information is input into a trained liveness detection model to obtain a classification result output by the liveness detection model, that is, whether the face captured in the image to be processed is the face of a real user, or a face disguised by a photo, video or mask.

[0029] With the improvement of imaging technology, multiple modality types of images can be provided for the same shooting field of view or the same shooting object, such as RGB images, depth images, infrared images, etc. This makes the modality type of the image to be processed highly random, and the image to be processed may even include multiple modality images for the same object at the same time. The applicant found that when performing classification tasks based on image content such as object recognition and liveness detection on the image to be processed, the information on the modality type of the image to be processed not only fails to improve the accuracy of the classification result, but may even interfere with the classification result.

[0030] Based on this, the present disclosure proposes an image processing method, which determines the modal relevance of each of the multiple image blocks based on the feature information of each of the multiple image blocks of the image to be processed, and based on the modal relevance of each of the multiple image blocks, uses the feature information of at least some of the multiple image blocks to perform classification for the image to be processed. Thus, the feature information of each image block in the image to be processed can be applied to the classification of the image to be processed based on the degree of correlation between the feature information and the modal type of the image to be processed, so that the classification processing of the image to be processed can be performed with reference to the modal relevance of each image block, thereby improving the perception of the degree of correlation between each image block and the modality, and improving the accuracy of classification.

[0031] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0032] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

[0033] In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable execution of an image processing method or a model training method.

[0034] In some embodiments, server 120 may also provide other services or software applications that may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0035] exist Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0036] The user may use client devices 101, 102, 103, 104, 105 and / or 106 to acquire or transmit an image to be processed. The client device may provide an interface that enables the user of the client device to interact with the client device. The client device may also output information to the user via the interface. Figure 1 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0037] Client devices 101, 102, 103, 104, 105 and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems; or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.

[0038] The network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0039] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0040] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0041] In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and / or 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and / or 106.

[0042] In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0043] The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. The databases 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the databases used by the server 120 may be, for example, relational databases. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.

[0044] In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0045] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0046] Figure 2 A flowchart of an image processing method according to an exemplary embodiment of the present disclosure is shown. Figure 2 As shown, the method 200 includes: step S201, dividing the image to be processed into multiple image blocks; step S202, determining the modal correlation of each of the multiple image blocks based on the feature information of each of the multiple image blocks, wherein the modal correlation of each image block represents the degree of correlation between the feature information of the image block and the modal type of the image to be processed; and step S203, based on the modal correlation of each of the multiple image blocks, using the feature information of at least part of the multiple image blocks to perform classification for the image to be processed.

[0047] Therefore, the feature information of each image block in the image to be processed can be applied to the classification of the image to be processed based on the degree of correlation between the feature information and the modal type of the image to be processed, so that the classification processing of the image to be processed can be performed with reference to the modal correlation of each image block, thereby improving the accuracy of the classification of the image to be processed.

[0048] At the same time, although there are a variety of models in the current technical field that can perform classification processing on the images to be processed, these models often require that the modality type of the image to be processed is consistent with the modality type of the sample image used in the training process of the model during application. For example, if the model uses RGB sample images for training during the training process, the image to be processed in the application process is required to be an RGB image, otherwise, the accuracy of the classification result cannot be guaranteed. For another example, if the model uses RGB sample images and depth sample images for fusion training during the training process, the image to be processed in the application process is required to include both RGB images and depth images, otherwise, the accuracy of the classification result cannot be guaranteed. This undoubtedly severely limits the scope of use of the model.

[0049] Different from this, the image processing method provided by the present invention can be executed on the basis of knowing the modal correlation of each image block. Even if the modal type of the image to be processed is different from the modal type of the sample image in the model training process, the accuracy of the model's classification results of the image to be processed can be improved by excluding or reducing the influence of image blocks with strong correlation on the basis of obtaining the modal correlation of each image block.

[0050] For step S201, the image to be processed may be divided into a plurality of image blocks based on a preset number of image blocks. For example, if the size of the image to be processed is 256×256, and if the preset number of image blocks is 64, the size of each image block is 32×32.

[0051] For step S202, the characteristic information of each of the multiple image blocks may include spectrum information of the image block. Since image data of different modal types often belong to different spectrum spaces, it is easier to obtain the modal correlation of each image block when the characteristic information of each of the multiple image blocks includes spectrum information.

[0052] In one implementation, the feature information of each of the multiple image blocks may include content information of the image block, location information of the image block, and spectrum information of the image block.

[0053] In one implementation, feature information of each of the plurality of image blocks may be acquired through a trained feature extraction model.

[0054] According to some embodiments, before determining the modal correlation of each of the multiple image blocks, for each of the multiple image blocks, feature information corresponding to the image block can be updated by fusing feature information of each of the multiple image blocks. The updated feature information of each image block can fuse the feature information of each of the multiple image blocks, so that the updated feature information can include context information of the image block, thereby more effectively representing the image block.

[0055] In one implementation, the above fusion can be achieved through a self-attention mechanism.

[0056] For step S203, on the basis of determining the modal correlation of each of the multiple image blocks, the feature information of each of the multiple image blocks can be differentially utilized based on the modal correlation of each of the multiple image blocks to perform classification for the image to be processed.

[0057] According to some embodiments, based on the modal correlation of each of the multiple image blocks, performing classification for the image to be processed using feature information of at least part of the multiple image blocks may include: selecting one or more target image blocks from the multiple image blocks based on the modal correlation of each of the multiple image blocks; and performing classification for the image to be processed using feature information of each of the one or more target image blocks.

[0058] Thus, multiple image blocks can be screened based on the modal relevance of each of the image blocks, so that the image blocks used to perform classification on the image to be processed can all meet the requirements of the classification process on the modal relevance.

[0059] According to some embodiments, based on the modal correlation of each of the multiple image blocks, selecting one or more target image blocks from the multiple image blocks may include: for each of the multiple image blocks, in response to the modal correlation of the image block being less than a preset threshold, determining the image block as a target image block.

[0060] In this way, image blocks with modality correlation greater than a preset threshold, that is, image blocks strongly correlated with the modality type of the image to be processed, can be excluded from the classification process, so that the classification process of the image to be processed is free from interference from modality information.

[0061] According to some embodiments, based on the modal correlation of each of a plurality of image blocks, utilizing feature information of at least a portion of the plurality of image blocks to perform classification of the image to be processed includes: calculating a weighted sum of the feature information of each of the plurality of image blocks, wherein the weight corresponding to each image block is negatively correlated with the modal correlation of the image block; and performing classification of the image to be processed based on the weighted sum.

[0062] In this way, the degree of influence of the characteristic information of each image block on the classification of the image to be processed can be influenced by adjusting the weight corresponding to each image block, that is, the influence of the modal information of the image to be processed on the classification can be reduced by reducing the weight of the image block that is strongly correlated with the modal information.

[0063] Figure 3 A schematic diagram of an image processing method according to an exemplary embodiment of the present disclosure is shown. Figure 3 As shown, the single-modal fusion model 300 based on the attention mechanism can be used to perform classification of the image to be processed.

[0064] In order to be able to obtain a feature set z composed of feature information of each of the multiple image blocks of the image to be processed pat 301 extracts valid information for performing classification and sets the modal information z of the image to be processed mod 302 and classification information of the image to be processed cls 303. Modal information z mod 302 and Classification Information cls 303 can be obtained by using a trained transformer to fuse the feature information of each of the multiple image blocks.

[0065] The feature set z pat 301 is input into the modal key vector calculation module 311 in the single modal fusion model 300, wherein the modal key vector calculation module 311 includes a parameter matrix In the modal key vector calculation module 311, by computing the feature set z pat 301 and parameter matrix Perform matrix operations to obtain the modal key vector

[0066] The modal information z mod 302 is input into the modal query vector calculation module 312 in the single modal fusion model 300, wherein the modal query vector calculation module 312 includes a parameter matrix In the modal query vector calculation module 312, by calculating the modal information z mod 302 and parameter matrix Perform matrix operations to obtain the modal query vector q mod .

[0067] By modal key vector and the modal query vector q mod By performing the multiplication operation, we can get the modal correlation matrix map mod . Modal correlation matrix map mod The modal correlation matrix map contains the modal correlation of each of the multiple image blocks. mod The modal correlation of each image block in the image block can be used to locate the image blocks that are strongly correlated with the modal information among multiple image blocks, that is, the image blocks whose modal correlation is greater than the threshold.

[0068] The feature set z pat 301 is input into the classification key vector calculation module 314 and the classification value vector calculation module 315 in the single modality fusion model 300. The classification key vector calculation module 314 includes the parameter matrix In the classification key vector calculation module 314, by computing the feature set z pat 301 and parameter matrix Perform matrix operations to obtain the classification key vector The classification value vector calculation module 315 includes a parameter matrix In the classification value vector calculation module 315, by pat 301 and parameter matrix Perform matrix operations to obtain a categorical value vector

[0069] Classify information cls 303 is input into the classification query vector calculation module 313, wherein the classification query vector calculation module 313 includes a parameter matrix In the classification query vector calculation module 313, by cls 303 and parameter matrix Perform matrix operations to obtain the classification query vector q cls .

[0070] By sorting the key vector and the classification query vector q cls By performing the multiplication operation, we can get the classification correlation matrix map cls . Classification correlation matrix map cls The influence degree of each image block in the multiple image blocks on the classification result is determined. In the classification relevance adjustment module 320, based on the modality relevance matrix map mod The image blocks that are strongly correlated with the modal information among the multiple image blocks determined can adjust the classification correlation matrix map cls The value in is used to exclude or reduce the influence of image blocks that are strongly related to modal information on the classification results.

[0071] After the softmax function of the softmax module 330 is calculated, the adjusted classification correlation matrix map cls with categorical value vector Perform matrix multiplication to obtain the updated classification information z cls '304. Such classified information cls '304 excludes or reduces the strong modal information in the image to be processed, and subsequent classification calculations are performed based on the updated classification information 304, which can improve the accuracy of the classification results.

[0072] For example, the above Figure 3 The execution process in can be expressed by the following formula:

[0073]

[0074]

[0075]

[0076]

[0077] M=Γ λ (map mod ),

[0078]

[0079] Among them, D represents the dimension of the feature information of each image block, and h represents the number of heads in the multi-head attention mechanism.

[0080] Γ λ (·) is the threshold selection function, which is used to determine the image blocks with strong correlation of modal information. modThe threshold is set in , and a mask matrix M with values ​​of 0 and 1 is output. The mask matrix M is composed of 0 and 1, where 0 indicates that the feature information of the image block corresponding to the position is weakly correlated with the modal information, and 1 indicates that the feature information of the image block corresponding to the position is strongly correlated with the modal information.

[0081] Selection function Γ' based on mask matrix M M (·) is further expressed as:

[0082]

[0083] For any matrix A, for a position greater than 0 in the mask matrix M, the value of the matrix A at that position is assigned to negative infinity; for a position equal to 0 in the mask matrix M, the value of the matrix A at that position remains unchanged.

[0084] It can be understood that the single-modal fusion model 300 only updates the classification information of the image to be processed, while the feature set and modality information constituted by the feature information of each of the multiple image blocks of the image to be processed remain unchanged.

[0085] According to some embodiments, the image to be processed includes multiple modal images of the same object, the modal type of each of the multiple modal images is different from each other, and wherein, using the feature information of at least part of the multiple image blocks to perform classification of the image to be processed may include: for each of the multiple modal images, using the feature information of at least part of the multiple image blocks in the modal image to determine the representation information of the modal image; by fusing the representation information of each of the multiple modal images, obtaining fusion information of the image to be processed; and performing classification of the image to be processed based on the fusion information.

[0086] Images of different modalities often have their own advantages and disadvantages in representing object features. By fusing multiple modality images of the same object and performing classification of the image to be processed based on the fused information, the accuracy of the classification of the image to be processed can be effectively improved.

[0087] Figure 4 A schematic diagram showing another image processing method according to an exemplary embodiment of the present disclosure is shown. Figure 4 As shown, when the image to be processed includes multiple modal images, a multimodal fusion model 400 based on an attention mechanism can be used to perform classification of the image to be processed.

[0088] For the convenience of description, the first modality image and the second modality image are taken as examples for description herein. Those skilled in the art know that the method is also applicable to processing of more than three modality images.

[0089] The representation information z of the first modality image r 401 and the representation information z of the second modality image d 402 Input the multimodal fusion model 400. Here, the representation information z r 401 and information d Each of the 402 can be Figure 3 The single modal fusion model 300 shown is obtained.

[0090] The representation information z of the first modality image r 401 is input into the classification query vector processing module 411, wherein the classification query vector processing module 411 includes a parameter matrix In the classification query vector processing module 411, by r 401 and parameter matrix Perform matrix operations to obtain the classification fusion query vector q r .

[0091] The representation information z of the second modality image d 402 are respectively input into the classification key vector processing module 412 and the classification value vector processing module 413. The classification key vector processing module 412 includes the parameter matrix By expressing information z d 402 and parameter matrix Perform matrix operations to obtain the classification fusion key vector k d The classification value vector processing module 413 includes a parameter matrix By expressing information z d 402 and parameter matrix Perform matrix operations to obtain the classification fusion value vector v d .

[0092] By fusion query vector q r and the classification fusion key vector k d Performing the multiplication operation, the first correlation matrix A between the first modality image and the second modality image can be obtained. rd .

[0093] The first correlation matrix A rd After the softmax function calculation of the softmax module 420, it can be combined with the classification fusion value vector v d Perform matrix multiplication to obtain the first fusion information z of the first modality image and the second modality image rd 403.

[0094] Similarly, the representation information z of the first modality image is swapped r 401 and the representation information z of the second modality imaged 402 input port, which will indicate information z r 401 is input into the classification key vector processing module 412 and the classification value vector processing module 413, and the information z is represented d 402 is input into the classification query vector processing module 411, and the second fusion information z of the first modality image and the second modality image can be obtained. dr .

[0095] By taking the first fusion information z rd and the second fusion information z dr After splicing, the fusion information can be finally used to perform classification.

[0096] The above process can be expressed by the following formula:

[0097]

[0098]

[0099] z rd =A rd ·v d

[0100] Z=[z rd ||z dr ]

[0101] Among them, D represents the dimension of the feature information of each image block, h represents the number of heads in the multi-head attention mechanism, “||” represents the splicing operation, and Z represents the fusion information used to perform classification.

[0102] Figure 5 A flowchart of a training method for an image processing model according to an exemplary embodiment of the present disclosure is shown. The image processing model at least includes a modality correlation calculation module, a classification information update module and a classification prediction module. Figure 5As shown, the method 500 includes: step S501, dividing a sample image into a plurality of image blocks, wherein the sample image has a classification label; step S502, determining a feature sequence corresponding to the sample image, wherein the feature sequence includes feature information corresponding to each of the plurality of image blocks, modal information of the sample image, and classification information of the sample image; step S503, inputting the feature information and modal information corresponding to each of the plurality of image blocks in the feature sequence into a modal correlation calculation module to obtain the modal correlation of each of the plurality of image blocks; step S504, inputting the feature information, classification information corresponding to each of the plurality of image blocks in the feature sequence, and the modal correlation of each of the plurality of image blocks into a classification information update module to obtain updated classification information; step S505, inputting the classification information into a classification prediction module to obtain a first predicted classification of the sample image; and step S506, adjusting the parameters of the image processing model based on the classification label of the sample image and the first predicted classification.

[0103] Therefore, the modal correlation calculated during the training process can be used to assist in updating the classification information used to determine the first predicted classification. In other words, with the iteration of training, the image processing model can learn to optimize the classification information with the help of the reference information provided by the modal correlation, thereby improving the classification accuracy of the trained image processing model for images to be processed of different modalities.

[0104] With respect to step S501 , the image to be processed may be divided into a preset number of image blocks based on the configuration of the image processing model.

[0105] With respect to step S502 , the feature sequence of the sample image includes feature information corresponding to each of the plurality of image blocks, modality information of the sample image, and classification information of the sample image.

[0106] In particular, the feature sequence may further include position information of each image block and spectrum information of each image block.

[0107] For example, the feature sequence corresponding to the sample image of modality type i is It can be expressed by the following formula:

[0108]

[0109] Among them, x cls Represents the classification information of the sample image, Represents the modal information of the sample image, represents the feature information corresponding to each of the multiple image blocks, x pos Position information corresponding to each of the plurality of image blocks, Represents the spectrum information corresponding to each of the multiple image blocks. For sample images of different modality types, and The value of is different, “||” indicates a concatenation operation.

[0110] And among them, x cls and is a 1×D matrix, is an n×D matrix, x pos and It is a (n+2)×D matrix, where D is the dimension of the feature information of each image block and n is the number of image blocks.

[0111] Figure 6 A schematic diagram of a characteristic sequence according to an exemplary embodiment of the present disclosure is shown, Figure 6 As shown, the feature information 623 corresponding to each of the multiple image blocks is spliced ​​with the classification information 621 of the sample image and the modal information 622 of the sample image. The spliced ​​result is then added to the spectrum information 610 corresponding to each of the multiple image blocks and the position information 630 corresponding to each of the multiple image blocks to obtain the final feature sequence.

[0112] It is understandable that for ease of processing, such as Figure 6 As shown, corresponding spectrum information and position information may also be determined for the classification information 621 of the sample image and the modal information 622 of the sample image.

[0113] According to some embodiments, the image processing model further comprises a feature extraction module, and wherein determining the feature sequence corresponding to the sample image comprises: inputting each of the plurality of image blocks into the feature extraction module to obtain the feature sequence corresponding to the sample image.

[0114] Thus, the feature extraction module in the image processing model can automatically extract the feature sequence corresponding to the sample image, making the feature sequence a learnable parameter. It can be understood that with the iteration of training, the ability of the feature extraction module to extract feature sequences from sample images can be gradually enhanced, so that the extracted feature sequences can be better applied to the classification of sample images.

[0115] According to some embodiments, the image processing model further includes a single-modal fusion module, and the training method may further include: before inputting the feature information and modal information corresponding to each of the multiple image blocks in the feature sequence into the modal relevance calculation module, inputting the feature sequence into the single-modal fusion module to obtain an updated feature sequence, wherein the modal information and classification information in the updated feature sequence are both fused with the feature information corresponding to each of the multiple image blocks. In this way, through the single-modal fusion module, learnable modal information and classification information can be generated for subsequent calculations.

[0116] In one implementation, the single-modal fusion module may obtain updated modality information and classification information based on a self-attention mechanism.

[0117] Subsequently, steps S503 to S506 can be executed in sequence, and the parameters of the image processing model can be adjusted based on the obtained classification label and the first predicted classification. After training iterations, the modal correlation calculation module in the image processing model can learn the ability to obtain the modal correlation of each of the multiple image blocks from the feature information and modal information corresponding to each of the multiple image blocks; the classification information update module can learn the reference modal correlation to optimize the classification information, and ultimately the trained image processing model can achieve good classification prediction effects.

[0118] According to some embodiments, the sample image also has a modal label, the image processing model also includes a modal prediction module, and the training method may further include: inputting the modal information in the updated feature sequence into the modal prediction module to obtain the predicted modality of the sample image; and adjusting the parameters of the image processing model based on the modal label and predicted modality of the sample image.

[0119] During the training process, the reverse adjustment of the modal label and predicted modality based on the sample image can improve the accuracy of the modal information extracted by the image processing model. When the accuracy of the modal information is improved, the accuracy of the modal correlation calculated by the modal correlation calculation module based on the modal information can be guaranteed, thereby improving the accuracy of the classification prediction.

[0120] According to some embodiments, the sample image includes multiple modal images for the same object, the modal type of each of the multiple modal images is different, the image processing model also includes a multimodal fusion module, the structure of the multimodal fusion module and the single modal fusion module are the same, the training method may also include: inputting the feature sequence corresponding to each of the multiple modal images into the multimodal fusion module to obtain fusion information of the image to be processed; inputting the fusion information into the classification prediction module to obtain a second predicted classification of the sample image; and adjusting the parameters of the image processing model based on the classification label and the second predicted classification of the sample image, wherein the parameters of the multimodal fusion module and the single modal fusion module correspond to the same in the adjustment.

[0121] Therefore, the parameters of the unimodal module can be corrected based on the training results of the multimodal fusion information. In other words, during the training process, the parameters of the unimodal module can be optimized by adjusting the parameters of the multimodal fusion module based on the fusion information. At the same time, the calculation mode of the unimodal image inside the unimodal module can be maintained, so that the trained image processing model can still rely on the unimodal module to classify the unimodal images to be processed.

[0122] Fig. 7A and 7B A schematic diagram showing a training method for an image processing device according to an exemplary embodiment of the present disclosure is shown.

[0123] For the purpose of describing the method, only the first modality image and the second modality image are used for description. Those skilled in the art will appreciate that the processing can also be performed when the images to be processed include more than three modality images.

[0124] like Fig. 7A As shown, the plurality of image blocks 701 of the first modality image and the plurality of image blocks 702 of the second modality image are processed by the feature extraction module 710 respectively to obtain feature sequences corresponding to the plurality of image blocks 701 and feature sequences corresponding to the plurality of image blocks 702. The feature sequences corresponding to the plurality of image blocks 701 and the feature sequences corresponding to the plurality of image blocks 702 may be in the form of: Figure 6 As shown, no further description is given here.

[0125] The feature sequences corresponding to the multiple image blocks 701 and the feature sequences corresponding to the multiple image blocks 702 are sequentially input into the single-modality fusion module 720 and the classification information update module 730, wherein the classification information update module 730 includes the above-mentioned modality correlation calculation module and the multi-modality fusion module. After being processed by the single-modality fusion module 720 and the classification information update module 730, the first classification information and the first modality information corresponding to the multiple image blocks 701 of the first modality image can be obtained, that is, Fig. 7AThe second classification information and the second modality information corresponding to the plurality of image blocks 702 of the second modality image are, that is, Fig. 7A The sequence 705 in , and the fusion information of the first modality image and the second modality image, that is, Fig. 7A Sequence 704 in .

[0126] It can be understood that the whole structure composed of the single-modality fusion module 720 and the classification information update module 730 can have a multi-layer structure. For the convenience of explanation, Fig. 7A Meanwhile, for the multi-layer structure, at least one layer has both the single-mode fusion module 720 and the classification information update module 730 , and the other layers may only contain the single-mode fusion module 720 .

[0127] The first modality information in sequence 703 and the second modality information in sequence 705 are respectively input into the modality prediction module 742, and the predicted modality A (i.e., the above-mentioned predicted modality) for the first modality image and the predicted modality B (i.e., the above-mentioned predicted modality) for the second modality image can be obtained respectively.

[0128] Based on the difference between any one of the predicted modality A and the predicted modality B and the modality label, the parameters of the image processing device can be adjusted.

[0129] The first classification information in sequence 703, the second classification information in sequence 705, and the fusion information in sequence 704 are respectively input into the classification prediction module 741, and the predicted classification A (i.e., the above-mentioned first predicted classification) for the first modality image, the predicted classification B (i.e., the above-mentioned first predicted classification) for the second modality image, and the predicted classification C (i.e., the above-mentioned second predicted classification) for the fusion information of the first modality image and the second modality image can be obtained respectively.

[0130] Based on the difference between any one of the predicted classification A, the predicted classification B, and the predicted classification C and the classification label, the parameters of the image processing device can be adjusted.

[0131] It can be understood that when a sample image of only one modality is input, the above training can be performed based only on the sample image, for example, adjusting the parameters of the image processing device based on the predicted modality A and the predicted classification A. When sample images of multiple modalities are input, training can be performed not only based on sample images of each modality, but also based on the fusion results of sample images of multiple modalities, for example, predicted classification C.

[0132] Figure 7B The working mechanism between the single-modality fusion module 720 and the classification information updating module 730 is further illustrated.

[0133] like Figure 7BAs shown, the unimodal fusion module 720 includes a multi-head self-attention module MSA722, two layer normalization modules 721 and 723, and a multi-layer perception machine 724. Fig. 7A The classification information update module 730 further includes Figure 7B The classification update submodule 731 and the multimodal fusion module 732 are shown. The classification update submodule 731 includes a multi-headed modal attention module MMDA 731-2, two layer normalization modules 731-1 and 731-3, and a multi-layer perceptron 731-4. The multimodal fusion module 732 includes a multi-headed cross-modal attention module MCMA 732-2, two layer normalization modules 732-1 and 732-3, and a multi-layer perceptron 732-4.

[0134] The multi-head self-attention module MSA722 is a standard transformer module. The structure of the multi-head modal attention module MMDA731-2 is Figure 3 The structure of the single-modal fusion model 300 shown in FIG. 7 is similar. In the processing flow formed by the cascade of the single-modal fusion module 720 and the classification update submodule 731, the feature sequence of the sample image of each modality is processed separately. Figure 7B As shown, the feature sequence corresponding to the first modality image and the feature sequence corresponding to the second modality image are not fused in the processing flow, and are processed separately.

[0135] It can be understood that, for each feature sequence, the classification update submodule 731 can specifically update the classification information in the feature sequence, while other information in the feature sequence can remain unchanged.

[0136] The multimodal fusion module 732 is connected in parallel with the single-modal fusion module 720 and the classification update submodule 731. The multimodal fusion module 732 is used only when at least two modal sample images are input at the same time. By fusing the feature sequences of each of the at least two modal sample images, the multimodal fusion module 732 can output fusion information.

[0137] In the first layer of the multi-layer structure composed of the single-modal fusion module 720 and the classification information update module 730, the multi-modal fusion module 732 only receives the output of the layer normalization module 721 from the single-modal fusion module 720 as the input of the multi-head cross-modal attention module MCMA732-2. The processing flow and Figure 4 The processing flow of the multimodal fusion model 400 shown is similar and will not be repeated here.

[0138] As for other layers in the multi-layer structure formed by the single-modal fusion module 720 and the classification information update module 730, if there is input of fusion information in the upper layer, the fusion information of the upper layer participates in the calculation of the multi-modal fusion module 732 of the layer.

[0139] Exemplarily, the following formula may be used to represent the processing flow of the kth layer in the multi-layer structure composed of the single-modality fusion module 720 and the classification information updating module 730:

[0140]

[0141]

[0142]

[0143] in, represents the feature sequence of the sample image of modality type i at the kth layer, i∈{r,d} ... represents the fusion information of the sample image of modality type r and the sample image of modality type d of the k-th layer, that is, the output of the multimodal fusion module 732, represents the output of the single modality fusion module 720.

[0144] It can be understood that in the above formula, the influence of the layer normalization module 723 and the multi-layer perceptron 724 in the single-modal fusion module 720, the layer normalization module 731-3 and the multi-layer perceptron 731-4 in the classification update submodule 731, and the layer normalization module 732-3 and the multi-layer perceptron 732-4 in the multi-modal fusion module 732 are ignored. It is known to those skilled in the art that the processing can also be implemented in the case of including the above modules.

[0145] On this basis, the classification prediction module 741 Classification information in as well as Classification information in Perform layer normalization LN and multi-layer perception MLP respectively cls Processing, to obtain the corresponding prediction classification; the modal prediction module 742 Modal information in Perform layer normalization LN and multi-layer perception MLP mod Processing to obtain the corresponding prediction mode.

[0146] For example, the total loss function during training can be expressed as:

[0147]

[0148]

[0149]

[0150]

[0151] Among them, y cls represents the classification label, y mod represents the modal label, BCE represents the binary cross entropy calculation, express The corresponding classification loss function is, express The corresponding classification loss function is, express The corresponding modal loss function. Represents the total loss function for sample images of modality type i.

[0152] It should be noted that the images in any of the above embodiments are from public data sets and cannot reflect the personal information of a specific user.

[0153] Figure 8 A structural block diagram of an image processing device according to an exemplary embodiment of the present disclosure is shown. Figure 8 As shown, the image processing device 800 includes: a first division unit 801, configured to divide the image to be processed into multiple image blocks; a first determination unit 802, configured to determine the modal correlation of each of the multiple image blocks based on the feature information of each of the multiple image blocks, wherein the modal correlation of each image block represents the degree of correlation between the feature information of the image block and the modal type of the image to be processed; and a classification unit 803, configured to perform classification for the image to be processed based on the modal correlation of each of the multiple image blocks and using the feature information of at least part of the multiple image blocks.

[0154] According to some embodiments, the classification unit includes: a selection subunit configured to select one or more target image blocks from a plurality of image blocks based on the modal correlation of each of the plurality of image blocks; and a first classification subunit configured to perform classification for the image to be processed using feature information of each of the one or more target image blocks.

[0155] According to some embodiments, selecting the sub-unit includes: for each of the plurality of image blocks, in response to a modality correlation of the image block being less than a preset threshold, determining the image block as a sub-unit of the target image block.

[0156] According to some embodiments, the classification unit includes: a calculation subunit, configured to calculate the weighted sum of feature information of each of a plurality of image blocks, wherein the weight corresponding to each image block is negatively correlated with the modal relevance of the image block; and a second classification subunit, configured to perform classification for the image to be processed based on the weighted sum.

[0157] According to some embodiments, the image to be processed includes multiple modal images of the same object, and the modal type of each of the multiple modal images is different from each other, and wherein the classification unit includes: a determination subunit, configured to determine, for each of the multiple modal images, representation information of the modal image using feature information of at least part of the multiple image blocks in the modal image; an acquisition subunit, configured to obtain fusion information of the image to be processed by fusing the representation information of each of the multiple modal images; and a third classification subunit, configured to perform classification of the image to be processed based on the fusion information.

[0158] According to some embodiments, the apparatus further includes: a first fusion unit configured to update, for each of the multiple image blocks, feature information corresponding to the image block by fusing feature information of each of the multiple image blocks.

[0159] According to some embodiments, the feature information of each of the plurality of image blocks comprises spectrum information of the image block.

[0160] Fig. 9A structural block diagram of a training device for an image processing model according to an exemplary embodiment of the present disclosure is shown. The image processing model at least includes a modal correlation calculation module, a classification information update module and a classification prediction module. The training device 900 includes: a second division unit 901, configured to divide a sample image into a plurality of image blocks, wherein the sample image has a classification label; a second determination unit 902, configured to determine a feature sequence corresponding to the sample image, wherein the feature sequence includes feature information corresponding to each of the plurality of image blocks, modal information of the sample image and classification information of the sample image; a first acquisition unit 903, configured to obtain a plurality of image blocks in the feature sequence; The feature information and modal information corresponding to each of the multiple image blocks in the feature sequence are input into the modal correlation calculation module to obtain the modal correlation of each of the multiple image blocks; the second acquisition unit 904 is configured to input the feature information, classification information corresponding to each of the multiple image blocks in the feature sequence and the modal correlation of each of the multiple image blocks into the classification information update module to obtain updated classification information; the third acquisition unit 905 is configured to input the classification information into the classification prediction module to obtain a first predicted classification of the sample image; and the first adjustment unit 906 is configured to adjust the parameters of the image processing model based on the classification label of the sample image and the first predicted classification.

[0161] According to some embodiments, the image processing model further comprises a feature extraction module, and wherein the second determination unit comprises: a subunit for inputting each of the plurality of image blocks into the feature extraction module to obtain a feature sequence corresponding to the sample image.

[0162] According to some embodiments, the image processing model also includes a single modal fusion module, and the training device also includes: a second fusion unit, configured to input the feature sequence into the single modal fusion module to obtain an updated feature sequence, wherein the modal information and classification information in the updated feature sequence are fused with the feature information corresponding to each of the multiple image blocks.

[0163] According to some embodiments, the sample image also has a modal label, the image processing model also includes a modal prediction module, and the training device also includes: a fourth acquisition unit, configured to input the modal information in the updated feature sequence into the modal prediction module to obtain the predicted modality of the sample image; and a second adjustment unit, configured to adjust the parameters of the image processing model based on the modal label and predicted modality of the sample image.

[0164] According to some embodiments, the sample image includes multiple modal images for the same object, the modal type of each of the multiple modal images is different, the image processing model also includes a multimodal fusion module, the structure of the multimodal fusion module and the single modal fusion module are the same, the training device also includes: a third fusion unit, configured to input the feature sequence corresponding to each of the multiple modal images into the multimodal fusion module to obtain fusion information of the image to be processed; a fifth acquisition unit, configured to input the fusion information into the classification prediction module to obtain a second predicted classification of the sample image; and a third adjustment unit, configured to adjust the parameters of the image processing model based on the classification label and the second predicted classification of the sample image, wherein the parameters of the multimodal fusion module and the single modal fusion module correspond to the same in the adjustment.

[0165] According to some embodiments of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the above methods.

[0166] According to some embodiments of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to enable a computer to execute any one of the above methods.

[0167] According to some embodiments of the present disclosure, a computer program product is also provided, including a computer program, wherein the computer program implements any one of the above methods when executed by a processor.

[0168] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0169] refer to Fig.10 , a block diagram of an electronic device 1000 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0170] like Fig.10As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0171] Multiple components in the electronic device 1000 are connected to the I / O interface 1005, including: an input unit 1006, an output unit 1007, a storage unit 1008, and a communication unit 1009. The input unit 1006 can be any type of device that can input information to the electronic device 1000. The input unit 1006 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 1007 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1008 can include but is not limited to a disk and an optical disk. The communication unit 10010 allows the electronic device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device and / or the like.

[0172] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as image processing methods or model training methods. For example, in some embodiments, the image processing method or model training method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the image processing method or model training method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the image processing method or the model training method in any other suitable manner (for example, by means of firmware).

[0173] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0174] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0175] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0176] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0177] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0178] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0179] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0180] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. An image processing method, comprising: Divide the image to be processed into multiple image blocks; Determining a modality correlation of each of the plurality of image blocks based on feature information of each of the plurality of image blocks comprises: Extracting modal information of the image to be processed from a feature set formed by feature information of each of a plurality of image blocks of the image to be processed; Inputting the feature set into a modal key vector calculation module in a single modal fusion model based on an attention mechanism to obtain a modal key vector; Inputting the modality information into a modality query vector calculation module in the single modality fusion model to obtain a modality query vector; and Based on the modal key vector and the modal query vector, a modal relevance matrix is ​​obtained, wherein the modal relevance matrix includes the modal relevance of each of the plurality of image blocks, wherein the modal relevance of each image block represents the degree of relevance between feature information of the image block and the modal type of the image to be processed; and Based on the modal correlation of each of the plurality of image blocks, the feature information of at least part of the plurality of image blocks is used to perform classification for the image to be processed, including: selecting one or more target image blocks from the plurality of image blocks based on the modality correlation of each of the plurality of image blocks; and Classification of the image to be processed is performed using feature information of each of the one or more target image blocks.

2. The method according to claim 1, wherein: The selecting one or more target image blocks from the plurality of image blocks based on the modality correlation of each of the plurality of image blocks comprises: For each of the plurality of image blocks, in response to a modality correlation of the image block being less than a preset threshold, the image block is determined as a target image block.

3. The method according to claim 1, wherein: The performing classification of the image to be processed by using feature information of at least part of the image blocks in the plurality of image blocks based on the modal correlation of each of the plurality of image blocks comprises: Calculating a weighted sum of feature information of each of the plurality of image blocks, wherein a weight corresponding to each image block is negatively correlated with a modality correlation of the image block; and Based on the weighted sum, classification is performed on the image to be processed.

4. The method according to any one of claims 1 to 3, wherein: The images to be processed include multiple modality images of the same object, and the modality type of each of the multiple modality images is different from each other. And wherein, using the feature information of at least part of the plurality of image blocks to perform classification on the image to be processed comprises: For each of the plurality of modality images, determining representation information of the modality image using feature information of at least some of the plurality of image blocks in the modality image; Obtaining fusion information of the image to be processed by fusing the representation information of each of the multiple modality images; and Based on the fusion information, classification is performed on the image to be processed.

5. The method according to any one of claims 1 to 3, further comprising: Before determining the modal correlation of each of the multiple image blocks, for each of the multiple image blocks, feature information corresponding to the image block is updated by fusing feature information of each of the multiple image blocks.

6. The method according to any one of claims 1 to 3, wherein: The feature information of each of the plurality of image blocks includes spectrum information of the image block.

7. A training method for an image processing model, the image processing model comprising at least a modality correlation calculation module, a classification information update module and a classification prediction module, the training method comprising: Dividing a sample image into a plurality of image blocks, wherein the sample image has a classification label; Determining a feature sequence corresponding to the sample image, wherein the feature sequence includes feature information corresponding to each of the plurality of image blocks, modality information of the sample image, and classification information of the sample image; Inputting the feature information and the modality information corresponding to each of the plurality of image blocks in the feature sequence into the modality correlation calculation module to obtain the modality correlation of each of the plurality of image blocks; Inputting the feature information corresponding to each of the plurality of image blocks in the feature sequence, the classification information, and the modal relevance of each of the plurality of image blocks into the classification information updating module to obtain updated classification information; Inputting the classification information into the classification prediction module to obtain a first predicted classification of the sample image; and Based on the classification label of the sample image and the first predicted classification, parameters of the image processing model are adjusted.

8. The method according to claim 7, wherein: The image processing model further includes a feature extraction module, and wherein determining the feature sequence corresponding to the sample image includes: Each of the plurality of image blocks is input into the feature extraction module to obtain the feature sequence corresponding to the sample image.

9. The method according to claim 7 or 8, wherein: The image processing model further includes a single-modality fusion module, and the training method further includes: Before inputting the feature information and the modal information corresponding to each of the multiple image blocks in the feature sequence into the modal relevance calculation module, the feature sequence is input into the single modal fusion module to obtain an updated feature sequence, wherein the modal information and the classification information in the updated feature sequence are both fused with the feature information corresponding to each of the multiple image blocks.

10. The method according to claim 9, wherein: The sample image also has a modality label, the image processing model also includes a modality prediction module, and the training method further includes: Inputting the modality information in the updated feature sequence into the modality prediction module to obtain the predicted modality of the sample image; and Based on the modality label and the predicted modality of the sample image, parameters of the image processing model are adjusted.

11. The method according to claim 9, wherein: The sample image includes multiple modality images for the same object, and the modality type of each of the multiple modality images is different from each other. The image processing model also includes a multimodal fusion module, and the structure of the multimodal fusion module is the same as that of the single modality fusion module. The training method also includes: Inputting the feature sequence corresponding to each of the multiple modality images into the multimodality fusion module to obtain fusion information of the sample image; Inputting the fusion information into the classification prediction module to obtain a second predicted classification of the sample image; and Based on the classification label of the sample image and the second predicted classification, the parameters of the image processing model are adjusted, wherein in the adjustment, the parameters of the multimodal fusion module and the single modal fusion module correspond to the same.

12. An image processing device, comprising: A first dividing unit is configured to divide the image to be processed into a plurality of image blocks; The first determining unit is configured to determine the modality correlation of each of the plurality of image blocks based on the feature information of each of the plurality of image blocks, including: Extracting modal information of the image to be processed from a feature set formed by feature information of each of a plurality of image blocks of the image to be processed; Inputting the feature set into a modal key vector calculation module in a single modal fusion model based on an attention mechanism to obtain a modal key vector; Inputting the modality information into a modality query vector calculation module in the single modality fusion model to obtain a modality query vector; and Based on the modal key vector and the modal query vector, a modal relevance matrix is ​​obtained, wherein the modal relevance matrix includes the modal relevance of each of the plurality of image blocks, wherein the modal relevance of each image block represents the degree of relevance between feature information of the image block and the modal type of the image to be processed; and A classification unit is configured to perform classification for the image to be processed based on the modal correlation of each of the multiple image blocks and using feature information of at least some of the multiple image blocks, the classification unit comprising: a selection subunit configured to select one or more target image blocks from the plurality of image blocks based on a modality correlation of each of the plurality of image blocks; and The first classification subunit is configured to perform classification on the image to be processed by utilizing feature information of each of the one or more target image blocks.

13. The device according to claim 12, wherein: The selection subunit comprises: For each of the plurality of image blocks, in response to a modality correlation of the image block being less than a preset threshold, determining the image block as a subunit of the target image block.

14. The device according to claim 12, wherein: The taxonomic units include: a calculation subunit configured to calculate a weighted sum of feature information of each of the plurality of image blocks, wherein a weight corresponding to each image block is negatively correlated with a modality correlation of the image block; and The second classification subunit is configured to perform classification on the image to be processed based on the weighted sum.

15. The device according to any one of claims 12 to 14, wherein: The images to be processed include multiple modality images of the same object, and the modality types of each of the multiple modality images are different from each other. And wherein, the classification unit includes: a determination subunit configured to determine, for each of the plurality of modality images, representation information of the modality image by using feature information of at least part of the plurality of image blocks in the modality image; an acquisition subunit configured to obtain fusion information of the image to be processed by fusing representation information of each of the plurality of modality images; and The third classification subunit is configured to perform classification on the image to be processed based on the fusion information.

16. The device according to any one of claims 12 to 14, further comprising: The first fusion unit is configured to update the feature information corresponding to each of the multiple image blocks by fusing the feature information of each of the multiple image blocks.

17. The device according to any one of claims 12 to 14, wherein: The feature information of each of the plurality of image blocks includes spectrum information of the image block.

18. A training device for an image processing model, the image processing model comprising at least a modality correlation calculation module, a classification information update module and a classification prediction module, the training device comprising: A second dividing unit is configured to divide a sample image into a plurality of image blocks, wherein the sample image has a classification label; A second determining unit is configured to determine a feature sequence corresponding to the sample image, wherein the feature sequence includes feature information corresponding to each of the plurality of image blocks, modality information of the sample image, and classification information of the sample image; A first acquisition unit is configured to input feature information and the modality information corresponding to each of the plurality of image blocks in the feature sequence into the modality correlation calculation module to obtain a modality correlation of each of the plurality of image blocks; a second acquisition unit configured to input the feature information corresponding to each of the plurality of image blocks in the feature sequence, the classification information, and the modal relevance of each of the plurality of image blocks into the classification information updating module to obtain updated classification information; A third acquisition unit is configured to input the classification information into the classification prediction module to obtain a first predicted classification of the sample image; and The first adjustment unit is configured to adjust the parameters of the image processing model based on the classification label of the sample image and the first predicted classification.

19. The device according to claim 18, wherein: The image processing model further comprises a feature extraction module, and wherein the second determining unit comprises: The method is used for inputting each of the plurality of image blocks into the feature extraction module to obtain a subunit of the feature sequence corresponding to the sample image.

20. The device according to claim 18 or 19, wherein: The image processing model further includes a single-modality fusion module, and the training device further includes: The second fusion unit is configured to input the feature sequence into the single-modal fusion module to obtain an updated feature sequence, wherein the modal information and the classification information in the updated feature sequence are both fused with the feature information corresponding to each of the multiple image blocks.

21. The device according to claim 20, wherein: The sample image also has a modality label, the image processing model also includes a modality prediction module, and the training device also includes: a fourth acquisition unit, configured to input the modality information in the updated feature sequence into the modality prediction module to obtain a predicted modality of the sample image; and A second adjustment unit is configured to adjust parameters of the image processing model based on the modality label and the predicted modality of the sample image.

22. The device according to claim 20, wherein: The sample image includes multiple modality images for the same object, and the modality type of each of the multiple modality images is different from each other. The image processing model also includes a multimodal fusion module, and the structure of the multimodal fusion module is the same as that of the single modality fusion module. The training device also includes: A third fusion unit is configured to input the feature sequence corresponding to each of the multiple modality images into the multimodal fusion module to obtain fusion information of the sample image; a fifth acquisition unit, configured to input the fusion information into the classification prediction module to obtain a second predicted classification of the sample image; and The third adjustment unit is configured to adjust the parameters of the image processing model based on the classification label of the sample image and the second predicted classification, wherein in the adjustment, the parameters of the multimodal fusion module and the single modal fusion module correspond to the same.

23. An electronic device, comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-11.

25. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.