Image processing method, device and equipment, readable storage medium and program product

Abstract

The embodiment of the invention provides an image processing method, device and equipment, a readable storage medium and a program product, and can be applied to the fields of artificial intelligence, image classification and the like, and the method comprises the steps: extracting the object feature of each to-be-classified object in a plurality of to-be-classified objects included in a to-be-processed image; performing feature cross processing on the first object feature and the second object feature to obtain a cross attention matrix between the first object feature and the second object feature, and determining correlation data between the first object feature and the second object feature according to the cross attention matrix; determining a correlation data set of the first object feature according to correlation data between the first object feature and each object feature except the first object feature; and performing image classification processing according to each object feature and the corresponding correlation data set to obtain a classification result of the to-be-processed image. According to the embodiment of the invention, the accuracy and integrity of classifying the plurality of objects in the image can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, an image processing apparatus, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Multi-label classification is a common image classification task used to identify multiple objects in an image and assign corresponding categories to each object. Multi-label classification methods typically use convolutional neural networks (CNNs) to extract features from different objects in an image, and then use a classifier to classify the image features of different objects separately. Using these methods for multi-label classification results in low classification accuracy and is prone to misclassification and omissions. Therefore, improving the accuracy and completeness of classifying multiple objects in an image is an urgent problem that needs to be solved. Summary of the Invention

[0003] The present application provides an image processing method, apparatus and device, readable storage medium, and program product, which can improve the accuracy and completeness of classifying multiple objects in an image.

[0004] In one aspect, the present application provides an image processing method, the method comprising:

[0005] Acquire an image to be processed, and extract object features of each of a plurality of objects to be classified included in the image to be processed;

[0006] performing feature cross processing on the first object feature and the second object feature to obtain a cross attention matrix between the first object feature and the second object feature, and determining correlation data between the first object feature and the second object feature based on the cross attention matrix; the first object feature is any one of the plurality of object features, and the second object feature is any one of the plurality of object features other than the first object feature;

[0007] determining a correlation data set of the first object feature based on correlation data between the first object feature and each object feature among the plurality of object features except the first object feature;

[0008] After determining the correlation data set of each of the multiple object features, image classification processing is performed based on each of the object features and the correlation data set of each of the object features to obtain the classification result of the image to be processed; the classification result includes the predicted category of each of the objects to be classified.

[0009] In another aspect, the present application provides an image processing device, comprising:

[0010] A preprocessing module, configured to obtain an image to be processed and extract object features of each of a plurality of objects to be classified included in the image to be processed;

[0011] a correlation detection module, configured to perform feature cross processing on the first object feature and the second object feature to obtain a cross-attention matrix between the first object feature and the second object feature, and determine correlation data between the first object feature and the second object feature based on the cross-attention matrix; the first object feature is any one of the plurality of object features, and the second object feature is any one of the plurality of object features other than the first object feature;

[0012] The correlation detection module is further configured to determine a correlation data set of the first object feature based on correlation data between the first object feature and each object feature of the plurality of object features other than the first object feature;

[0013] A classification module is used to perform image classification processing based on each of the above-mentioned object features and the correlation data set of each of the above-mentioned object features after determining the correlation data set of each of the above-mentioned object features, so as to obtain the classification result of the above-mentioned image to be processed; the above-mentioned classification result includes the predicted category of each of the above-mentioned objects to be classified.

[0014] Accordingly, the present application provides a computer device comprising: a processor, a storage device and a communication interface, wherein the processor, the communication interface and the storage device are interconnected, wherein the storage device stores executable program code, and the processor is used to call the executable program code to implement the above-mentioned image processing method.

[0015] Accordingly, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the program instructions are executed by a processor to implement the image processing method as described above.

[0016] Accordingly, the present application provides a computer program product, which includes a computer program or computer instructions. The computer program or computer instructions are executed by a processor to implement the above-mentioned image processing method.

[0017] The embodiment of the present application extracts object features of each object to be classified from a plurality of objects to be classified included in an image to be processed, and for each object feature in the plurality of object features, performs feature cross-processing on the object feature with other object features to obtain a cross-attention matrix between the object feature and other object features, and determines correlation data between the object feature and other object features based on the cross-attention matrix between the object feature and other object features, and then integrates to obtain a correlation data set of the object feature. The above method can determine a correlation data set for each object feature in the plurality of object features; since each object feature represents the image features of the object to be classified itself, and the correlation data set of each object feature represents the association relationship between the object to be classified and other objects to be classified, then, performing image classification processing based on each object feature and the correlation data set of each object feature can improve the accuracy and completeness of classifying multiple objects in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 is a schematic diagram of the architecture of an image processing system provided by an exemplary embodiment of the present application;

[0020] Figure 2 is a flowchart of an image processing method provided by an exemplary embodiment of the present application;

[0021] Figure 3 is a flowchart of another image processing method provided by an exemplary embodiment of the present application;

[0022] Figure 4A This is a flowchart of a multi-label image classification process provided by an exemplary embodiment of the present application;

[0023] Figure 4B This is a flowchart of another multi-label image classification process provided by an exemplary embodiment of the present application;

[0024] Figure 5 is a structural diagram of an image processing device provided by an exemplary embodiment of the present application;

[0025] Figure 6 It is a structural diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0027] The embodiments of the present application can be applied to various fields or scenarios such as cloud technology, artificial intelligence, vehicle-mounted, image recognition, etc. Typical fields or scenarios will be introduced below.

[0028] Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or digital computer-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results. Artificial intelligence technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. The solution provided in the embodiment of this application mainly involves computer vision technology and machine learning technology under artificial intelligence technology, which will be described below:

[0029] Computer vision technology (CV) is a science that studies how to make machines "see". To put it more specifically, it means using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing so that the computer processing becomes an image that is more suitable for human eye observation or transmission to instruments for detection. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies. It also includes common biometric recognition technologies such as face recognition and fingerprint recognition. In an embodiment of the present application, the object features of each object to be classified in the image to be processed can be extracted by computer vision technology, and the correlation data of the image to be processed and the image classification processing are subsequently performed based on the accurately identified object features, which ensures the accuracy of image classification to a certain extent.

[0030] Machine Learning (ML) is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structures to continuously improve their own performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning generally include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching techniques. In an embodiment of the present application, the object features of each object to be classified in the image to be processed and the correlation data set of each object feature can be input into a machine learning model (such as a learnable target classification model) for image classification processing, thereby obtaining a classification result of the image to be processed.

[0031] The present application will be illustrated by the following examples.

[0032] See also Figure 1 , which is a schematic diagram of the architecture of an image processing system provided by an exemplary embodiment of the present application. The image processing system may specifically include a terminal device 101 and a server 102. The terminal device 101 and the server 102 are connected via a network, such as a local area network, a wide area network, or the mobile internet.

[0033] The terminal device 101 is also referred to as a terminal, user equipment (UE), access terminal, subscriber unit, mobile device, user terminal, wireless communication device, user agent, or user apparatus. The terminal device may be, but is not limited to, a smart home appliance, a handheld device with wireless communication capabilities (e.g., a smartphone or tablet), a computing device (e.g., a personal computer (PC)), an in-vehicle terminal, an intelligent voice interaction device, a wearable device, or other smart device.

[0034] Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0035] In one possible implementation, the server 102 may obtain an image to be processed from the terminal device 101, and extract object features of each object to be classified from a plurality of objects to be classified included in the image to be processed; the server 102 performs feature cross-processing on the first object feature and the second object feature to obtain a cross-attention matrix between the first object feature and the second object feature, and determines correlation data between the first object feature and the second object feature based on the cross-attention matrix, wherein the first object feature is any one of the plurality of object features, and the second object feature is any one of the plurality of object features except the first object feature; the server 102 then determines a correlation data set of the first object feature based on the correlation data between the first object feature and each object feature in the plurality of object features except the first object feature; after determining the correlation data set of each object feature in the plurality of object features, the server 102 performs image classification processing based on each object feature and the correlation data set of each object feature to obtain a classification result of the image to be processed (the classification result may include a predicted category of each object to be classified).

[0036] In one possible implementation, in a payment scenario (such as face-scanning payment or palm-scanning payment), the terminal device may refer to a payment device, which may include an image acquisition module for acquiring a target image after obtaining authorization from a target (such as a user), and then performing image classification processing on the target image using the method provided in an embodiment of the present application to obtain a classification result of the target image. Payment authorization is performed by combining the classification result and the target's identity authentication result, thereby further ensuring payment security.

[0037] In one possible implementation, the architecture of the image processing system proposed in this application may also include a database. The database may be a database located in the server, that is, it may be a database built into or brought by the server; the database may also be a database located in the terminal device, that is, it may be a database built into or brought by the terminal device; the database may also be an external database connected to the server, such as a cloud database (i.e., a database deployed in the cloud), which may be specifically based on any one of private cloud, public cloud, hybrid cloud, edge cloud, etc., so that the cloud database focuses on different functions. Then, the server can directly obtain the image to be processed from the database, and obtain the classification result of the image to be processed by processing the image to be processed.

[0038] It is understood that the system architecture diagram described in the embodiment of the present application is for the purpose of more clearly illustrating the technical solution of the embodiment of the present application and does not constitute a limitation on the technical solution provided by the embodiment of the present application. For example, the image processing method provided in the embodiment of the present application can be executed by the server 102 as well as by other servers or server clusters that are different from the server 102 and can communicate with the terminal device 101 and / or the server 102. It is known to those skilled in the art that Figure 1 The number of terminal devices and servers in the embodiment is merely illustrative. Any number of terminal devices and servers can be configured according to business implementation needs. Moreover, as the system architecture evolves and new business scenarios emerge, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems. In subsequent embodiments, the term "terminal device" will refer to the terminal device 101, and the term "server" will refer to the server 102. These will not be repeated in the subsequent embodiments.

[0039] See also Figure 2 , which is a flowchart of an image processing method provided by an exemplary embodiment of the present application, and is described by taking the application of the method to a server as an example, the method may include the following steps:

[0040] S201 : Acquire an image to be processed, and extract object features of each of a plurality of objects to be classified included in the image to be processed.

[0041] In this embodiment of the present application, the image to be processed may include multiple objects to be classified. These objects are the objects that need to be classified and identified within the image to be processed. For example, if the image to be processed is a photo of a soccer scene, the multiple objects to be classified may refer to each player, soccer ball, and other objects within the image to be processed. The server then extracts object features for each object to be classified within the image to be processed, providing a data foundation for subsequent feature cross-processing and image classification.

[0042] S202. Perform feature cross processing on the first object feature and the second object feature to obtain a cross attention matrix between the first object feature and the second object feature, and determine correlation data between the first object feature and the second object feature based on the cross attention matrix.

[0043] In one possible implementation, the server can first perform object detection on the image to be processed, determining the location information of each object to be classified in the image, such as obtaining the location boxes of each player and ball in the image. The server can then perform feature extraction on each detected object to be classified, obtaining object features for each object to be classified. By combining object detection and feature extraction to obtain object features for each object to be classified, the obtained object features can be guaranteed to be accurate.

[0044] The first object feature is any one of the multiple object features, and the second object feature is any one of the multiple object features other than the first object feature. For example, if the image to be processed includes three objects to be classified, and the server extracts object features for each of the objects to be classified (object feature 1, object feature 2, and object feature 3, respectively), then if the first object feature is object feature 1, the corresponding second object feature can be object feature 2 or object feature 3.

[0045] In one possible implementation, feature cross processing refers to the process of calculating the cross-attention matrix between the first object feature and the second object feature, and specifically refers to the process of modeling and characterizing the correlation (i.e., the association relationship) between the two object features. The cross-attention matrix between the first object feature and the second object feature can be obtained through feature cross processing. The cross-attention matrix refers to a matrix that measures the degree of cross-attention correlation between two object features, and the first object feature (or the second object feature) can include multiple specific features such as shape, color, and texture. Then each element in the cross-attention matrix (i.e., each attention data) can represent the degree of correlation between a specific feature in the first object feature and a specific feature in the second object feature, and the degree of correlation can be expressed as a probability value or a weight value.

[0046] In an embodiment of the present application, the server obtains a cross-attention matrix between the first object feature and the second object feature through feature cross processing, and then obtains correlation data between the first object feature and the second object feature based on the cross-attention matrix, thereby characterizing the association between the first object feature and the second object feature through the correlation data, and the correlation data can guide subsequent image classification processing, thereby improving the accuracy of image classification.

[0047] S203: Determine a correlation data set of the first object feature based on correlation data between the first object feature and each object feature other than the first object feature in the plurality of object features.

[0048] In an embodiment of the present application, the server can perform corresponding processing on the first object feature and each object feature of the multiple object features except the first object feature through the method provided in step S202 to obtain correlation data between the first object feature and each of the other object features, and obtain a correlation data set of the first object feature by integrating (or fusing) the multiple correlation data of the first object feature obtained above.

[0049] Exemplarily, the object features corresponding to the three objects to be classified are object feature 1, object feature 2 and object feature 3 respectively. Then, taking the first object feature being object feature 1 as an example, the server determines the correlation data between object feature 1 and object feature 2 (recorded as correlation data 1.1), and determines the correlation data between object feature 1 and object feature 3 (recorded as correlation data 1.2), and then integrates the correlation data 1.1 and the correlation data 1.2 to obtain the correlation data set of object feature 1 (the correlation data set includes correlation data 1.1 and correlation data 1.2).

[0050] It should be noted that, for each of the multiple object features, the server can use the above method to determine the correlation data set of each object feature. For example, the object features corresponding to the three objects to be classified are object feature 1, object feature 2, and object feature 3, respectively. Then, the correlation data set of object feature 1 includes correlation data 1.1 and correlation data 1.2; the correlation data set of object feature 2 includes correlation data 2.1 (correlation data between object feature 2 and object feature 1) and correlation data 2.2 (correlation data between object feature 2 and object feature 3); the correlation data set of object feature 3 includes correlation data 3.1 (correlation data between object feature 3 and object feature 1) and correlation data 3.2 (correlation data between object feature 3 and object feature 2).

[0051] It should be noted that the calculation of the above correlation data can be simplified. For example, correlation data 1.1 and correlation data 2.1 are both correlation data between object feature 1 and object feature 2. Then, the server can only calculate correlation data 1.1 without repeatedly calculating correlation data 2.1, thereby improving calculation efficiency.

[0052] S204: After determining the correlation data set of each object feature among the multiple object features, perform image classification processing according to each object feature and the correlation data set of each object feature to obtain a classification result of the image to be processed.

[0053] In an embodiment of the present application, the server performs image classification processing on the image to be processed based on the object features of each object to be classified in the image to be processed and the correlation data set of each object feature, thereby obtaining a classification result of the image to be processed. The classification result may include the predicted category of each object to be classified, thereby realizing multi-label classification of the image. Since the object features of each object to be classified represent the image features of the object to be classified itself, and the correlation data set of the object features of each object to be classified represents the association relationship between the object to be classified and other objects to be classified contained in the image to be processed, this makes the data used for image classification richer. By combining the object features and the correlation data set of each object feature, richer and more accurate guidance can be provided for the image classification task, thereby better understanding the semantics and content of the image to be processed and improving the accuracy of image classification.

[0054] Based on the above embodiments, the beneficial effect of the present application is that: the embodiment of the present application extracts the object features of each object to be classified from the multiple objects to be classified included in the image to be processed, and for each object feature in the multiple object features, the object feature is subjected to feature cross-processing with other object features to obtain a cross-attention matrix between the object feature and other object features, and the correlation data between the object feature and other object features is determined based on the cross-attention matrix between the object feature and other object features, and then the correlation data set of the object feature is integrated to obtain the correlation data set of the object feature. The above method can determine the correlation data set of each object feature in the multiple object features; since each object feature represents the image features of the object to be classified itself, and the correlation data set of each object feature represents the association relationship between the object to be classified and other objects to be classified, then, image classification processing is performed based on each object feature and the correlation data set of each object feature, which provides richer and more accurate guidance for the image classification task, and can improve the accuracy and completeness of classifying multiple objects in the image.

[0055] See also Figure 3 , which is a flowchart of another image processing method provided by an exemplary embodiment of the present application, and is described by taking the method applied to a terminal device as an example, the method may include the following steps:

[0056] S301 : Acquire an image to be processed, and extract object features of each of a plurality of objects to be classified included in the image to be processed.

[0057] In one possible implementation, the server can first use a target detection algorithm (such as Faster R-CNN, YOLO, SSD, etc.) to perform object detection on the image to be processed, and determine the position information of each object to be classified in the image to be processed, such as obtaining the position frame of each player and football in the image to be processed. Then, the server can perform feature extraction on each detected object to be classified to obtain the object features of each object to be classified, such as through feature extraction networks such as Convolutional Neural Networks (CNN), Residual Network (ResNet), and VGG network to extract features from the image area corresponding to the position frame of each object to be classified, and obtain the object features of each object to be classified. By combining the above-mentioned object detection and feature extraction to obtain the object features of each object to be classified, the accuracy of the obtained object features can be guaranteed.

[0058] In one possible implementation, the image to be processed may include multiple objects to be classified. The server can input the image to be processed into a target detector (such as a detector based on the YoloX algorithm) for target detection to obtain the position frame of each object to be classified, and crop the image area corresponding to the position frame of each object to be classified to obtain an image block (patch) corresponding to each object to be classified, such as target image block 1, target image block 2, and target image block 3 corresponding to the three targets in the image to be processed.

[0059] In one possible implementation, the server may input the obtained image blocks corresponding to each object to be classified into a feature encoder for processing to obtain the object features of each object to be classified (one image block is processed by the feature encoder to obtain one object feature). For example, the feature encoder may use three residual blocks, each of which may be composed of a convolutional layer, an activation function layer (ReLU layer), and a batch normalization layer. The specific structure is not limited in this embodiment of the present application.

[0060] The object features of the object to be classified can be recorded as embeddings L , the calculation formula is as follows:

[0061] embeddings L =F(patch L ) Formula 1

[0062] Among them, L is the identifier of the object feature of the object to be classified, patch L represents the object feature of the Lth object to be classified, F(patch L) means extracting the object features of the Lth object to be classified, embeddings L Represents the object features of the Lth object to be classified in the image to be processed.

[0063] The following describes in detail the method for determining the cross attention matrix between the first object feature and the second object feature through steps S302-S303:

[0064] S302: Mapping the first object features using the first parameter matrix to obtain a query vector matrix, and mapping the second object features using the second parameter matrix to obtain a key-value vector matrix.

[0065] The first object feature is any one of the plurality of object features, and the second object feature is any one of the plurality of object features except the first object feature.

[0066] In the embodiment of the present application, the first parameter matrix is ​​a matrix used to map the first object features. It maps the first object features into the query vector space through a linear transformation and is used to calculate the correlation between the first object features and the second object features. The second parameter matrix is ​​a matrix used to map the second object features. It also maps the second object features into the key-value vector space through a linear transformation and is used to calculate the correlation between the first object features and the second object features.

[0067] In one possible implementation, the first parameter matrix may refer to W Q Parameter matrix (i.e. query matrix), the second parameter matrix can refer to W K Parameter matrix (i.e. key value matrix), W Q and W K The first parameter matrix and the second parameter matrix are learnable parameter matrices. The first parameter matrix and the second parameter matrix can be dynamically learned through a training process. The first parameter matrix and the second parameter matrix can help map object features, thereby better capturing the relationship between features.

[0068] For example, the query vector matrix is ​​denoted as Q L , the key-value vector matrix is ​​recorded as K L , the calculation formulas are as follows:

[0069] Q L =W Q ×embeddings L Formula 2

[0070] K L =W K ×embeddings L Formula 3

[0071] The first object feature is recorded as embeddings1, and the query vector matrix obtained by the above formula is recorded as Q1. The second object feature is recorded as embeddings2, and the key-value vector matrix obtained by the above formula is recorded as K2.

[0072] S303. Determine a cross attention matrix between the first object feature and the second object feature according to the query vector matrix, the key-value vector matrix, and the vector distance between the query vector matrix and the key-value vector matrix.

[0073] Exemplarily, the cross attention matrix between the first object feature and the second object feature is recorded as cross_attention, and the calculation formula can be as follows:

[0074] cross_attention=softmax((Q1×K2 T ) / d) Formula 4

[0075] Among them, K2 T represents the transposed matrix of K2, d represents the key-value vector matrix and the vector distance between the query vector matrix and the key-value vector matrix, where the vector distance may refer to the cosine distance between Q1 and K2.

[0076] The following describes in detail the method for determining the correlation data between the first object feature and the second object feature through steps S304-S306:

[0077] S304. Sort the multiple attention data included in the cross-attention matrix in descending order of the numerical values ​​corresponding to the attention data in the cross-attention matrix to obtain a sorting result.

[0078] S305. Select the top M attention data from the sorting results; M is a positive integer.

[0079] In the above steps S304-S305, each cross-attention matrix includes multiple attention data, and the attention data can refer to attention scores. Each attention score is used to represent the degree of association between a specific feature in the first object feature and a specific feature in the second object feature. The degree of association can be expressed as a probability or a weight value.

[0080] For example, the query vector matrix Q1 is a 2*2 matrix, and the key-value vector matrix K2 is a 2*3 matrix. Then the cross-attention matrix between the first object feature and the second object feature is a 2*3 matrix, that is, the cross-attention matrix includes 6 attention data. The 6 attention data are sorted in descending order, and the M attention data ranked in the top M positions are selected from the sorting results, that is, the M attention data with the largest values ​​are selected from the 6 attention data.

[0081] It should be noted that M can be a small positive integer, such as 1 or 2. This is because a small M means that a small amount of attention data with the largest values ​​is selected, thereby focusing on feature pairs with high correlation between the first object feature and the second object feature. This helps improve the accuracy of subsequent image classification based on correlation data. At the same time, by only selecting a small amount of attention data with the largest values, the subsequent computational effort can be reduced, helping to improve the efficiency of image classification.

[0082] S306. Determine the correlation data between the first object feature and the second object feature based on the selected M attention data.

[0083] In one possible implementation, the server may determine the M selected attention data as the correlation data between the first object feature and the second object feature. For example, if M is 1, the server selects the maximum attention data from the cross-attention matrix between the first object feature and the second object feature, and then uses the maximum attention data as the correlation data between the first object feature and the second object feature.

[0084] In a possible implementation, step S306 may be implemented as follows:

[0085] (1) Determine the position information of the target attention data in the cross-attention matrix, and determine the position encoding data of the target attention data according to the position information; the target attention data is any one of the M selected attention data.

[0086] In an embodiment of the present application, the position information of the target attention data in the cross-attention matrix can be used to indicate the specific position of the target attention data in the cross-attention matrix. For example, the position information can be used to indicate the row and column of the target attention data in the cross-attention matrix. The server can further obtain position coding data of the target attention data through the position information. The position coding data is used to mark the spatial position of the target attention data, so as to characterize the relationship between the attention data at different positions through the position coding data, which helps to provide a more comprehensive and detailed feature representation.

[0087] For example, the position encoding data may refer to position encoding (PE), and the calculation formula of the position encoding may be as follows:

[0088]

[0089]

[0090] Among them, d modleRepresents the sequence length, such as the length of each row in the cross-attention matrix. p represents the sequence index of the attention data in the cross-attention matrix, such as the position of the attention data in a row in the cross-attention matrix. The position encoding of the attention data with even digits is calculated using the sin() function, and the position encoding of the attention data with odd digits is calculated using the cos() function.

[0091] (2) The target attention data and the position encoding data of the target attention data are fused to obtain the fused data of the target attention data.

[0092] In an embodiment of the present application, the server performs fusion processing on the target attention data and the position coding data of the target attention data to obtain fused data of the target attention data. The fused data combines the attention data and its position coding information to provide a richer feature representation.

[0093] In one possible implementation, the fusion process may refer to a splicing process, where the server splices the target attention data with the corresponding position-encoded data to obtain fused data of the target attention data. Similarly, through the above method, the server can obtain fused data of each of the M selected attention data.

[0094] For example, taking M as 1, we select the attention data with the largest value, record this attention data as embeddings_top1, and record the fusion data of the attention data as correlation_matrix. The calculation formula can be as follows:

[0095] correlation_matrix=cat(cross_attention,PE(embeddings_top1)) Equation 7

[0096] Among them, cat() represents the splicing process, cross_attention represents the current object feature, such as the first object feature, and PE(embeddings_top1) represents the position encoding data of the attention data embeddings_top1.

[0097] (3) Determine the correlation data between the first object feature and the second object feature based on the fusion data of each of the M selected attention data.

[0098] In an embodiment of the present application, the server may integrate the fusion data of each of the M selected attention data to obtain correlation data between the first object feature and the second object feature.

[0099] Exemplarily, the cross-attention matrix is ​​a 2*3 matrix, which includes a total of 6 attention data, namely: 0.11, 0.67, 0.42, 0.51, 0.3 and 0.41. Then, when M is 1, the server splices 0.67 and the corresponding position coding data, and uses the splicing result as the correlation data between the first object feature and the second object feature. When M is 2, the server splices 0.67 and the corresponding position coding data to obtain a first splicing result, and splices 0.51 and the corresponding position coding data to obtain a second splicing result, and then splices the first splicing result and the second splicing result to obtain a final splicing result, and uses the final splicing result as the correlation data between the first object feature and the second object feature. When M is greater than 2, the specific processing process can be found in the above description, which will not be repeated here.

[0100] S307: Determine a correlation data set of the first object feature based on correlation data between the first object feature and each object feature in the plurality of object features except the first object feature.

[0101] In one possible implementation, for ease of explanation, take M as 1 and the number of object features as 4 as an example. When the correlation data does not include the position coding data of the selected attention data, the correlation data set of the first object feature determined by the server includes: 1 attention data selected from the 3 cross-attention matrices between the first object feature and the other 3 object features (a total of 3 attention data). When the correlation data includes the position coding data of the selected attention data, the correlation data set of the first object feature determined by the server includes: 1 attention data selected from the 3 cross-attention matrices between the first object feature and the other 3 object features (a total of 3 attention data) and the position coding information of each selected attention data (a total of 3 position coding information, the position coding information corresponds one to one with the attention data).

[0102] S308 : After determining the correlation data set of each object feature among the multiple object features, perform image classification processing according to each object feature and the correlation data set of each object feature to obtain a classification result of the image to be processed.

[0103] In the embodiment of the present application, the server can use the above method to obtain a correlation data set for each object feature to characterize the relationship between the object feature and other object features. By combining the object features and the correlation data set of each object feature to perform image classification processing, this provides richer and more accurate guidance for the image classification task, thereby improving the accuracy of image classification. The classification result includes the predicted category of each object to be classified.

[0104] In a possible implementation, the server may further perform the following steps:

[0105] (1) Evaluate the image complexity of the image to be processed, and determine a first weight parameter according to the image complexity of the image to be processed.

[0106] In one possible implementation, image complexity may refer to the feature complexity of the image to be processed. For example, the image complexity of the image to be processed may be obtained by evaluating feature dimensions such as content, texture, and color distribution of the image to be processed. The server may determine a first weight parameter based on the image complexity of the image to be processed. The first weight parameter is one of the parameters used to calculate the amount of attention data M and may affect the size of M.

[0107] This is because the relationship between target objects in an image with higher complexity is generally more complex, so increasing M may be considered, while the relationship between target objects in an image with lower complexity is generally simpler, so decreasing M may be considered.

[0108] (2) Obtain processing performance data of the image processing device, and determine a second weight parameter based on the processing performance data of the image processing device.

[0109] In one possible implementation, the image processing device may refer to a device that performs image processing tasks on an image to be processed, such as a server, a computer, or a GPU. The processing performance data of the image processing device may refer to performance parameters of the image processing device, such as computing speed and parallel processing capability. The server may determine a second weight parameter based on the processing performance data of the image processing device. The second weight parameter is another parameter used to calculate the amount of attention data M and can affect the size of M.

[0110] This is because if the processing performance of the image processing device is poor, then M can be reduced (such as setting M to 1) to reduce the computing load of the image processing device, thereby ensuring efficient image processing. If the processing performance of the image processing device is high, then M can be increased (such as setting M to 3) to fully utilize the performance of the image processing device, which can greatly improve the accuracy of image processing.

[0111] (3) Determine the number M of attention data selected from the cross attention matrix based on the first weight parameter and the second weight parameter.

[0112] For example, the greater the image complexity of the image to be processed, the larger the first weight parameter is, the better the processing performance indicated by the processing performance data of the image processing device is, the larger the second weight parameter is, and accordingly, the larger the number M of selected attention data is.

[0113] In an embodiment of the present application, the server determines the number M of attention data selected from the cross-attention matrix based on the first weight parameter and the second weight parameter, so as to determine the appropriate number M of attention data in combination with the image complexity and the performance of the processing device, and better balance the image processing accuracy and image processing efficiency while ensuring the accuracy of image processing.

[0114] It should be noted that the server may also determine weight parameters corresponding to other evaluation dimensions, and determine the amount M of selected attention data by combining the first weight parameter and the second weight parameter with the weight parameters corresponding to the other evaluation dimensions, thereby avoiding the singleness of the evaluation dimensions and improving the accuracy of the determined amount M. Other evaluation dimensions may include: image processing time limit requirements, image scenes to be processed, etc.

[0115] In a possible implementation, step S308 may be implemented as follows:

[0116] (1) Inputting each object feature and the correlation data set of each object feature into the target classification model for processing to obtain prediction information of the image to be processed; the prediction information of the image to be processed includes the category prediction probability of each object to be classified.

[0117] (2) Determine the classification result of the image to be processed based on the category prediction probability of each object to be classified.

[0118] In one possible implementation, the predicted category probability for any object to be classified can include the predicted probabilities of K preset categories (K is a positive integer, such as 1000, representing 1000 common categories). For example, for a "soccer" goal in an image to be processed, K predicted probabilities can be obtained. The server can use the preset category corresponding to the largest predicted probability among the K predicted probabilities as the predicted category for the object to be classified. Similarly, each object to be classified will receive a corresponding predicted category, resulting in the final classification result.

[0119] In this embodiment of the present application, the server uses a pre-trained target classification model to perform image classification processing on each object feature and the data set related to each object feature, eliminating the need for manual intervention and significantly improving image classification efficiency. Furthermore, the target classification model can efficiently and accurately understand each object feature and the data set related to each object feature, ensuring the accuracy of the classification results.

[0120] For example, taking one of the multiple object features as an example, the object feature and the correlation data set of the object feature can be used as a set of object feature data, and the object feature data is recorded as patch_embedding_final. The calculation formula can be as follows:

[0121] patch_embedding_final=cat(embeddings,correlation_matrix_comb)Equation 8

[0122] Among them, embeddings represents object features, and correlation_matrix_comb represents a correlation data set of the object features (including multiple correlation data correlation_matrix).

[0123] For example, the target classification model may be a classifier, and the classifier may adopt a transformer structure. The processing process of the target classification model may be as follows:

[0124] labels=classifier(correlation_matrix_comb) equation 9

[0125] Among them, labels is the classification label, that is, the classification result of the image to be processed, and correlation_matrix_comb here can refer to the object feature data corresponding to each object feature.

[0126] In one possible implementation, the server may train the initial classification model to obtain a target classification model. Based on this, the server may further perform the following steps:

[0127] (1) Obtain a sample image and annotation information of the sample image; the annotation information includes the annotation categories of multiple target objects in the sample image.

[0128] (2) Extracting the object features of each target object included in the sample image, and determining a correlation data set of the object features of each target object based on the object features of each target object.

[0129] Among them, the specific implementation process of step (2) can refer to the relevant description in the aforementioned embodiment (such as steps S202, S203 and related steps). The method of calculating the correlation data set of object features during the training process is consistent with the method of calculating the correlation data set of object features during the application process, and the embodiment of this application will not elaborate on this.

[0130] (3) Inputting the object features of each target object and the correlation data set of the object features of each target object into the initial classification model for processing to obtain the prediction information of the sample image; the prediction information of the sample image includes the category prediction probability of each target object.

[0131] The specific implementation of step (3) is described in the above embodiments and will not be repeated here.

[0132] (4) Determine the category difference data of each target object based on the labeled category and category prediction probability of each target object, and determine the target difference data based on the category difference data of multiple target objects.

[0133] In one possible implementation, for each target object, the server determines category difference data between the labeled category of the target object and the predicted category probability (i.e., the category difference data of the target object). The category difference data may refer to a loss function, such as a squared loss function, an absolute value loss function, a logarithmic loss function, an exponential loss function, a cross-entropy loss function, etc. The category difference data of multiple target objects are then summed to obtain target difference data (e.g., a total loss function).

[0134] (5) Using the target difference data, the model parameters of the initial classification model are adjusted, and the target classification model is determined based on the initial classification model after the model parameters are adjusted.

[0135] In one possible implementation, the server can use the target difference data to adjust the model parameters of the initial classification model, and optimize the initial classification model by minimizing the total loss function (such as using a gradient descent algorithm to minimize the cost function to obtain the optimal parameters of the model).

[0136] It should be noted that the server can optimize the initial classification model after adjusting the model parameters multiple times, such as using multiple sample images to adjust the model parameters in sequence. When the training stop condition is met, the model with the adjusted parameters at this time is used as the target classification model. The training stop condition can be when the current round exceeds the preset training round number or when the model accuracy meets the accuracy requirement. The classification accuracy of the target classification model obtained through multiple rounds of iterative training will be greatly improved, so that the target classification model can better identify target objects of different categories, improve classification accuracy and robustness, and adapt to various different image classification tasks.

[0137] The image processing method proposed in the embodiment of the present application will be specifically described below through examples. Figure 4A As shown, Figure 4AThis is a flowchart of a multi-label image classification process provided by an exemplary embodiment of the present application. First, a target image (such as an image to be processed, which includes multiple targets, i.e., objects to be classified) is acquired. Target detection processing is then performed on the target image to obtain multiple target image blocks (including image blocks corresponding to each object to be classified). Feature extraction processing is then performed on each of the multiple target image blocks to obtain object features of each target image block, and feature relationships between different targets are constructed (i.e., a correlation data set of each object feature is obtained). The object features of each target image block and the feature relationships between different targets are then input into a classifier (such as a target classification model) for processing, and finally, the classification result of the image to be processed is output.

[0138] The above method will be further explained below. Figure 4B As shown, Figure 4B This is a flowchart of another multi-label image classification process provided by an exemplary embodiment of the present application. The image to be processed includes multiple objects to be classified. The server can input the target image into a target detector (such as a target detector based on the YoloX algorithm) for target detection, obtain the position frame of each object to be classified in the target image, and crop the image area corresponding to the position frame of each object to be classified to obtain the image patch corresponding to each object to be classified, such as Figure 4B Image block 1, image block 2, and image block 3 in .

[0139] In one possible implementation, Figure 4B As shown, the server can input the obtained image blocks corresponding to each object to be classified into a feature encoder for processing to obtain the features of each object to be classified (the features are the object features in the aforementioned embodiment, and one image block is processed by the feature encoder to obtain an object feature). At the same time, the correlation data set of each feature is obtained through the method provided by the aforementioned embodiment, and the features of each object to be classified (such as feature 1, feature 2, feature 3) and the correlation data set of each feature are input into a classifier (such as a target classification model) for processing to obtain the predicted category of each object to be classified (such as predicted category 1 corresponding to feature 1, predicted category 2 corresponding to feature 2, and predicted category 3 corresponding to feature 3).

[0140] It should be noted that in addition to pre-training the classifier (such as the target classification model), the server can also pre-train the target detector and feature encoder separately, thereby improving the accuracy of the output results of the target detector and the feature encoder, thereby ensuring the accuracy of image classification. The embodiments of this application do not limit the pre-training method of the target detector and feature encoder.

[0141] It should be noted that the method provided in the embodiment of the present application can be applied not only to the image multi-label classification task, but also to the text multi-label classification task. For example, the server first obtains the text to be processed and extracts multiple text features of the text to be processed; at the same time, the server performs feature cross-processing on the multiple text features to obtain the cross-attention matrix between the various text features, and then determines the correlation data set corresponding to each text feature according to the cross-attention matrix; after determining the correlation data set of each text feature in the multiple text features, the server performs text multi-label classification processing according to each text feature and the correlation data set of each text feature to obtain the classification result of the text to be processed; the classification result includes multiple predicted categories of the text to be processed, such as an article is classified into two predicted categories of economy and entrepreneurship through text multi-label classification, indicating that the article is an economic and entrepreneurial article. The specific implementation process can refer to the relevant statements in the aforementioned embodiment, and the embodiment of the present application will not elaborate on this. The above method ensures the accuracy of text multi-label classification.

[0142] The beneficial effect of the present application is that: the present application proposes a multi-label image classification method based on multi-target relationship modeling, which solves the problems of classification errors (i.e., misclassification of labels) and classification omissions (i.e., missed classification of labels) by introducing guidance on the relationships between different labels, and greatly improves the accuracy and completeness of classifying multiple objects in an image.

[0143] See also Figure 5 , which is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device may specifically include:

[0144] A pre-processing module 501 is configured to obtain an image to be processed and extract object features of each of a plurality of objects to be classified included in the image to be processed;

[0145] a correlation detection module 502 configured to perform feature cross processing on the first object feature and the second object feature to obtain a cross-attention matrix between the first object feature and the second object feature, and determine correlation data between the first object feature and the second object feature based on the cross-attention matrix; the first object feature is any one of the plurality of object features, and the second object feature is any one of the plurality of object features other than the first object feature;

[0146] The correlation detection module 502 is further configured to determine a correlation data set of the first object feature based on correlation data between the first object feature and each object feature other than the first object feature in the plurality of object features;

[0147] The classification module 503 is used to perform image classification processing based on each of the above-mentioned object features and the correlation data set of each of the above-mentioned object features after determining the correlation data set of each of the above-mentioned object features, so as to obtain the classification result of the above-mentioned image to be processed; the above-mentioned classification result includes the predicted category of each of the above-mentioned objects to be classified.

[0148] In one possible implementation, the cross-attention matrix includes a plurality of attention data; and the correlation detection module 502, when used to determine the correlation data between the first object feature and the second object feature according to the cross-attention matrix, is specifically configured to:

[0149] Sort the multiple attention data included in the cross-attention matrix according to the numerical values ​​corresponding to the attention data from high to low to obtain a sorting result;

[0150] Select the top M attention data from the above sorting results; M is a positive integer;

[0151] Determine the correlation data between the first object feature and the second object feature based on the selected M attention data.

[0152] In one possible implementation, when the correlation detection module 502 is used to determine the correlation data between the first object feature and the second object feature based on the selected M attention data, it is specifically used to:

[0153] Determine the position information of the target attention data in the cross attention matrix, and determine the position encoding data of the target attention data according to the position information; the target attention data is any one of the M selected attention data;

[0154] fusing the target attention data and the position encoding data of the target attention data to obtain fused data of the target attention data;

[0155] According to the fusion data of each attention data among the M attention data selected above, the correlation data between the above-mentioned first object feature and the above-mentioned second object feature is determined.

[0156] In a possible implementation, the correlation detection module 502 is further configured to:

[0157] Evaluating the image complexity of the image to be processed, and determining a first weight parameter according to the image complexity of the image to be processed;

[0158] Acquiring processing performance data of an image processing device, and determining a second weight parameter based on the processing performance data of the image processing device;

[0159] The number M of attention data selected from the above-mentioned cross-attention matrix is ​​determined according to the above-mentioned first weight parameter and the above-mentioned second weight parameter.

[0160] In one possible implementation, when the correlation detection module 502 is used to perform feature cross processing on the first object feature and the second object feature to obtain a cross attention matrix between the first object feature and the second object feature, it is specifically used to:

[0161] Mapping the first object features using the first parameter matrix to obtain a query vector matrix, and mapping the second object features using the second parameter matrix to obtain a key-value vector matrix;

[0162] A cross-attention matrix between the first object feature and the second object feature is determined based on the query vector matrix, the key-value vector matrix, and a vector distance between the query vector matrix and the key-value vector matrix.

[0163] In one possible implementation, when the classification module 503 is used to perform image classification processing based on the object features and the correlation data set of the object features to obtain the classification result of the image to be processed, it is specifically used to:

[0164] Inputting each of the above-mentioned object features and the correlation data set of each of the above-mentioned object features into the target classification model for processing to obtain prediction information of the above-mentioned image to be processed; the prediction information of the above-mentioned image to be processed includes the category prediction probability of each of the above-mentioned objects to be classified;

[0165] The classification result of the above-mentioned image to be processed is determined according to the category prediction probability of each of the above-mentioned objects to be classified.

[0166] In a possible implementation, the classification module 503 is further configured to:

[0167] Acquire a sample image and annotation information of the sample image; the annotation information includes the annotation categories of multiple target objects in the sample image;

[0168] Extracting object features of each of the target objects included in the sample image, and determining a correlation data set of the object features of each of the target objects based on the object features of each of the target objects;

[0169] Inputting the object features of each of the target objects and the correlation data set of the object features of each of the target objects into the initial classification model for processing to obtain prediction information of the sample image; the prediction information of the sample image includes the category prediction probability of each of the target objects;

[0170] Determining category difference data for each of the target objects based on the labeled category and category prediction probability of each of the target objects, and determining target difference data based on the category difference data for the plurality of target objects;

[0171] The target difference data is used to adjust the model parameters of the initial classification model, and the target classification model is determined based on the initial classification model after the model parameters are adjusted.

[0172] It should be noted that the functions of the various functional modules of the image processing device in the embodiment of the present application can be specifically implemented according to the method in the above method embodiment. The specific implementation process can refer to the relevant description of the above method embodiment and will not be repeated here.

[0173] See also Figure 6 , which is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 6 The computer device in the embodiment shown may include: a processor 601, a storage device 602, and a communication interface 603. The processor 601, the storage device 602, and the communication interface 603 may exchange data.

[0174] The above-mentioned storage device 602 may include a volatile memory (volatile memory), such as a random-access memory (RAM); the storage device 602 may also include a non-volatile memory (non-volatile memory), such as a flash memory (flash memory), a solid-state drive (SSD), etc.; the above-mentioned storage device 602 may also include a combination of the above-mentioned types of memory.

[0175] The processor 601 may be a central processing unit (CPU). In one embodiment, the processor 601 may also be a graphics processing unit (GPU). The processor 601 may also be a combination of a CPU and a GPU. In one possible implementation, the storage device 602 is used to store program instructions, and the processor 601 may call the program instructions to perform the following operations:

[0176] Acquire an image to be processed, and extract object features of each of a plurality of objects to be classified included in the image to be processed;

[0177] performing feature cross processing on the first object feature and the second object feature to obtain a cross attention matrix between the first object feature and the second object feature, and determining correlation data between the first object feature and the second object feature based on the cross attention matrix; the first object feature is any one of the plurality of object features, and the second object feature is any one of the plurality of object features other than the first object feature;

[0178] determining a correlation data set of the first object feature based on correlation data between the first object feature and each object feature among the plurality of object features except the first object feature;

[0179] After determining the correlation data set of each of the multiple object features, image classification processing is performed based on each of the object features and the correlation data set of each of the object features to obtain the classification result of the image to be processed; the classification result includes the predicted category of each of the objects to be classified.

[0180] In one possible implementation, the cross-attention matrix includes a plurality of attention data; and when the processor 601 is used to determine the correlation data between the first object feature and the second object feature according to the cross-attention matrix, the processor 601 is specifically used to:

[0181] Sort the multiple attention data included in the cross-attention matrix according to the numerical values ​​corresponding to the attention data from high to low to obtain a sorting result;

[0182] Select the top M attention data from the above sorting results; M is a positive integer;

[0183] Determine the correlation data between the first object feature and the second object feature based on the selected M attention data.

[0184] In one possible implementation, when the processor 601 is used to determine the correlation data between the first object feature and the second object feature based on the selected M attention data, it is specifically used to:

[0185] Determine the position information of the target attention data in the cross attention matrix, and determine the position encoding data of the target attention data according to the position information; the target attention data is any one of the M selected attention data;

[0186] fusing the target attention data and the position encoding data of the target attention data to obtain fused data of the target attention data;

[0187] According to the fusion data of each attention data among the M attention data selected above, the correlation data between the above-mentioned first object feature and the above-mentioned second object feature is determined.

[0188] In one possible implementation, the processor 601 is further configured to:

[0189] Evaluating the image complexity of the image to be processed, and determining a first weight parameter according to the image complexity of the image to be processed;

[0190] Acquiring processing performance data of an image processing device, and determining a second weight parameter based on the processing performance data of the image processing device;

[0191] The number M of attention data selected from the above-mentioned cross-attention matrix is ​​determined according to the above-mentioned first weight parameter and the above-mentioned second weight parameter.

[0192] In one possible implementation, when the processor 601 is used to perform feature cross processing on the first object feature and the second object feature to obtain a cross attention matrix between the first object feature and the second object feature, it is specifically used to:

[0193] Mapping the first object features using the first parameter matrix to obtain a query vector matrix, and mapping the second object features using the second parameter matrix to obtain a key-value vector matrix;

[0194] A cross-attention matrix between the first object feature and the second object feature is determined based on the query vector matrix, the key-value vector matrix, and a vector distance between the query vector matrix and the key-value vector matrix.

[0195] In one possible implementation, when the processor 601 is configured to perform image classification processing based on the object features and the correlation data set of the object features to obtain the classification result of the image to be processed, the processor 601 is specifically configured to:

[0196] Inputting each of the above-mentioned object features and the correlation data set of each of the above-mentioned object features into the target classification model for processing to obtain prediction information of the above-mentioned image to be processed; the prediction information of the above-mentioned image to be processed includes the category prediction probability of each of the above-mentioned objects to be classified;

[0197] The classification result of the above-mentioned image to be processed is determined according to the category prediction probability of each of the above-mentioned objects to be classified.

[0198] In one possible implementation, the processor 601 is further configured to:

[0199] Acquire a sample image and annotation information of the sample image; the annotation information includes the annotation categories of multiple target objects in the sample image;

[0200] Extracting object features of each of the target objects included in the sample image, and determining a correlation data set of the object features of each of the target objects based on the object features of each of the target objects;

[0201] Inputting the object features of each of the target objects and the correlation data set of the object features of each of the target objects into the initial classification model for processing to obtain prediction information of the sample image; the prediction information of the sample image includes the category prediction probability of each of the target objects;

[0202] Determining category difference data for each of the target objects based on the labeled category and category prediction probability of each of the target objects, and determining target difference data based on the category difference data for the plurality of target objects;

[0203] The target difference data is used to adjust the model parameters of the initial classification model, and the target classification model is determined based on the initial classification model after the model parameters are adjusted.

[0204] In a specific implementation, the processor 601, storage device 602 and communication interface 603 described in the embodiments of the present application can execute the aforementioned embodiments of the present application. Figure 2 or Figure 3 The implementation described in the relevant embodiments of the provided image processing method can also be performed in the embodiments of this application Figure 5 The implementation methods described in the relevant embodiments of the provided image processing device will not be repeated here.

[0205] In the several embodiments provided in this application, it should be understood that the disclosed methods, devices and systems can be implemented in other ways. The device embodiments described above are merely schematic, and the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0206] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the aforementioned image processing device, and the computer program includes program instructions. When the processor executes the above-mentioned program instructions, it can execute the method in the aforementioned embodiment, so it will not be described in detail here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the program instructions can be deployed on a computer device, or executed on multiple computer devices located at one location, or executed on multiple computer devices distributed at multiple locations and interconnected by a communication network. Multiple computer devices distributed at multiple locations and interconnected by a communication network can constitute a blockchain system.

[0207] According to one aspect of the present application, a computer program product is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, thereby enabling the computer device to perform the method of the aforementioned embodiment. Therefore, a detailed description thereof will not be given here.

[0208] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0209] It can be understood that in the specific implementation of this application, related data such as the image to be processed and the processing performance data of the image processing equipment are involved. When the above embodiments of this application are applied to specific products or technologies, the collection, use and processing of relevant data need to comply with relevant laws and standards in the relevant regions.

[0210] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0211] It should be noted that the terms "first" and "second" in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature designated as "first" or "second" may explicitly or implicitly include at least one such feature.

[0212] The above disclosure is only part of the embodiments of the present application, and it is certainly not intended to limit the scope of the rights of the present application. A person skilled in the art can understand that all or part of the processes of the above embodiments and equivalent changes made in accordance with the claims of the present application are still within the scope of the invention.

Claims

1. An image processing method, characterized in that: The method comprises: Acquiring an image to be processed, and extracting object features of each of a plurality of objects to be classified included in the image to be processed; performing feature cross processing on a first object feature and a second object feature to obtain a cross attention matrix between the first object feature and the second object feature, and determining correlation data between the first object feature and the second object feature based on the cross attention matrix; the first object feature is any one of a plurality of the object features, and the second object feature is any one of the plurality of the object features other than the first object feature; determining a correlation data set of the first object feature based on correlation data between the first object feature and each object feature among the plurality of object features except the first object feature; After determining the correlation data set of each of the multiple object features, image classification processing is performed based on each of the object features and the correlation data set of each of the object features to obtain a classification result of the image to be processed; the classification result includes the predicted category of each of the objects to be classified.

2. The method according to claim 1, wherein The cross-attention matrix includes a plurality of attention data; and determining the correlation data between the first object feature and the second object feature according to the cross-attention matrix includes: Sorting the plurality of attention data included in the cross-attention matrix in descending order of the values ​​corresponding to the attention data to obtain a sorting result; Selecting the top M attention data from the sorting results; M is a positive integer; Determine correlation data between the first object feature and the second object feature based on the selected M attention data.

3. The method according to claim 2, wherein The determining, based on the selected M attention data, correlation data between the first object feature and the second object feature includes: Determine position information of target attention data in the cross attention matrix, and determine position encoding data of the target attention data according to the position information; the target attention data is any one of the selected M attention data; performing fusion processing on the target attention data and the position encoding data of the target attention data to obtain fused data of the target attention data; The correlation data between the first object feature and the second object feature is determined based on the fusion data of each attention data in the selected M attention data.

4. The method according to claim 2 or 3, wherein: The method further comprises: evaluating the image complexity of the image to be processed, and determining a first weight parameter according to the image complexity of the image to be processed; Acquiring processing performance data of an image processing device, and determining a second weight parameter according to the processing performance data of the image processing device; The number M of attention data selected from the cross-attention matrix is ​​determined according to the first weight parameter and the second weight parameter.

5. The method according to any one of claims 1 to 3, wherein The performing feature cross processing on the first object feature and the second object feature to obtain a cross attention matrix between the first object feature and the second object feature includes: Mapping the first object features using the first parameter matrix to obtain a query vector matrix, and mapping the second object features using the second parameter matrix to obtain a key-value vector matrix; A cross-attention matrix between the first object feature and the second object feature is determined according to the query vector matrix, the key-value vector matrix, and a vector distance between the query vector matrix and the key-value vector matrix.

6. The method according to any one of claims 1 to 3, wherein The performing image classification processing according to each of the object features and the correlation data set of each of the object features to obtain the classification result of the image to be processed includes: Inputting each of the object features and the correlation data set of each of the object features into a target classification model for processing to obtain prediction information of the image to be processed; the prediction information of the image to be processed includes the category prediction probability of each of the objects to be classified; The classification result of the image to be processed is determined according to the category prediction probability of each of the objects to be classified.

7. The method according to claim 6, wherein The method further comprises: Acquire a sample image and annotation information of the sample image; the annotation information includes annotation categories of multiple target objects in the sample image; Extracting object features of each target object included in the sample image, and determining a correlation data set of the object features of each target object according to the object features of each target object; Inputting the object features of each target object and the correlation data set of the object features of each target object into the initial classification model for processing to obtain prediction information of the sample image; the prediction information of the sample image includes the category prediction probability of each target object; Determining category difference data for each target object based on the labeled category and category prediction probability of each target object, and determining target difference data based on the category difference data for the multiple target objects; The target difference data is used to adjust the model parameters of the initial classification model, and the target classification model is determined based on the initial classification model after the model parameters are adjusted.

8. An image processing device, characterized in that: The device comprises: A preprocessing module, configured to acquire an image to be processed and extract object features of each of a plurality of objects to be classified included in the image to be processed; a correlation detection module, configured to perform feature cross processing on a first object feature and a second object feature to obtain a cross-attention matrix between the first object feature and the second object feature, and determine correlation data between the first object feature and the second object feature based on the cross-attention matrix; the first object feature is any one of the plurality of object features, and the second object feature is any one of the plurality of object features other than the first object feature; The correlation detection module is further configured to determine a correlation data set of the first object feature based on correlation data between the first object feature and each object feature among the plurality of object features except the first object feature; A classification module is used to perform image classification processing based on each object feature and the correlation data set of each object feature after determining the correlation data set of each object feature among the multiple object features, so as to obtain a classification result of the image to be processed; the classification result includes the predicted category of each object to be classified.

9. A computer device, characterized in that: include: A processor, a storage device and a communication interface, wherein the processor, the communication interface and the storage device are interconnected, wherein the storage device stores an executable program code, and the processor is used to call the executable program code to implement the image processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the image processing method according to any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that The computer program product includes a computer program or computer instructions, and when the computer program or computer instructions are executed by a processor, they are used to implement the image processing method according to any one of claims 1 to 7.

Images