A data processing method, device, and computer-readable storage medium

Abstract

The application discloses a data processing method, device and computer readable storage medium. The method comprises: obtaining a label word vector corresponding to each category label in a category label set, obtaining an image attribute initial feature corresponding to an image, and an image semantic initial feature corresponding to the image; performing feature fusion on the label word vector matrix and the image attribute initial feature to obtain an image attribute feature, and performing self-attention enhancement on the image attribute feature to obtain an image attribute enhanced feature; performing feature fusion on the image attribute feature and the image semantic initial feature to obtain an image semantic enhanced feature; and determining a target category label of the image in the category label set according to the label word vector matrix, the image semantic enhanced feature and the image attribute enhanced feature. The application can improve the recognition accuracy of the target category label of the image. The embodiments of the application can be applied to various scenes such as cloud technology, artificial intelligence, intelligent transportation and auxiliary driving.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet, and particularly relates to a data processing method, device and computer readable storage medium. BACKGROUND

[0002] In an image recognition scenario, an image classification scenario, an image recommendation scenario, a video attribute recognition scenario and the like, it is crucial to determine a target class label of an image or a video frame (i.e., to recognize the class of the image or the video frame).

[0003] Existing image class label determination methods are mostly based on prior information of attribute features (such as style attributes, color attributes and the like) of an image to perform supervised training of a model, and select intermediate layer features of the model as representation features of the image, and determine a target class label of the image according to the representation features. However, the representation features are limited to attribute information of the image itself, and may not accurately represent the overall content or overall information of the image. In the case that the image content is not accurately represented, the recognition accuracy of the target class label of the image is reduced. SUMMARY

[0004] The embodiments of the present application provide a data processing method, device and computer readable storage medium, which can improve the recognition accuracy of a target class label of an image.

[0005] In an aspect, the embodiments of the present application provide a data processing method, comprising:

[0006] Obtaining a label word vector corresponding to each class label in a class label set, obtaining an image attribute initial feature corresponding to an image, and obtaining an image semantic initial feature corresponding to the image;

[0007] Performing feature fusion on the label word vector matrix and the image attribute initial feature to obtain an image attribute feature, and performing self-attention enhancement on the image attribute feature to obtain an image attribute enhanced feature. The label word vector matrix is generated based on the label word vector corresponding to each class label;

[0008] Performing feature fusion on the image attribute feature and the image semantic initial feature to obtain an image semantic enhanced feature;

[0009] Determining a target class label of the image in the class label set according to the label word vector matrix, the image semantic enhanced feature and the image attribute enhanced feature.

[0010] In an aspect, the embodiments of the present application provide a data processing device, comprising:

[0011] The data acquisition module is configured to acquire a label word vector corresponding to each category label in the category label set, acquire an image attribute initial feature corresponding to the image, and acquire an image semantic initial feature corresponding to the image.

[0012] The first fusion module is configured to perform feature fusion on the label word vector matrix and the image attribute initial feature to obtain an image attribute feature, and perform self-attention enhancement on the image attribute feature to obtain an image attribute enhanced feature.

[0013] The second fusion module is configured to perform feature fusion on the image attribute feature and the image semantic initial feature to obtain an image semantic enhanced feature.

[0014] The label determination module is configured to determine a target category label of the image from the category label set according to the label word vector matrix, the image semantic enhanced feature, and the image attribute enhanced feature.

[0015] The first fusion module includes:

[0016] The first acquisition unit is configured to acquire an image recognition model, and input the label word vector matrix and the image attribute initial feature into the image recognition model. The image recognition model includes a cross-modal fusion network layer, and the cross-modal fusion network layer includes a cross-modal cross-attention fusion component and a feature weighting component.

[0017] The first fusion unit is configured to perform feature fusion on the label word vector matrix and the image attribute initial feature through the cross-modal cross-attention fusion component to obtain a cross-modal association matrix associated with the label word vector matrix.

[0018] The first weighting unit is configured to input the cross-modal association matrix and the image attribute initial feature into the feature weighting component.

[0019] The first weighting unit is further configured to perform feature weighting on the image attribute initial feature according to the cross-modal association matrix in the feature weighting component to obtain the image attribute feature.

[0020] The cross-modal cross-attention fusion component includes a full connection layer subcomponent, a feature point product subcomponent, a matrix scaling subcomponent, and a matrix normalization subcomponent.

[0021] The first fusion unit includes:

[0022] The first transformation subunit is configured to perform feature transformation on the image attribute initial feature through the full connection layer subcomponent to obtain a to-be-fused image attribute feature.

[0023] The first input subunit is configured to input the image attribute feature to be fused and the label word vector matrix into the feature point multiplication subcomponent respectively.

[0024] The first input subunit is further configured to perform feature point multiplication on the image attribute feature to be fused and the label word vector matrix by using the feature point multiplication subcomponent, to obtain a cross-modal association matrix to be scaled.

[0025] The second input subunit is configured to input the cross-modal association matrix to be scaled into the matrix scaling subcomponent, and perform feature scaling on the cross-modal association matrix to be scaled by using the matrix scaling subcomponent, to obtain a cross-modal association matrix to be normalized.

[0026] The first generation subunit is configured to input the cross-modal association matrix to be normalized into the matrix normalization subcomponent, and perform feature normalization on the cross-modal association matrix to be normalized by using the matrix normalization subcomponent, to obtain the cross-modal association matrix associated with the label word vector matrix.

[0027] The feature weighting component includes a full connection layer subcomponent, a weighted weight subcomponent, a weighted normalization subcomponent, and an attribute weighting subcomponent.

[0028] The first weighting unit includes:

[0029] The second transformation subunit is configured to perform feature transformation on the image attribute initial feature by using the full connection layer subcomponent, to obtain the image attribute feature to be fused.

[0030] The second generation subunit is configured to perform feature multiplication on the cross-modal association matrix and the label word vector matrix by using the weighted weight subcomponent, to obtain the weighted weight to be normalized for the image attribute feature to be fused.

[0031] The third input subunit is configured to input the weighted weight to be normalized into the weighted normalization subcomponent, and perform weight normalization on the weighted weight to be normalized by using the weighted normalization subcomponent, to obtain the feature weighting weight for the image attribute feature to be fused.

[0032] The fourth input subunit is configured to input the feature weighting weight and the image attribute feature to be fused into the attribute weighting subcomponent respectively.

[0033] The fourth input subunit is further configured to perform feature weighting on the image attribute feature to be fused based on the feature weighting weight in the attribute weighting subcomponent, to obtain the image attribute feature.

[0034] The cross-modal fusion network layer further includes a cross-modal self-attention enhancement component, and the cross-modal self-attention enhancement component includes a first full connection layer subcomponent, a second full connection layer subcomponent, a third full connection layer subcomponent, and a cross-modal self-attention enhancement subcomponent.

[0035] The first fusion module comprises:

[0036] The first input unit is configured to input the image attribute feature into the first full connection layer subcomponent, perform feature transformation on the image attribute feature through the first full connection layer subcomponent, and obtain a first to-be-fused image attribute feature:

[0037] The second input unit is configured to input the image attribute feature into the second full connection layer subcomponent, perform feature transformation on the image attribute feature through the second full connection layer subcomponent, and obtain a second to-be-fused image attribute feature;

[0038] The third input unit is configured to input the image attribute feature into the third full connection layer subcomponent, perform feature transformation on the image attribute feature through the third full connection layer subcomponent, and obtain a to-be-enhanced image attribute feature;

[0039] The first enhancement unit is configured to input the first to-be-fused image attribute feature, the second to-be-fused image attribute feature, and the to-be-enhanced image attribute feature into the cross-modal self-attention enhancement subcomponent respectively.

[0040] The first enhancement unit is further configured to perform feature enhancement on the to-be-enhanced image attribute feature through the first to-be-fused image attribute feature and the second to-be-fused image attribute feature in the cross-modal self-attention enhancement subcomponent, and obtain an image attribute enhanced feature.

[0041] The cross-modal self-attention enhancement subcomponent comprises a first feature transpose subcomponent, a first enhancement weight subcomponent, a first enhancement scaling subcomponent, a first enhancement normalization subcomponent, and an attribute enhancement subcomponent.

[0042] The first enhancement unit comprises:

[0043] The third generation subunit is configured to perform feature transposition on the second to-be-fused image attribute feature through the first feature transpose subcomponent, and obtain a transposed image attribute feature corresponding to the second to-be-fused image attribute feature.

[0044] The fourth generation subunit is configured to input the transposed image attribute feature and the first to-be-fused image attribute feature into the first enhancement weight subcomponent respectively.

[0045] The fourth generation subunit is further configured to perform feature multiplication on the first to-be-fused image attribute feature and the transposed image attribute feature through the first enhancement weight subcomponent, and obtain a first to-be-scaled enhancement weight for the to-be-enhanced image attribute feature.

[0046] The fifth input subunit is configured to input the first to-be-scaled enhancement weight into the first enhancement scaling subcomponent, perform weight scaling on the first to-be-scaled enhancement weight through the first enhancement scaling subcomponent, and obtain a first to-be-normalized enhancement weight.

[0047] The fifth generating subunit is configured to input the first to-be-normalized enhancement weight into the first enhancement normalization subcomponent, perform weight normalization on the first to-be-normalized enhancement weight through the first enhancement normalization subcomponent, and obtain a first feature enhancement weight;

[0048] The sixth input subunit is configured to input the first feature enhancement weight and the to-be-enhanced image attribute feature into the attribute enhancement subcomponent, respectively.

[0049] The sixth input subunit is further configured to perform feature enhancement on the to-be-enhanced image attribute feature based on the first feature enhancement weight in the attribute enhancement subcomponent, and obtain an image attribute enhanced feature.

[0050] The image recognition model further includes an intra-modal fusion network layer, and the intra-modal fusion network layer includes an intra-modal self-attention enhancement component, an intra-modal cross-attention fusion component, and a feature fusion component.

[0051] The second fusion module includes:

[0052] The second enhancement unit is configured to input the image semantic initial feature into the intra-modal self-attention enhancement component, perform feature enhancement on the image semantic initial feature through the intra-modal self-attention enhancement component, and obtain a first image semantic feature.

[0053] The second fusion unit is configured to input the image semantic initial feature and the image attribute feature into the intra-modal cross-attention enhancement component, respectively.

[0054] The second fusion unit is further configured to perform feature fusion on the image semantic initial feature and the image attribute feature through the intra-modal cross-attention fusion component, and obtain a second image semantic feature.

[0055] The third fusion unit is configured to input the first image semantic feature and the second image semantic feature into the feature fusion component, respectively.

[0056] The third fusion unit is further configured to perform feature fusion on the first image semantic feature and the second image semantic feature through the feature fusion component, and obtain an image semantic enhanced feature.

[0057] The intra-modal self-attention enhancement component includes a fourth full connection layer subcomponent, a fifth full connection layer subcomponent, a sixth full connection layer subcomponent, and an intra-modal self-attention enhancement subcomponent.

[0058] The second enhancement unit includes:

[0059] The third transformation subunit is configured to perform feature transformation on the image semantic initial feature through the fourth full connection layer subcomponent, and obtain a first to-be-fused image semantic feature.

[0060] a fourth transformation subunit configured to perform feature transformation on the image semantic initial feature through a fifth full connection layer subcomponent to obtain second image semantic features to be fused;

[0061] a fifth transformation subunit configured to perform feature transformation on the image semantic initial feature through a sixth full connection layer subcomponent to obtain image semantic features to be enhanced;

[0062] a seventh input subunit configured to input the first image semantic features to be fused, the second image semantic features to be fused, and the image semantic features to be enhanced into an intra-modal self-attention enhancement subcomponent respectively;

[0063] the seventh input subunit is further configured to perform feature enhancement on the image semantic features to be enhanced through the first image semantic features to be fused and the second image semantic features to be fused in the intra-modal self-attention enhancement subcomponent to obtain first image semantic features.

[0064] The intra-modal self-attention enhancement subcomponent includes a second feature transposition subcomponent, a second enhancement weight subcomponent, a second enhancement scaling subcomponent, a second enhancement normalization subcomponent, and a first semantic enhancement subcomponent.

[0065] The seventh input subunit includes:

[0066] a first transposition subunit configured to perform feature transposition on the second image semantic features to be fused through the second feature transposition subcomponent to obtain transposed image semantic features corresponding to the second image semantic features to be fused;

[0067] a first multiplication subunit configured to input the transposed image semantic features and the first image semantic features to be fused into the second enhancement weight subcomponent respectively;

[0068] the first multiplication subunit is further configured to perform feature multiplication on the first image semantic features to be fused and the transposed image semantic features through the second enhancement weight subcomponent to obtain second to-be-scaled enhancement weights for the image semantic features to be enhanced;

[0069] a first scaling subunit configured to input the second to-be-scaled enhancement weights into the second enhancement scaling subcomponent, and perform weight scaling on the second to-be-scaled enhancement weights through the second enhancement scaling subcomponent to obtain second to-be-normalized enhancement weights;

[0070] a first normalization subunit configured to input the second to-be-normalized enhancement weights into the second enhancement normalization subcomponent, and perform weight normalization on the second to-be-normalized enhancement weights through the second enhancement normalization subcomponent to obtain second feature enhancement weights;

[0071] The first enhancement unit is configured to input the second feature enhancement weight and the image semantic feature to be enhanced into the first semantic enhancement subcomponent respectively.

[0072] The first enhancement unit is further configured to perform feature enhancement on the image semantic feature to be enhanced based on the second feature enhancement weight in the first semantic enhancement subcomponent, to obtain the first image semantic feature.

[0073] The intra-modal cross-attention fusion component includes a fourth full connection layer subcomponent, a second full connection layer subcomponent, a third full connection layer subcomponent, and an intra-modal cross-attention enhancement subcomponent.

[0074] The second fusion unit includes:

[0075] The sixth transformation subunit is configured to perform feature transformation on the image semantic initial feature through the fourth full connection layer subcomponent, to obtain the first image semantic feature to be fused.

[0076] The seventh transformation subunit is configured to perform feature transformation on the image attribute feature through the second full connection layer subcomponent, to obtain the second image attribute feature to be fused.

[0077] The eighth transformation subunit is configured to perform feature transformation on the image attribute feature through the third full connection layer subcomponent, to obtain the image attribute feature to be enhanced.

[0078] The eighth input subunit is configured to input the first image semantic feature to be fused, the second image attribute feature to be fused, and the image attribute feature to be enhanced into the intra-modal cross-attention enhancement subcomponent respectively.

[0079] The eighth input subunit is further configured to perform feature enhancement on the image attribute feature to be enhanced through the first image semantic feature to be fused and the second image attribute feature to be fused in the intra-modal cross-attention enhancement subcomponent, to obtain the second image semantic feature.

[0080] The intra-modal cross-attention enhancement subcomponent includes a first feature transpose subcomponent, a third enhancement weight subcomponent, a third enhancement scaling subcomponent, a third enhancement normalization subcomponent, and a second semantic enhancement subcomponent.

[0081] The eighth input subunit includes:

[0082] The second transpose subunit is configured to perform feature transposition on the second image attribute feature to be fused through the first feature transpose subcomponent, to obtain the transposed image attribute feature corresponding to the second image attribute feature to be fused.

[0083] The second multiplication subunit is configured to input the transposed image attribute feature and the first image semantic feature to be fused into the third enhancement weight subcomponent respectively.

[0084] The second product subunit is further configured to perform feature multiplication on the first to-be-fused image semantic feature and the transposed image attribute feature by a third enhancement weight subcomponent to obtain a third to-be-scaled enhancement weight for the to-be-enhanced image attribute feature;

[0085] The second scaling subunit is configured to input the third to-be-scaled enhancement weight into a third enhancement scaling subcomponent, perform weight scaling on the third to-be-scaled enhancement weight by the third enhancement scaling subcomponent, and obtain a third to-be-normalized enhancement weight;

[0086] The second normalization subunit is configured to input the third to-be-normalized enhancement weight into a third enhancement normalization subcomponent, perform weight normalization on the third to-be-normalized enhancement weight by the third enhancement normalization subcomponent, and obtain a third feature enhancement weight;

[0087] The second enhancement subunit is configured to input the third feature enhancement weight and the to-be-enhanced image attribute feature into a second semantic enhancement subcomponent, respectively.

[0088] The second enhancement subunit is further configured to perform feature enhancement on the to-be-enhanced image attribute feature based on the third feature enhancement weight in the second semantic enhancement subcomponent, and obtain a second image semantic feature.

[0089] The image recognition model further includes a label enhancement network layer, and the label enhancement network layer includes a cross-modal semantic enhancement component, a feature pooling component, and a feature classification component.

[0090] The determining label module includes:

[0091] The second weighting unit is configured to input the label word vector matrix and the image semantic enhancement feature into the cross-modal semantic enhancement component, respectively.

[0092] The second weighting unit is further configured to perform feature weighting on the image semantic enhancement feature based on the label word vector matrix in the cross-modal semantic enhancement component, and obtain an image semantic weighted feature.

[0093] The second obtaining unit is configured to input the image semantic weighted feature into the feature pooling component, and obtain a semantic feature classification result corresponding to the image semantic weighted feature by the feature pooling component.

[0094] The third obtaining unit is configured to input the image attribute enhanced feature into the feature classification component, and obtain an attribute feature classification result corresponding to the image attribute enhanced feature by the feature classification component.

[0095] The fourth fusion unit is configured to perform result fusion on the semantic feature classification result and the attribute feature classification result, and obtain a target classification result.

[0096] The fourth fusion unit is further configured to determine a target category label of the image from the target classification result in the category label set.

[0097] The label word vector matrix includes at least two label word vectors. The at least two label word vectors include a label word vector A b and a label word vector A c b and c are positive integers, b is less than or equal to a total number of the at least two label word vectors, and c is less than or equal to the total number of the at least two label word vectors. The cross-modal semantic enhancement component includes a label similarity subcomponent, a seventh fully connected layer subcomponent, and a semantic weighting subcomponent.

[0098] The second weighting unit includes:

[0099] The sixth generation subunit is configured to obtain a vector similarity between the label word vector A b and the label word vector A c by the label similarity subcomponent, and generate a label similarity matrix based on the obtained vector similarity.

[0100] The ninth transformation subunit is configured to perform feature transformation on the image semantic enhancement feature by the seventh fully connected layer subcomponent to obtain a to-be-weighted image semantic enhancement feature.

[0101] The seventh generation subunit is configured to input the label similarity matrix and the to-be-weighted image semantic enhancement feature into the semantic weighting subcomponent.

[0102] The seventh generation subunit is further configured to perform feature weighting on the to-be-weighted image semantic enhancement feature based on the label similarity matrix in the semantic weighting subcomponent to obtain an image semantic weighting feature.

[0103] The data acquisition module includes:

[0104] The fourth acquisition unit is configured to acquire a sample image set. Each sample image in the sample image set carries a category label.

[0105] The fourth acquisition unit is further configured to generate a category label set according to the category label carried by each sample image.

[0106] The fifth acquisition unit is configured to acquire a word vector model and acquire a label word vector corresponding to each category label in the category label set according to the word vector model.

[0107] The sixth acquisition unit is configured to acquire an image, perform cropping processing on the image to acquire D sub-images, and acquire image attribute initial features corresponding to the D sub-images. D is a positive integer, the image sizes corresponding to the D sub-images are all the same, and the image sizes corresponding to the D sub-images are all less than or equal to an image size corresponding to the image.

[0108] The seventh obtaining unit is configured to obtain an image feature extraction model, and obtain initial image semantic features corresponding to an image according to the image feature extraction model.

[0109] In an aspect, the present application provides a computer device, comprising: a processor, a memory, a network interface;

[0110] The processor is connected with the memory and the network interface, wherein the network interface is configured to provide a data communication function, the memory is configured to store a computer program, and the processor is configured to call the computer program to enable the computer device to execute the method in the embodiments of the present application.

[0111] In an aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is adapted to be loaded by a processor and execute the method in the embodiments of the present application.

[0112] In an aspect, the present application provides a computer program product or a computer program, wherein the computer program product or the computer program comprises computer instructions stored in a computer readable storage medium; a processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to execute the method in the embodiments of the present application.

[0113] In the embodiment of the present application, by acquiring the label word vector, the image attribute initial feature and the image semantic initial feature, the label word vector matrix (the category label belongs to the text mode) and the image attribute initial feature can be cross-modal feature fusion, that is, the cross-modal cross attention enhancement of the image attribute initial feature is realized by the label word vector matrix, and the image attribute feature can be obtained; further, the image attribute feature is self-attention enhanced to obtain the image attribute enhanced feature; further, the image attribute feature and the image semantic initial feature are feature fused, that is, the intra-modal cross attention enhancement of the image semantic initial feature is realized by the image attribute feature, and the image semantic enhanced feature can be obtained; further, in combination with the multi-modal feature (including the label word vector matrix, the image semantic enhanced feature and the image attribute enhanced feature), the target category label of the image can be determined in the category label set. As can be seen from the above, not only the cross-modal feature fusion of the text modal feature (the label word vector matrix) and the image modal feature (the image attribute initial feature) is performed to enhance the image modal feature and obtain the image attribute enhanced feature, but also the intra-modal feature fusion of the image attribute feature and the image semantic initial feature is performed to enhance the image semantic initial feature and obtain the image semantic enhanced feature; through the cross-modal feature fusion and the intra-modal feature fusion, the multi-modal information can be integrated to generate the feature including the multi-modal information, and then the feature representation capability of the image can be improved, so that the recognition accuracy of the target category label of the image can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0114] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0115] Figure 1 is a system architecture schematic diagram provided by an embodiment of the present application;

[0116] Figure 2 is a data processing scene schematic diagram provided by an embodiment of the present application;

[0117] Figure 3 is a flowchart of a data processing method provided by an embodiment of the present application;

[0118] Figure 4 is a data processing scene schematic diagram provided by an embodiment of the present application;

[0119] Figure 5 is a data processing scene schematic diagram provided by an embodiment of the present application;

[0120] Figure 6 is a scenario diagram of data processing provided by an embodiment of the present application;

[0121] Figure 7 is a scenario diagram of data processing provided by an embodiment of the present application;

[0122] Figure 8 is a flow diagram of a data processing method provided by an embodiment of the present application;

[0123] Figure 9 is a structural diagram of a data processing apparatus provided by an embodiment of the present application;

[0124] Figure 10 is a structural diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0125] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0126] For ease of understanding, first, some nouns are simply explained as follows:

[0127] Artificial intelligence (AI) is to use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0128] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, automatic driving, intelligent transportation, etc.

[0129] Computer Vision (CV) is a science that studies how to make machines "see". More specifically, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further process graphics so that the computer processing becomes images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, and attempts to establish artificial intelligence systems that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, autonomous driving, intelligent transportation, and other technologies. It also includes common face recognition, fingerprint recognition, and other biometric identification technologies. In the embodiments of the present application, computer vision technology can be used to identify the category labels (such as people, dogs, cats, birds, etc.) in the image.

[0130] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and other disciplines. It is a specialized study of how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. Its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and inductive learning. In the embodiments of the present application, the image recognition model and the image feature extraction model are AI models based on machine learning technology. The image recognition model can be used for image recognition processing, and the image feature extraction model can be used for image feature extraction.

[0131] Please refer to Figure 1 , Figure 1 is a system architecture diagram provided by an embodiment of the present application. As Figure 1 shown, the system can include a business server 100 and a terminal cluster, which can include terminal devices 200a, 200b, 200c,..., and 200n. It can be understood that the above system can include one or more terminal devices, and the present application does not limit the number of terminal devices.

[0132] Among them, there can be a communication connection between the terminal clusters, for example, there is a communication connection between the terminal device 200a and the terminal device 200b, and there is a communication connection between the terminal device 200a and the terminal device 200c. At the same time, any terminal device in the terminal cluster can have a communication connection with the service server 100, for example, there is a communication connection between the terminal device 200a and the service server 100, wherein the above communication connection is not limited to the connection mode, which can be connected directly or indirectly through wired communication, or directly or indirectly through wireless communication, or through other ways, which are not limited in this application.

[0133] It should be understood that each terminal device in the terminal cluster as shown in Figure 1 may be installed with an application client, which can respectively interact with the service server 100 as shown in Figure 1 above when running in each terminal device, that is, the above communication connection. Among them, the application client can be a short video application, a video application, a live application, a social application, an instant messaging application, a game application, a music application, a shopping application, a novel application, a payment application, a browser and other application clients with image loading function. Among them, the application client can be an independent client, or an embedded sub-client integrated in a certain client (such as a social client, an education client, and a multimedia client, etc.), which is not limited here. Taking the social application as an example, the service server 100 can be a collection of multiple servers including the background server corresponding to the social application, the data processing server, etc., so that each terminal device can transmit data with the service server 100 through the application client corresponding to the social application, such as each terminal device can upload its local image to the service server 100 through the application client of the social application, and then the service server 100 can distribute the image to other terminal devices or transmit it to the cloud server. It should be noted that the terminal device uploads the image after obtaining the permission granted by the device user corresponding to the terminal device.

[0134] For the convenience of subsequent understanding and description, the embodiments of the present application can be in Figure 1A terminal device in the terminal cluster is selected as a target terminal device, for example, terminal device 200a is selected as a target terminal device. When an image is acquired and needs to be processed, for example, a class label of the image is marked, the terminal device 200a can send the image to the service server 100. After receiving the image sent by the terminal device 200a, the service server 100 can obtain the label word vector corresponding to each class label in the class label set respectively, wherein the class label set is generated based on the class label carried by each sample image in the sample image set, and the sample image in the sample image set is used to train the initial image recognition model to obtain the image recognition model used for image recognition (classification); further, the service server 100 can obtain the initial feature of the image attribute corresponding to the image, and the initial feature of the image semantics corresponding to the image, wherein the initial feature of the image attribute refers to the initial feature corresponding to the image attribute, for example, the initial feature of the color attribute, the initial feature of the contour attribute, etc. It can be understood that the initial feature of the image attribute is the low-level initial feature of the image; the initial feature of the image semantics refers to the initial feature of the image semantics extracted by the image feature extraction model. It can be understood that the initial feature of the image semantics is the high-level initial feature of the image. Further, based on the pre-trained image recognition model, the service server 100 performs feature fusion on the label word vector matrix and the initial feature of the image attribute, and can obtain the image attribute feature with better image representation capability than the initial feature of the image attribute. The feature fusion process can be understood as a feature enhancement process of the initial feature of the image attribute by the label word vector matrix. Obviously, the feature fusion process is a cross-modal (i.e. text modal and image modal) fusion process, wherein the label word vector matrix is generated based on the label word vector corresponding to each class label; further, the service server 100 performs self-attention enhancement on the image attribute feature, and can obtain the image attribute enhanced feature with better image representation capability than the image attribute feature; further, based on the image recognition model, the service server 100 performs feature fusion on the image attribute feature and the initial feature of the image semantics, and can obtain the image semantic enhanced feature with better image representation capability than the initial feature of the image semantics. The feature fusion process here can be understood as a feature enhancement process of the initial feature of the image semantics by the image attribute feature. Obviously, the feature fusion process here is an intra-modal (i.e. image modal) fusion process; further, in the image recognition model, the service server 100 can determine the target class label of the image in the class label set according to the label word vector matrix, the image semantic enhanced feature and the image attribute enhanced feature.

[0135] Subsequently, the business server 100 can mark the target category label on the image, and send the image carrying the target category label to the terminal device 200a. After receiving the image carrying the target category label sent by the business server 100, the terminal device 200a can display the image carrying the target category label on the corresponding screen. It can be understood that the business server 100 can also send only the target category label to the terminal device 200a, or send the target category label in association with the image to the terminal device 200a.

[0136] Optionally, if the terminal device 200a locally stores the image recognition model and the label word vector matrix, the terminal device 200a can locally generate the image attribute initial feature corresponding to the image and the image semantic initial feature corresponding to the image, and then determine the target category label corresponding to the image through the image recognition model, the image attribute initial feature, the image semantic initial feature, and the label word vector matrix. It can be understood that the process of determining the target category label locally in the terminal device 200a is consistent with the process of determining the target category label in the business server 100, and therefore will not be described here. Please refer to the description of determining the target category label in the business server 100 above. Since training the image recognition model involves a large amount of offline calculation, the image recognition model locally in the terminal device 200a can be sent to the terminal device 200a after being trained by the business server 100. Similarly, the label word vector matrix can be generated by the business server 100 and then sent to the terminal device 200a.

[0137] Optionally, it can be understood that the system architecture can include multiple business servers, and one terminal device can be connected to one business server. Each business server can obtain the image uploaded by the terminal device connected thereto, so as to recognize the image and determine the target category label corresponding to the image, and return the target category label to the terminal device connected thereto.

[0138] It should be noted that the business server 100, the terminal device 200a, the terminal device 200b, the terminal device 200c,..., and the terminal device 200n can all be blockchain nodes in a blockchain network. The data described throughout the text (such as images and target category labels corresponding to the images) can be stored. The storage method can be that the blockchain nodes generate blocks according to the data and add the blocks to the blockchain for storage.

[0139] The blockchain is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism and encryption algorithm, and is mainly used for arranging data in chronological order and encrypting the data into a ledger so that the data cannot be tampered with and forged, and the data can be verified, stored and updated. The blockchain is essentially a decentralized database, each node in the database stores a same blockchain, and the blockchain network can distinguish the nodes into core nodes, data nodes and light nodes. The core nodes, data nodes and light nodes together constitute the blockchain nodes. The core nodes are responsible for the consensus of the whole blockchain network, that is, the core nodes are consensus nodes in the blockchain network. The process of writing transaction data in the ledger in the blockchain network can be that the data nodes or light nodes in the blockchain network obtain transaction data, deliver the transaction data in the blockchain network (that is, the nodes deliver the transaction data in the manner of a baton), until the consensus nodes receive the transaction data, the consensus nodes package the transaction data into a block, perform consensus on the block, and write the transaction data into the ledger after the consensus is completed. Here, the transaction data is an example of image and target category label corresponding to the image, and the business server 100 (blockchain node) generates a block according to the transaction data after consensus on the transaction data, and stores the block into the blockchain network; and for reading of the transaction data (that is, the image and the target category label corresponding to the image), the blockchain node can obtain the block containing the transaction data in the blockchain network, and further obtain the transaction data in the block.

[0140] It can be understood that the method provided by the embodiments of the present application can be executed by a computer device, which includes but is not limited to a terminal device or a business server. The business server can be a standalone physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud database, cloud service, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform. The terminal device includes but is not limited to a mobile phone, a computer, a smart voice interactive device, a smart home appliance, a vehicle-mounted terminal, etc. The terminal device and the business server can be directly or indirectly connected through a wired or wireless manner, which is not limited in the embodiments of the present application.

[0141] It can be understood that the above system architecture can be applicable to business scenarios such as image recognition, image classification, image recommendation, image distribution, image search, video attribute recognition, etc. Here, specific business scenarios will not be enumerated one by one.

[0142] Further, please refer to Figure 2 , Figure 2is a scenario diagram of data processing provided by an embodiment of the present application. The implementation process of the data processing scenario can be performed in a business server, in a terminal device, or in interaction between the terminal device and the business server, which is not limited here. The terminal device can be the terminal device 200a described above. Figure 1 any one of the terminal devices in the terminal cluster of the corresponding embodiment, Figure 2 The business server can be the business server 100 of the corresponding embodiment of the terminal device 200a described above. Figure 1 The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, intelligent transportation, and assisted driving.

[0143] As shown in Figure 2 The user 201a can upload an image 202a to a certain application client (for example, a browser) on the terminal device 200a that has an image recognition function. Optionally, the application client can be integrated with a camera function, so the image 202a can be an image collected through the camera function on the application client. Optionally, the image 202a is an image obtained by the user 201a from other platforms, and the specific content and source of the image are not limited in the embodiments of the present application. Further, when the user 201a clicks on the recognition control 203a on the application client, the terminal device 200a can respond to the recognition operation and send the image 202a to the business server 100.

[0144] Further, after obtaining the image 202a sent by the terminal device 200a, the business server 100 obtains the label word vector corresponding to each category label in the category label set 20b, as shown in Figure 2 The category label set 20b can include category labels such as person, cat,..., and house, wherein the label word vector corresponding to the category label person is the label word vector 201c, the label word vector corresponding to the category label cat is the label word vector 202c,..., and the label word vector corresponding to the category label house is the label word vector 203c. It should be noted that the content of the category label is not limited in the embodiments of the present application, and can be set according to the actual scenario in actual application. Similarly, the number of category labels is not limited, and can be set according to the actual scenario in actual application.

[0145] Further, according to the label word vector corresponding to each category label, the business server 100 can generate a label word vector matrix 20c. For ease of understanding, it is assumed that the label word vector corresponding to each category label is a 100-dimensional vector, and the category label set 20b has 3 category labels (for example Figure 2If the image 202a is an example of a person, a cat, and a house, the business server 100 can generate a 3*100 label word vector matrix 20c. The business server 100 obtains the image attribute initial feature 20d corresponding to the image 202a, and the image semantic initial feature 20e corresponding to the image 202a. The above obtaining process is not described in the embodiment of the application, please refer to the following Figure 3 The description of step S101 in the corresponding embodiment.

[0146] Further, the business server 100 inputs the label word vector matrix 20c, the image attribute initial feature 20d, and the image semantic initial feature 20e into the image recognition model trained in advance. The framework of the image recognition model is roughly divided into three parts, which are Figure 2 The cross-modal fusion network layer 20f, the intra-modal fusion network layer 20g, and the label enhancement network layer 20h shown in the figure; through the cross-modal fusion network layer 20f, the business server 100 fuses the label word vector matrix 20c and the image attribute initial feature 20d to obtain the image attribute feature 201d. It can be understood that the label word vector matrix 20c belongs to the text mode, and the image attribute initial feature 20d belongs to the image mode, so the fusion process is cross-modal feature fusion, and the generated image attribute feature 201d not only contains image modal features, but also includes text modal features; further, through the cross-modal fusion network layer 20f, the service server 100 performs self-attention enhancement on the image attribute feature 201d to obtain the image attribute enhanced feature 202d. It can be understood that the image attribute enhanced feature 202d not only contains image modal features, but also includes text modal features.

[0147] Please refer to Figure 2 Through the intra-modal fusion network layer 20g, the business server 100 fuses the image attribute feature 201d and the image semantic initial feature 20e to obtain the image semantic enhanced feature 201e. It can be understood that the image attribute feature 201d and the image semantic initial feature 20e both belong to image modal features, but the image attribute feature 201d is the attribute feature of the image 202a, such as color feature, contour feature, etc., and the image semantic initial feature 20e is the semantic feature of the image 202a, so the image semantic enhanced feature 201e not only contains the high-level features of the image 202a, but also can contain the low-level features of the image 202a.

[0148] It should be noted that the embodiments of the present application do not limit the number of network structures composed of the cross-modal fusion network layer 20f and the intra-modal fusion network layer 20g, which can be one or more network structures. If the image recognition model includes the above-mentioned one network structure, that is, the image recognition model only includes one cross-modal fusion network layer 20f and one intra-modal fusion network layer 20g, the service server 100 inputs the image semantic enhanced feature 201e and the image attribute enhanced feature 202d into the label enhancement network layer 20h. If the image recognition model includes the above-mentioned N network structures (N is a positive integer greater than 1), that is, the image recognition model only includes N cross-modal fusion network layers 20f and N intra-modal fusion network layers 20g, the output data (for example, the image attribute enhanced feature 202d) of the previous cross-modal fusion network layer 20f is taken as the input data of the next cross-modal fusion network layer 20f, and the output data (for example, the image semantic enhanced feature 201e) of the previous intra-modal fusion network layer 20g is taken as the input data of the next intra-modal fusion network layer 20g. The above process is repeated for N times to continuously enhance the attribute feature and the semantic feature of the image 202a. The service server 100 inputs the output data of the last (Nth) cross-modal fusion network layer 20f into the label enhancement network layer 20h, inputs the output data of the last (Nth) intra-modal fusion network layer 20g into the label enhancement network layer 20h, and inputs the label word vector matrix 20c into the label enhancement network layer 20h. Figure 2 Figure 2

[0149] Further, in the label enhancement network layer 20h, the service server 100 obtains the target classification result of the image 202a according to the label word vector matrix 20c, the image semantic enhanced feature and the image attribute enhanced feature. The embodiments of the present application do not describe the generation process of the target classification result, please refer to the description of step S104 in the corresponding embodiments below. Figure 3 Figure 2 As shown in the figure, according to the target classification result, the service server 100 determines that the target category label of the image 202a includes a person and a house in the category label set 20b. Subsequently, the service server 100 returns the target category label (i.e., the person and the house) to the terminal device 200a.

[0150] ​​​In summary, the input data of the embodiment of the present application can include two modal data, i.e., text modal data and image modal data. In the cross-modal fusion network layer, the image attribute initial feature (belonging to the image modal data) is enhanced by the label word vector matrix (belonging to the text modal data) to obtain the image attribute enhanced feature. In the intra-modal fusion network layer, the image semantic initial feature (belonging to the image modal data) is enhanced by the image attribute feature to obtain the image semantic enhanced feature. In the label enhancement network layer, the image semantic enhanced feature is weighted by the label word vector matrix to further enhance the image semantic enhanced feature. Therefore, the image attribute enhanced feature and the image semantic weighted feature can accurately represent the image information, and the recognition accuracy of the target class label of the image can be improved.

[0151] Further, referring to Figure 3 , Figure 3 is a flowchart of a data processing method provided by an embodiment of the present application. The data processing method can be executed by a business server (for example, the business server 100 shown in the above Figure 1 , can also be executed by a terminal device (for example, the terminal device 200a shown in the above Figure 1 , and can also be executed by interaction of the business server and the terminal device. For ease of understanding, the embodiment of the present application takes the method executed by the business server as an example for description. As shown in the above Figure 3 , the data processing method can at least include the following steps S101-S104.

[0152] Step S101, obtaining a label word vector corresponding to each class label in a class label set, obtaining an image attribute initial feature corresponding to an image, and obtaining an image semantic initial feature corresponding to the image.

[0153] Specifically, a sample image set is obtained. Each sample image in the sample image set carries a class label. A class label set is generated according to the class label carried by each sample image. A word vector model is obtained, and a label word vector corresponding to each class label in the class label set is obtained according to the word vector model. An image is obtained, and D sub-images are obtained by cropping the image. An image attribute initial feature corresponding to each of the D sub-images is obtained. D is a positive integer, the image sizes corresponding to the D sub-images are the same, and the image sizes corresponding to the D sub-images are all less than or equal to the image size corresponding to the image. An image feature extraction model is obtained, and an image semantic initial feature corresponding to the image is obtained according to the image feature extraction model.

[0154] Modality refers to the form in which data exists, such as text, audio, image, and video file formats. Some data may exist in different forms but all describe the same thing or event. The input data in this application embodiment includes two modalities: text modality and image modality. For text modality, this application embodiment can obtain the tag word vectors corresponding to the category tag words through a word vector model. Please refer to [link to relevant documentation]. Figure 4 , Figure 4 This is a schematic diagram of a data processing scenario provided in an embodiment of this application. For example... Figure 4 As shown, the business server 100 obtains a sample image set 40a, which may include multiple sample images, such as... Figure 4 The example sample images 401a, 402a, ..., 403a are not limited in the total number or content of the sample images in this embodiment; they should be set according to the actual application scenario. Each sample image in the sample image set 40a carries a category label, and sample image 401a carries category label 1 (...). Figure 4 Abbreviated as Label 1) and Category Label 2 ( Figure 4 Abbreviated as label 2), sample image 402a carries category label 1 and category label 3 (abbreviated as label 2). Figure 4 (Abbreviated as label 3), ..., sample image 403a carries category label 2. Based on the category label carried by each sample image, the business server 100 generates a category label set 40b. For ease of description and understanding, this embodiment of the application assumes that the category label set 40b includes 3 category labels, namely... Figure 4 The example includes labels 1, 2, and 3. It is understood that the sample image set 40a is used to train the initial image recognition model to obtain the final image recognition model.

[0155] Furthermore, the business server 100 retrieves the word vector model 40c. This embodiment does not limit the model type of the word vector model 40c; it can be any natural language processing model, such as a deep self-attention transform network (Transformer, a deep learning model widely used in natural language translation and image processing), Word2Vec (a model used to generate word vectors), a bidirectional encoder representation from transformers (BERT), etc. The business server 100 obtains the tag word vectors corresponding to each category tag in the category tag set 40a based on the word vector model 40c, such as... Figure 4 As shown in the example, label 1 corresponds to label word vector 401d, label 2 corresponds to label word vector 402d, and label 3 corresponds to label word vector 403d.

[0156] For the image modality, after the image is acquired, the business server can acquire an image feature extraction model. The model type of the image feature extraction model is not limited in the embodiments of the present application, and the image feature extraction model can be composed of any one or more neural network models, such as Conditional Random Fields (CRFs), Convolutional Neural Networks (CNN), Fully Convolutional Networks (FCNs), etc. According to the image feature extraction model, the business server can acquire the initial image semantic features corresponding to the image. It should be noted that the initial image semantic features in the embodiments of the present application are high-level semantic features of the image. In order to distinguish from the enhanced high-level semantic features (for example, image semantic enhanced features) in the following, the semantic features output by the image feature extraction model are referred to as initial image semantic features.

[0157] Optionally, the business server performs cropping processing on the image to acquire D sub-images, wherein the image contents contained in the D sub-images can be independent or partially overlapped. Optionally, the business server takes the image as a sub-image. Further, the business server acquires initial image attribute features corresponding to the D sub-images. It should be noted that the initial image attribute features are low-level attribute features of the image. In order to distinguish from the enhanced attribute features (for example, image attribute features) in the following, the features are referred to as initial features. The attribute type is not limited in the embodiments of the present application, and can be any one of image attribute types, such as color attribute, texture attribute, etc. The method for acquiring the initial image attribute features is not limited in the embodiments of the present application, and can be set according to actual application scenarios. For example, if the attribute type is a color attribute, the business server can acquire the initial image attribute features of the image through a color histogram. For example, if the attribute type is a texture attribute, the business server can acquire the initial image attribute features of the image through a gray level co-occurrence matrix. The initial image attribute features can also be acquired through a neural network model.

[0158] In step S102, the initial image attribute features are fused with the label word vector matrix to obtain image attribute features, and the image attribute features are enhanced through self-attention to obtain image attribute enhanced features. The label word vector matrix is generated based on the label word vector corresponding to each category label.

[0159] Specifically, the business server acquires a trained image recognition model. The image recognition model mainly includes a cross-modal fusion network layer, an intra-modal fusion network layer, and a label enhancement network layer. The number of network layers corresponding to the cross-modal fusion network layer and the intra-modal fusion network layer is not limited in the embodiments of the present application, and the number of the above two network layers can be set according to actual application scenarios. For details, please refer to the aboveFigure 2 The description in the corresponding embodiment.

[0160] The business server first performs feature enhancement on the image attribute initial feature through a cross-modal fusion network layer. For details, please refer to Figure 5 , Figure 5 is a scene diagram of data processing provided by the embodiment of the application, as Figure 5 shown, the cross-modal fusion network layer can include a cross-modal cross-attention fusion component, a feature weighting component, and a cross-modal self-attention enhancement component. The business server inputs the label word vector matrix 40d and the image attribute initial feature 50a into the cross-modal cross-attention fusion component. The embodiment of the application assumes that there are three category labels in the category label set, such as Figure 4 label 1, label 2, and label 3 as shown in the example. Therefore, the label word vector matrix 40d includes the label word vector 401d corresponding to label 1, the label word vector 402d corresponding to label 2, and the label word vector 403d corresponding to label 3. It should be noted that the above example is only for ease of understanding. In actual application, the category label set can include one or more category labels. Therefore, the business server can obtain one or more label word vectors.

[0161] Through the cross-modal cross-attention fusion component, the business server performs feature fusion on the label word vector matrix 40d and the image attribute initial feature 50a to obtain a cross-modal association matrix 50c associated with the label word vector matrix 40d. The above process can be understood as a multi-modal cross-attention fusion process, that is, a process of fusing the text modality (label word vector matrix 40d) and the image modality (image attribute initial feature 50a). Further, the business server inputs the cross-modal association matrix 50c and the image attribute initial feature 50a into the feature weighting component. In the feature weighting component, the weight for the image attribute initial feature 50a is extracted through the cross-modal association matrix 50c and the label word vector matrix 40d. The image attribute initial feature 50a is weighted according to the extracted weight. This process can enhance the response of the co-occurring category label in the image attribute initial feature 50a to obtain an image attribute feature 501a. Further, the business server inputs the image attribute feature 501a into the cross-modal self-attention enhancement component. The image representation capability of the image attribute feature 501a is further enhanced through self-attention to obtain an image attribute enhanced feature 502a.

[0162] In step S103, the image attribute feature and the image semantic initial feature are fused to obtain an image semantic enhanced feature.

[0163] Specifically, after obtaining the image attribute feature and the image semantic initial feature, the business server performs intra-modal feature fusion and enhancement on the image semantic initial feature through the image attribute feature in the intra-modal fusion network layer, please refer to Figure 6 , Figure 6 is a scene diagram of data processing provided by the embodiment of the present application. As shown in Figure 6 , the intra-modal fusion network layer can include an intra-modal self-attention enhancement component, an intra-modal cross-attention fusion component, and a feature fusion component 60b. The business server inputs the image semantic initial feature 60a into the intra-modal self-attention enhancement component, performs self-attention feature enhancement on the image semantic initial feature 60a, and obtains a first image semantic feature 601a. The image attribute feature 501a and the image semantic initial feature 60a are respectively input into the intra-modal cross-attention fusion component. The business server performs intra-modal cross-attention fusion and enhancement on the image semantic initial feature 60a by using the image attribute feature 501a (which is the weighted and enhanced image attribute initial feature), and obtains a second image semantic feature 602a. It can be understood that the image semantic initial feature 60a is enhanced by using the enhanced image attribute feature and the image semantic initial feature 60a for fusion. Finally, the first image semantic feature 601a (the image semantic feature after self-attention enhancement) and the second image semantic feature 602a (the image semantic feature after cross-attention enhancement) are respectively input into the feature fusion component 60b. The business server performs feature fusion on the first image semantic feature 601a and the second image semantic feature 602a through the feature fusion component 60b, and obtains an image semantic enhanced feature 603a.

[0164] In step S104, the target category label of the image is determined in the category label set according to the label word vector matrix, the image semantic enhanced feature, and the image attribute enhanced feature.

[0165] Specifically, the label enhancement network layer includes a cross-modal semantic enhancement component, a feature pooling component, and a feature classification component. The label word vector matrix and the image semantic enhanced feature are respectively input into the cross-modal semantic enhancement component. In the cross-modal semantic enhancement component, the image semantic enhanced feature is weighted based on the label word vector matrix, and an image semantic weighted feature is obtained. The image semantic weighted feature is input into the feature pooling component, and the semantic feature classification result corresponding to the image semantic weighted feature is obtained through the feature pooling component. The image attribute enhanced feature is input into the feature classification component, and the attribute feature classification result corresponding to the image attribute enhanced feature is obtained through the feature classification component. The semantic feature classification result and the attribute feature classification result are fused to obtain a target classification result. The target category label of the image is determined in the category label set according to the target classification result.

[0166] The label word vector matrix includes at least two label word vectors; the at least two label word vectors include a label word vector A b and a label word vector A c , b and c are positive integers, b is less than or equal to the total number of the at least two label word vectors, and c is less than or equal to the total number of the at least two label word vectors; the cross-modal semantic enhancement component includes a label similarity subcomponent, a seventh fully connected layer subcomponent, and a semantic weighting subcomponent; the specific process of performing feature weighting on the image semantic enhanced feature based on the label word vector matrix can include: obtaining, by the label similarity subcomponent, a vector similarity between the label word vector A b and the label word vector A c , generating a label similarity matrix based on the obtained vector similarity; performing feature transformation on the image semantic enhanced feature by the seventh fully connected layer subcomponent to obtain a to-be-weighted image semantic enhanced feature; inputting the label similarity matrix and the to-be-weighted image semantic enhanced feature into the semantic weighting subcomponent; and performing feature weighting on the to-be-weighted image semantic enhanced feature based on the label similarity matrix in the semantic weighting subcomponent to obtain the image semantic weighted feature.

[0167] In combination with steps S102-S103, the image attribute initial feature is first enhanced by the cross-modal fusion network layer, and then the image semantic initial feature is enhanced by the intra-modal fusion network layer. After N rounds of circulation, the label word vector matrix, the enhanced image attribute feature (i.e., the image attribute enhanced feature), and the enhanced image semantic feature (i.e., the image semantic enhanced feature) are input into the label enhancement network layer. In the label enhancement network layer, the image semantic enhanced feature is weighted and enhanced by using the label correlation in the textual modal information. Please refer to Figure 7 , Figure 7 is a scene diagram of data processing provided by an embodiment of the present application. As shown in Figure 7 , the label enhancement network layer can include a cross-modal semantic enhancement component 80d, a feature pooling component, and a feature classification component. The business server inputs the label word vector matrix 40d and the image semantic enhanced feature 603a into the cross-modal semantic enhancement component 80d, which can include a label similarity subcomponent, a seventh fully connected layer subcomponent, and a semantic weighting subcomponent.

[0168] The business server obtains, by the label similarity subcomponent, a vector similarity between every two label word vectors in the label word vector matrix 40d (which can include the vector similarity of the label word vector itself, i.e., 1), as shown in Figure 8As shown in the example, the business server obtains the vector similarity between the label word vector 401d and the label word vector 402d, the vector similarity between the label word vector 401d and the label word vector 403d, and the vector similarity between the label word vector 401d and the label word vector 401d, and the same applies to the remaining label word vectors, which will not be described herein, so that an N*N label similarity matrix 80a can be obtained, where N is equal to the total number of category labels in the category label set, such as Figure 8 As shown in the example, N=3, that is, the label similarity matrix 80a is a 3*3 matrix. The embodiments of the present application do not limit the method for obtaining the vector similarity, which can be any similarity calculation method, such as the Euclidean distance, the Pearson correlation coefficient, and the cosine similarity algorithm.

[0169] Through the seventh full connection layer subcomponent, the business server performs feature transformation on the image semantic enhanced feature 603a to adjust the feature dimension thereof to the same dimension as the label similarity matrix 80a, and obtains a to-be-weighted image semantic enhanced feature 80b; further, the label similarity matrix 80a and the to-be-weighted image semantic enhanced feature 80b are input into a semantic weighting subcomponent 80c, in which the to-be-weighted image semantic enhanced feature 80b is weighted based on the label similarity matrix 80a, that is, the to-be-weighted image semantic enhanced feature is weighted and enhanced by using the correlation between the label word vectors, and an image semantic weighted feature 80c is obtained.

[0170] Further, the image semantic weighted feature 80c is input into a feature pooling component, which can be a global pooling layer in the embodiments of the present application. Through the feature pooling component, the business server obtains a semantic feature classification result 801e corresponding to the image semantic weighted feature 80c, which can be an N-dimensional probability vector. In addition, the business server inputs the image attribute enhanced feature 502a into a feature classification component, through which an attribute feature classification result 802e corresponding to the image attribute enhanced feature 502a is obtained, which can be an N-dimensional probability vector. Further, the business server fuses the semantic feature classification result 801e and the attribute feature classification result 802e to obtain a target classification result 803e, which can be represented as formula (1).

[0171] CLS final = CLS sem + CLS patch (1)

[0172] In formula (1), CLS sem represents the semantic feature classification result 801e, CLS patchThe attribute feature classification result 802e can be represented as CLS final The target classification result 803e can be represented as CLS

[0173] Please refer to Figure 7 The target classification result 803e can be represented as CLS

[0174] As can be seen from the above, the present scheme can assist users in accurately identifying the attributes and contents of pictures through cross-modal and intra-modal feature fusion enhancement operations. The cross-modal and intra-modal feature fusion enhancement operations can enhance the image features (including image semantic features and image attribute features) by using the correlation between the labels, and can enhance the image semantic features and the image attribute features within the modal, so as to assist users in accurately identifying the attributes and contents of pictures.

[0175] Please refer to Figure 8 , Figure 8 is a flowchart of a data processing method provided by an embodiment of the present application. The method can be executed by a business server (for example, the business server 100 shown in the above Figure 1 , the terminal device 200a shown in the above Figure 1 , or the business server and the terminal device can interactively execute the method. For ease of understanding, the embodiment of the present application takes the method executed by the business server as an example for description. As shown in the above Figure 8 , the method can at least include the following steps.

[0176] Step S201, obtaining a label word vector corresponding to each category label in a category label set, obtaining an image attribute initial feature corresponding to an image, and obtaining an image semantic initial feature corresponding to the image.

[0177] Step S202, obtaining an image recognition model, inputting the label word vector matrix and the image attribute initial feature into the image recognition model respectively; the image recognition model includes a cross-modal fusion network layer, and the cross-modal fusion network layer includes a cross-modal cross-attention fusion component and a feature weighting component.

[0178] Specifically, the image recognition model of the embodiment of the present application mainly includes three parts, which are a cross-modal fusion network layer between the image attribute initial feature and the label word vector matrix, an intra-modal fusion network layer between the image attribute feature and the image semantic initial feature, and a cross-modal semantic enhancement network layer (also referred to as a label enhancement network layer) between the image semantic enhanced feature and the label word vector matrix. Specifically, the cross-modal fusion network layer is to fuse the text modal information and the image attribute initial feature in the image modal, so as to enhance the image attribute initial feature; the intra-modal fusion network layer is to fuse the enhanced image attribute feature and the image semantic initial feature, so as to enhance the image semantic initial feature; and the cross-modal semantic enhancement network layer is to weight and enhance the image semantic enhanced feature by using the label correlation in the text modal information.

[0179] The specific implementation process of steps S201-S202 is described above Figure 3 The specific implementation process of steps S101-S102 in the corresponding embodiment is not described herein.

[0180] In step S203, the cross-modal cross-attention fusion component is used to fuse the label word vector matrix and the image attribute initial feature, so as to obtain the cross-modal association matrix associated with the label word vector matrix.

[0181] Specifically, the cross-modal cross-attention fusion component includes a full connection layer subcomponent, a feature point product subcomponent, a matrix scaling subcomponent, and a matrix normalization subcomponent. The image attribute initial feature is transformed by the full connection layer subcomponent to obtain the image attribute feature to be fused. The image attribute feature to be fused and the label word vector matrix are input into the feature point product subcomponent. The image attribute feature to be fused and the label word vector matrix are subjected to feature point product by the feature point product subcomponent to obtain the cross-modal association matrix to be scaled. The cross-modal association matrix to be scaled is input into the matrix scaling subcomponent, and the cross-modal association matrix to be scaled is subjected to feature scaling by the matrix scaling subcomponent to obtain the cross-modal association matrix to be normalized. The cross-modal association matrix to be normalized is input into the matrix normalization subcomponent, and the cross-modal association matrix to be normalized is subjected to feature normalization by the matrix normalization subcomponent to obtain the cross-modal association matrix associated with the label word vector matrix.

[0182] In the cross-modal cross-attention fusion component, the cross-modal association matrix is obtained by the cross-attention of the label word vector matrix and the image attribute initial feature by the business server. The feature fusion can be represented by the following formula (2).

[0183]

[0184] In formula (2), N Pdenotes the number of sub-images, N C denotes the total number of class labels in the class label set, T patch denotes the image attribute initial feature, the number of which corresponds to the number of sub-images, and the feature channel number and feature dimension of the image attribute initial feature are not limited in the embodiments of the present application and can be adjusted according to actual application scenarios; T label denotes the label word vector matrix, that is, composed of N C label word vectors; denotes the full connection layer subcomponent, which can also be understood as the weight parameters of a full connection layer, T patch The purpose of both is to align the feature dimension of the image attribute initial feature with the vector dimension corresponding to the label word vector, denotes the image attribute feature to be fused, the feature dimension of which is equal to the vector dimension corresponding to the label word vector; D(.) in formula (2) denotes a scaling dot product operation, which can include a feature dot product subcomponent and a matrix scaling subcomponent. First, the feature dot product subcomponent is used to perform feature dot product on the image attribute feature to be fused and the label word vector matrix T label , to obtain a to-be-scaled cross-modal correlation matrix. Then, the matrix scaling subcomponent is used to perform feature scaling on the to-be-scaled cross-modal correlation matrix to obtain a to-be-normalized cross-modal correlation matrix. The scaling factor is not limited in the embodiments of the present application and can be adjusted according to actual application scenarios, for example, the arithmetic square root of the vector dimension of the label word vector. is a normalization function, i∈N P ×N C , j represents (∑(N P ×N C )), which is equivalent to the matrix normalization subcomponent described above. The matrix normalization subcomponent is used to perform feature normalization on the to-be-normalized cross-modal correlation matrix to obtain a cross-modal correlation matrix M label associated with the label word vector matrix T inter .

[0185] In step S204, the cross-modal correlation matrix and the image attribute initial feature are respectively input into a feature weighting component. In the feature weighting component, the image attribute initial feature is weighted according to the cross-modal correlation matrix to obtain an image attribute feature.

[0186] ​Specifically, the feature weighting component comprises a full connection layer subcomponent, a weighted weight subcomponent, a weighted normalization subcomponent, and an attribute weighting subcomponent; the image attribute initial feature is subjected to feature transformation through the full connection layer subcomponent to obtain the image attribute feature to be fused; the cross-modal correlation matrix and the label word vector matrix are subjected to feature multiplication through the weighted weight subcomponent to obtain the weighted weight to be normalized for the image attribute feature to be fused; the weighted weight to be normalized is input into the weighted normalization subcomponent, and the weighted weight to be normalized is subjected to weight normalization through the weighted normalization subcomponent to obtain the feature weighting weight for the image attribute feature to be fused; the feature weighting weight and the image attribute feature to be fused are input into the attribute weighting subcomponent; in the attribute weighting subcomponent, the image attribute feature to be fused is subjected to feature weighting based on the feature weighting weight to obtain the image attribute feature.

[0187] In the feature weighting component, the cross-modal correlation matrix M patch is used by the business server to perform weighted enhancement on the image attribute initial feature T

[0188]

[0189] In the formula (3), the meaning of M is described in the formula (2), and M inter T label represents the weighted weight to be normalized obtained by the weighted weight subcomponent through feature multiplication on the cross-modal correlation matrix M inter and the label word vector matrix T label ; softmax is a normalization function, which is equivalent to the weighted normalization subcomponent described above, and softmax(M inter T label ) represents the feature weighting weight, the symbol "⊙" in the formula (3) represents dot multiplication, and the symbol "+" represents the attribute weighting subcomponent, through which the image attribute feature is obtained

[0190] In step S205, the image attribute feature is input into the first full connection layer subcomponent, the image attribute feature is subjected to feature transformation through the first full connection layer subcomponent to obtain the first image attribute feature to be fused; the image attribute feature is input into the second full connection layer subcomponent, the image attribute feature is subjected to feature transformation through the second full connection layer subcomponent to obtain the second image attribute feature to be fused; and the image attribute feature is input into the third full connection layer subcomponent, the image attribute feature is subjected to feature transformation through the third full connection layer subcomponent to obtain the image attribute feature to be enhanced.

[0191] ​Specifically, the cross-modal fusion network layer further comprises a cross-modal self-attention enhancement component, and the cross-modal self-attention enhancement component comprises a first full connection layer subcomponent, a second full connection layer subcomponent, a third full connection layer subcomponent and a cross-modal self-attention enhancement subcomponent.

[0192] The first full connection layer subcomponent, the second full connection layer subcomponent and the third full connection layer subcomponent are in a parallel relationship and are all used for feature transformation of the image attribute features. In addition to the function of aligning the vector dimension with the label word vector, the processing can also improve the representation ability and the generalization. The representation can be seen from the following step S206.

[0193] In step S206, the first to-be-fused image attribute feature, the second to-be-fused image attribute feature and the to-be-enhanced image attribute feature are respectively input into the cross-modal self-attention enhancement subcomponent. In the cross-modal self-attention enhancement subcomponent, the to-be-enhanced image attribute feature is enhanced by the first to-be-fused image attribute feature and the second to-be-fused image attribute feature to obtain an image attribute enhanced feature.

[0194] Specifically, the cross-modal self-attention enhancement subcomponent comprises a first feature transposition subcomponent, a first enhancement weight subcomponent, a first enhancement scaling subcomponent, a first enhancement normalization subcomponent and an attribute enhancement subcomponent. The second to-be-fused image attribute feature is transposed by the first feature transposition subcomponent to obtain a transposed image attribute feature corresponding to the second to-be-fused image attribute feature. The transposed image attribute feature and the first to-be-fused image attribute feature are respectively input into the first enhancement weight subcomponent. The first to-be-fused image attribute feature and the transposed image attribute feature are multiplied by the first enhancement weight subcomponent to obtain a first to-be-scaled enhancement weight for the to-be-enhanced image attribute feature. The first to-be-scaled enhancement weight is input into the first enhancement scaling subcomponent, and the first to-be-scaled enhancement weight is scaled by the first enhancement scaling subcomponent to obtain a first to-be-normalized enhancement weight. The first to-be-normalized enhancement weight is input into the first enhancement normalization subcomponent, and the first to-be-normalized enhancement weight is normalized by the first enhancement normalization subcomponent to obtain a first feature enhancement weight. The first feature enhancement weight and the to-be-enhanced image attribute feature are respectively input into the attribute enhancement subcomponent. In the attribute enhancement subcomponent, the to-be-enhanced image attribute feature is enhanced based on the first feature enhancement weight to obtain an image attribute enhanced feature.

[0195] In this step, the self-attention is used to further enhance the connection between the image attribute features and the representation ability, and a further enhanced image attribute enhanced feature is obtained. The process can be represented by the following formula (4).

[0196]

[0197] wherein, the formula (4) is denotes a first full connection layer subcomponent, which can be understood as a weight parameter of a full connection layer, denotes a second full connection layer subcomponent, which can be understood as a weight parameter of another full connection layer, denotes a third full connection layer subcomponent, which can be understood as a weight parameter of still another full connection layer; denotes a first to-be-fused image attribute feature, denotes a second to-be-fused image attribute feature, denotes a to-be-enhanced image attribute feature; the symbol "T" denotes a first feature transposition subcomponent, denotes a transposed image attribute feature, denotes a first to-be-scaled enhancement weight, D h denotes a scaling factor in the first enhancement scaling subcomponent, and embodiments of the present application do not limit D h , which can be set according to an actual application scenario and can be an arithmetic square root of a vector dimension of a label word vector; softmax denotes a first enhancement normalization subcomponent, and softmax() denotes a first feature enhancement weight, denotes an image attribute enhanced feature.

[0198] In step S207, the image semantic initial feature is input to the intra-modal self-attention enhancement component, the image semantic initial feature is subjected to feature enhancement through the intra-modal self-attention enhancement component, and a first image semantic feature is obtained.

[0199] Specifically, the intra-modal self-attention enhancement component includes a fourth full connection layer subcomponent, a fifth full connection layer subcomponent, a sixth full connection layer subcomponent, and an intra-modal self-attention enhancement subcomponent; the image semantic initial feature is subjected to feature transformation through the fourth full connection layer subcomponent, and a first to-be-fused image semantic feature is obtained; the image semantic initial feature is subjected to feature transformation through the fifth full connection layer subcomponent, and a second to-be-fused image semantic feature is obtained; the image semantic initial feature is subjected to feature transformation through the sixth full connection layer subcomponent, and a to-be-enhanced image semantic feature is obtained; the first to-be-fused image semantic feature, the second to-be-fused image semantic feature, and the to-be-enhanced image semantic feature are input to the intra-modal self-attention enhancement subcomponent respectively; in the intra-modal self-attention enhancement subcomponent, the to-be-enhanced image semantic feature is subjected to feature enhancement through the first to-be-fused image semantic feature and the second to-be-fused image semantic feature, and a first image semantic feature is obtained.

[0200] The intra-modal self-attention enhancement sub-component includes a second feature transposition sub-component, a second enhancement weight sub-component, a second enhancement scaling sub-component, a second enhancement normalization sub-component, and a first semantic enhancement sub-component. In the intra-modal self-attention enhancement sub-component, the specific process of obtaining the first image semantic feature can include: performing feature transposition on the second to-be-fused image semantic feature through the second feature transposition sub-component to obtain a transposed image semantic feature corresponding to the second to-be-fused image semantic feature; inputting the transposed image semantic feature and the first to-be-fused image semantic feature into the second enhancement weight sub-component; performing feature multiplication on the first to-be-fused image semantic feature and the transposed image semantic feature through the second enhancement weight sub-component to obtain a second to-be-scaled enhancement weight for the to-be-enhanced image semantic feature; inputting the second to-be-scaled enhancement weight into the second enhancement scaling sub-component to perform weight scaling on the second to-be-scaled enhancement weight through the second enhancement scaling sub-component to obtain a second to-be-normalized enhancement weight; inputting the second to-be-normalized enhancement weight into the second enhancement normalization sub-component to perform weight normalization on the second to-be-normalized enhancement weight through the second enhancement normalization sub-component to obtain a second feature enhancement weight; inputting the second feature enhancement weight and the to-be-enhanced image semantic feature into the first semantic enhancement sub-component; and performing feature enhancement on the to-be-enhanced image semantic feature based on the second feature enhancement weight in the first semantic enhancement sub-component to obtain the first image semantic feature.

[0201] The fourth fully connected layer sub-component, the fifth fully connected layer sub-component, and the sixth fully connected layer sub-component are in a parallel relationship and are all used for feature transformation of the image attribute feature. In addition to the function of aligning the vector dimension with the label word vector, the processing can also improve the representation ability and the generalization.

[0202] In this step, for the image attribute feature and the image semantic initial feature T sem , the image semantic initial feature can be first enhanced by self-attention to obtain an enhanced image semantic feature, i.e., a first image semantic feature. The process can be represented by the following formula (5).

[0203]

[0204] In formula (5), W4 represents the fourth fully connected layer sub-component, which can be understood as a weight parameter of a fully connected layer, represents the fifth fully connected layer sub-component, which can be understood as a weight parameter of another fully connected layer, represents the sixth fully connected layer sub-component, which can be understood as a weight parameter of another fully connected layer; and D h, and softmax can refer to the explanation of formula (4) above, the functions in the two formulas are consistent, the difference is that the data processed is not the same; denotes the semantic feature of the first image to be fused, denotes the semantic feature of the second image to be fused, denotes the semantic feature of the image to be enhanced; denotes the transposed image semantic feature, denotes the semantic feature of the first image; it can be understood that the processing process and the representation process corresponding to formula (4) and formula (5) are basically similar, the difference is only that the data processed is not the same, so the representation of part of formula (5) is not described here, please refer to the description of formula (4) above.

[0205] In step S208, the image semantic initial feature and the image attribute feature are input into the intra-modal cross-attention enhancement component; the image semantic initial feature and the image attribute feature are fused by the intra-modal cross-attention fusion component to obtain the second image semantic feature.

[0206] Specifically, the intra-modal cross-attention fusion component includes a fourth full connection layer sub-component, a second full connection layer sub-component, a third full connection layer sub-component, and an intra-modal cross-attention enhancement sub-component; the image semantic initial feature is transformed by the fourth full connection layer sub-component to obtain the first image semantic feature to be fused; the image attribute feature is transformed by the second full connection layer sub-component to obtain the second image attribute feature to be fused; the image attribute feature is transformed by the third full connection layer sub-component to obtain the image attribute feature to be enhanced; the first image semantic feature to be fused, the second image attribute feature to be fused, and the image attribute feature to be enhanced are input into the intra-modal cross-attention enhancement sub-component; in the intra-modal cross-attention enhancement sub-component, the image attribute feature to be enhanced is enhanced by the first image semantic feature to be fused and the second image attribute feature to be fused to obtain the second image semantic feature.

[0207] The intramodal cross-attention enhancement subcomponent includes a first feature transpose subcomponent, a third enhancement weight subcomponent, a third enhancement scaling subcomponent, a third enhancement normalization subcomponent, and a second semantic enhancement subcomponent. The specific process of obtaining the second image semantic features within the intramodal cross-attention enhancement subcomponent may include: transposing the second image attribute features to be fused using the first feature transpose subcomponent to obtain the transposed image attribute features corresponding to the second image attribute features; inputting the transposed image attribute features and the first image semantic features to be fused to the third enhancement weight subcomponent; and performing feature multiplication on the first image semantic features and the transposed image attribute features using the third enhancement weight subcomponent to obtain... The process involves obtaining a third scaling weight for the image attribute features to be enhanced; inputting this third scaling weight into a third scaling subcomponent to scale the weights, resulting in a third normalized weight; inputting this normalized weight into a third normalization subcomponent to normalize the weights, resulting in a third feature enhancement weight; inputting the third feature enhancement weight and the image attribute features to be enhanced into a second semantic enhancement subcomponent; and performing feature enhancement on the image attribute features based on the third feature enhancement weight to obtain the second image semantic features.

[0208] In this step, the business server uses image attribute features to perform intramodal feature enhancement on the initial semantic features of the image through cross attention. This process can be represented by the following formula (6).

[0209]

[0210] Among them, in formula (6) The second image semantic feature is represented by the symbol. The other symbols and expressions can be found in the descriptions of formulas (4) and (5) above. They will not be repeated here. It is understood that the difference between formula (6) and formula (5) is only that the data processed is different, but the processing process is the same.

[0211] Step S209: Input the first image semantic features and the second image semantic features into the feature fusion component respectively; through the feature fusion component, perform feature fusion on the first image semantic features and the second image semantic features to obtain image semantic enhancement features.

[0212] Specifically, the business server fuses the image semantic feature enhanced by self-attention (i.e., the first image semantic feature) and the image semantic feature enhanced by cross-attention (i.e., the second image semantic feature) in the form of addition to obtain the final enhanced image semantic feature, i.e., the image semantic enhanced feature. This process can be represented by formula (7).

[0213]

[0214] In formula (7), the first image semantic feature is represented by The second image semantic feature is represented by The image semantic enhanced feature is represented by

[0215] In step S210, the target class label of the image is determined in the class label set according to the label word vector matrix, the image semantic enhanced feature, and the image attribute enhanced feature.

[0216] The specific implementation process of step S210 is described above Figure 3 in the embodiment corresponding to step S104, which will not be described here.

[0217] As described above, the present application not only performs cross-modal feature fusion on the text modal feature (label word vector matrix) and the image modal feature (image attribute initial feature) to enhance the image modal feature and obtain the image attribute enhanced feature, but also performs intra-modal feature fusion on the image attribute feature and the image semantic initial feature to enhance the image semantic initial feature and obtain the image semantic enhanced feature. Through cross-modal feature fusion and intra-modal feature fusion, multi-modal information can be integrated to generate features including multi-modal information, which can improve the feature representation capability of the image and thus improve the recognition accuracy of the target class label of the image.

[0218] Further, please refer to Figure 9 , Figure 9 is a structural schematic diagram of a data processing apparatus provided by an embodiment of the present application. The data processing apparatus can be a computer program (including program code) running in a computer device, for example, the data processing apparatus is an application software. The apparatus can be used to execute the corresponding steps in the method provided by the embodiments of the present application. As Figure 9 shown, the data processing apparatus 1 can include an acquisition data module 11, a first fusion module 12, a second fusion module 13, and a determination label module 14.

[0219] The acquisition data module 11 is configured to acquire the label word vector corresponding to each class label in the class label set, acquire the image attribute initial feature corresponding to the image, and acquire the image semantic initial feature corresponding to the image.​

[0220] The first fusion module 12 is configured to perform feature fusion on the label word vector matrix and the image attribute initial feature to obtain an image attribute feature, and perform self-attention enhancement on the image attribute feature to obtain an image attribute enhanced feature. The label word vector matrix is generated based on a label word vector corresponding to each category label.

[0221] The second fusion module 13 is configured to perform feature fusion on the image attribute feature and an image semantic initial feature to obtain an image semantic enhanced feature.

[0222] The determining label module 14 is configured to determine a target category label of the image from the category label set according to the label word vector matrix, the image semantic enhanced feature, and the image attribute enhanced feature.

[0223] The specific function implementation of the data obtaining module 11, the first fusion module 12, the second fusion module 13, and the determining label module 14 can be referred to the steps S101-S104 in the above Figure 3 corresponding embodiments, which will not be repeated here.

[0224] Please refer to Figure 9 The first fusion module 12 can include a first obtaining unit 121, a first fusion unit 122, and a first weighting unit 123.

[0225] The first obtaining unit 121 is configured to obtain an image recognition model, and input the label word vector matrix and the image attribute initial feature into the image recognition model. The image recognition model includes a cross-modal fusion network layer, and the cross-modal fusion network layer includes a cross-modal cross-attention fusion component and a feature weighting component.

[0226] The first fusion unit 122 is configured to perform feature fusion on the label word vector matrix and the image attribute initial feature through the cross-modal cross-attention fusion component to obtain a cross-modal association matrix associated with the label word vector matrix.

[0227] The first weighting unit 123 is configured to input the cross-modal association matrix and the image attribute initial feature into the feature weighting component.

[0228] The first weighting unit 123 is further configured to perform feature weighting on the image attribute initial feature according to the cross-modal association matrix in the feature weighting component to obtain the image attribute feature.

[0229] The specific function implementation of the first obtaining unit 121, the first fusion unit 122, and the first weighting unit 123 can be referred to the steps S202-S204 in the above Figure 8 corresponding embodiments, which will not be repeated here.

[0230] Referring back to Figure 9 The cross-modal cross-attention fusion component includes a full connection layer subcomponent, a feature dot product subcomponent, a matrix scaling subcomponent, and a matrix normalization subcomponent.

[0231] The first fusion unit 122 can include a first transformation subunit 1221, a first input subunit 1222, a second input subunit 1223, and a first generation subunit 1224.

[0232] The first transformation subunit 1221 is configured to perform feature transformation on the image attribute initial feature through the full connection layer subcomponent to obtain a to-be-fused image attribute feature.

[0233] The first input subunit 1222 is configured to input the to-be-fused image attribute feature and the label word vector matrix to the feature dot product subcomponent, respectively.

[0234] The first input subunit 1222 is further configured to perform feature dot product on the to-be-fused image attribute feature and the label word vector matrix through the feature dot product subcomponent to obtain a to-be-scaled cross-modal correlation matrix.

[0235] The second input subunit 1223 is configured to input the to-be-scaled cross-modal correlation matrix to the matrix scaling subcomponent, and perform feature scaling on the to-be-scaled cross-modal correlation matrix through the matrix scaling subcomponent to obtain a to-be-normalized cross-modal correlation matrix.

[0236] The first generation subunit 1224 is configured to input the to-be-normalized cross-modal correlation matrix to the matrix normalization subcomponent, and perform feature normalization on the to-be-normalized cross-modal correlation matrix through the matrix normalization subcomponent to obtain a cross-modal correlation matrix associated with the label word vector matrix.

[0237] The specific function implementation manners of the first transformation subunit 1221, the first input subunit 1222, the second input subunit 1223, and the first generation subunit 1224 can be referred to the above Figure 8 Corresponding to step S203 in the above embodiment, details are not repeated here.

[0238] Referring back to Figure 9 The feature weighting component includes a full connection layer subcomponent, a weighted weight subcomponent, a weighted normalization subcomponent, and an attribute weighting subcomponent.

[0239] The first weighting unit 123 can include a second transformation subunit 1231, a second generation subunit 1232, a third input subunit 1233, and a fourth input subunit 1234.

[0240] The second transformation subunit 1231 is configured to perform feature transformation on the initial image attribute feature through the full connection layer subassembly, to obtain an image attribute feature to be fused;

[0241] The second generation subunit 1232 is configured to perform feature multiplication on the cross-modal correlation matrix and the label word vector matrix through the weighting weight subassembly, to obtain a weighting weight to be normalized for the image attribute feature to be fused;

[0242] The third input subunit 1233 is configured to input the weighting weight to be normalized into the weighted normalization subassembly, and perform weight normalization on the weighting weight to be normalized through the weighted normalization subassembly, to obtain a feature weighting weight for the image attribute feature to be fused.

[0243] The fourth input subunit 1234 is configured to input the feature weighting weight and the image attribute feature to be fused into the attribute weighting subassembly respectively.

[0244] The fourth input subunit 1234 is further configured to perform feature weighting on the image attribute feature to be fused based on the feature weighting weight in the attribute weighting subassembly, to obtain the image attribute feature.

[0245] The specific function implementation manners of the second transformation subunit 1231, the second generation subunit 1232, the third input subunit 1233, and the fourth input subunit 1234 can be referred to the above Figure 8 Corresponding to step S204 in the embodiment, details are not described herein.

[0246] Please refer to Figure 9 The cross-modal fusion network layer further includes a cross-modal self-attention enhancement assembly, and the cross-modal self-attention enhancement assembly includes a first full connection layer subassembly, a second full connection layer subassembly, a third full connection layer subassembly, and a cross-modal self-attention enhancement subassembly.

[0247] The first fusion module 12 can include a first input unit 124, a second input unit 125, a third input unit 126, and a first enhancement unit 127.

[0248] The first input unit 124 is configured to input the image attribute feature into the first full connection layer subassembly, and perform feature transformation on the image attribute feature through the first full connection layer subassembly, to obtain a first image attribute feature to be fused.

[0249] The second input unit 125 is configured to input the image attribute feature into the second full connection layer subassembly, and perform feature transformation on the image attribute feature through the second full connection layer subassembly, to obtain a second image attribute feature to be fused.

[0250] The third input unit 126 is configured to input the image attribute feature into the third full connection layer subcomponent, perform feature transformation on the image attribute feature through the third full connection layer subcomponent, and obtain the image attribute feature to be enhanced.

[0251] The first enhancement unit 127 is configured to input the first image attribute feature to be fused, the second image attribute feature to be fused, and the image attribute feature to be enhanced into the cross-modal self-attention enhancement subcomponent respectively.

[0252] The first enhancement unit 127 is further configured to perform feature enhancement on the image attribute feature to be enhanced through the first image attribute feature to be fused and the second image attribute feature to be fused in the cross-modal self-attention enhancement subcomponent, and obtain the image attribute enhanced feature.

[0253] The specific function implementation manners of the first input unit 124, the second input unit 125, the third input unit 126, and the first enhancement unit 127 can be referred to the above Figure 8 The steps S205-S206 in the corresponding embodiments will not be repeated here.

[0254] Please refer to Figure 9 The cross-modal self-attention enhancement subcomponent includes a first feature transpose subcomponent, a first enhancement weight subcomponent, a first enhancement scaling subcomponent, a first enhancement normalization subcomponent, and an attribute enhancement subcomponent.

[0255] The first enhancement unit 127 can include a third generation subunit 1271, a fourth generation subunit 1272, a fifth input subunit 1273, a fifth generation subunit 1274, and a sixth input subunit 1275.

[0256] The third generation subunit 1271 is configured to perform feature transposition on the second image attribute feature to be fused through the first feature transpose subcomponent, and obtain the transposed image attribute feature corresponding to the second image attribute feature to be fused:

[0257] The fourth generation subunit 1272 is configured to input the transposed image attribute feature and the first image attribute feature to be fused into the first enhancement weight subcomponent respectively.

[0258] The fourth generation subunit 1272 is further configured to perform feature multiplication on the first image attribute feature to be fused and the transposed image attribute feature through the first enhancement weight subcomponent, and obtain the first scaling enhancement weight for the image attribute feature to be enhanced.

[0259] The fifth input subunit 1273 is configured to input the first scaling enhancement weight into the first enhancement scaling subcomponent, perform weight scaling on the first scaling enhancement weight through the first enhancement scaling subcomponent, and obtain the first normalization enhancement weight.

[0260] The fifth generation sub-unit 1274 is configured to input the first to-be-normalized enhancement weight into the first enhancement normalization sub-component, perform weight normalization on the first to-be-normalized enhancement weight through the first enhancement normalization sub-component, and obtain a first feature enhancement weight;

[0261] The sixth input sub-unit 1275 is configured to input the first feature enhancement weight and the to-be-enhanced image attribute feature into the attribute enhancement sub-component, respectively.

[0262] The sixth input sub-unit 1275 is further configured to perform feature enhancement on the to-be-enhanced image attribute feature based on the first feature enhancement weight in the attribute enhancement sub-component, and obtain an image attribute enhanced feature.

[0263] The specific function implementation manners of the third generation sub-unit 1271, the fourth generation sub-unit 1272, the fifth input sub-unit 1273, the fifth generation sub-unit 1274, and the sixth input sub-unit 1275 can be referred to the above Figure 8 Corresponding to step S206 in the embodiment, details are not described herein again.

[0264] Please refer to Figure 9 The image recognition model further includes an intra-modal fusion network layer, and the intra-modal fusion network layer includes an intra-modal self-attention enhancement component, an intra-modal cross-attention fusion component, and a feature fusion component.

[0265] The second fusion module 13 can include a second enhancement unit 131, a second fusion unit 132, and a third fusion unit 133.

[0266] The second enhancement unit 131 is configured to input the image semantic initial feature into the intra-modal self-attention enhancement component, perform feature enhancement on the image semantic initial feature through the intra-modal self-attention enhancement component, and obtain a first image semantic feature.

[0267] The second fusion unit 132 is configured to input the image semantic initial feature and the image attribute feature into the intra-modal cross-attention enhancement component, respectively.

[0268] The second fusion unit 132 is further configured to perform feature fusion on the image semantic initial feature and the image attribute feature through the intra-modal cross-attention fusion component, and obtain a second image semantic feature.

[0269] The third fusion unit 133 is configured to input the first image semantic feature and the second image semantic feature into the feature fusion component, respectively.

[0270] The third fusion unit 133 is further configured to perform feature fusion on the first image semantic feature and the second image semantic feature through the feature fusion component, and obtain an image semantic enhanced feature.

[0271] The specific function implementation manners of the second enhancement unit 131, the second fusion unit 132, and the third fusion unit 133 can be referred to the above Figure 8 The steps S207-S209 in the corresponding embodiment will not be repeated here.

[0272] Please refer to Figure 9 The intra-modal self-attention enhancement component includes a fourth full connection layer subcomponent, a fifth full connection layer subcomponent, a sixth full connection layer subcomponent, and an intra-modal self-attention enhancement subcomponent.

[0273] The second enhancement unit 131 can include a third transformation subunit 1311, a fourth transformation subunit 1312, a fifth transformation subunit 1313, and a seventh input subunit 1314.

[0274] The third transformation subunit 1311 is configured to perform feature transformation on the image semantic initial feature through the fourth full connection layer subcomponent to obtain first to-be-fused image semantic features.

[0275] The fourth transformation subunit 1312 is configured to perform feature transformation on the image semantic initial feature through the fifth full connection layer subcomponent to obtain second to-be-fused image semantic features.

[0276] The fifth transformation subunit 1313 is configured to perform feature transformation on the image semantic initial feature through the sixth full connection layer subcomponent to obtain to-be-enhanced image semantic features.

[0277] The seventh input subunit 1314 is configured to input the first to-be-fused image semantic features, the second to-be-fused image semantic features, and the to-be-enhanced image semantic features into the intra-modal self-attention enhancement subcomponent, respectively.

[0278] The seventh input subunit 1314 is further configured to perform feature enhancement on the to-be-enhanced image semantic features through the first to-be-fused image semantic features and the second to-be-fused image semantic features in the intra-modal self-attention enhancement subcomponent to obtain first image semantic features.

[0279] The specific function implementation manners of the third transformation subunit 1311, the fourth transformation subunit 1312, the fifth transformation subunit 1313, and the seventh input subunit 1314 can be referred to the above Figure 8 The step S207 in the corresponding embodiment will not be repeated here.

[0280] Please refer to Figure 9 The intra-modal self-attention enhancement subcomponent includes a second feature transpose subcomponent, a second enhancement weight subcomponent, a second enhancement scaling subcomponent, a second enhancement normalization subcomponent, and a first semantic enhancement subcomponent.

[0281] The seventh input subunit 1314 can include a first transposition subunit 13141, a first product subunit 13142, a first scaling subunit 13143, a first normalization subunit 13144, and a first enhancement subunit 13145.

[0282] The first transposition subunit 13141 is configured to perform feature transposition on the second to-be-fused image semantic feature through a second feature transposition subcomponent, to obtain a transposed image semantic feature corresponding to the second to-be-fused image semantic feature.

[0283] The first product subunit 13142 is configured to input the transposed image semantic feature and the first to-be-fused image semantic feature into a second enhancement weight subcomponent, respectively.

[0284] The first product subunit 13142 is further configured to perform feature multiplication on the first to-be-fused image semantic feature and the transposed image semantic feature through the second enhancement weight subcomponent, to obtain a second to-be-scaled enhancement weight for the to-be-enhanced image semantic feature.

[0285] The first scaling subunit 13143 is configured to input the second to-be-scaled enhancement weight into a second enhancement scaling subcomponent, and perform weight scaling on the second to-be-scaled enhancement weight through the second enhancement scaling subcomponent, to obtain a second to-be-normalized enhancement weight.

[0286] The first normalization subunit 13144 is configured to input the second to-be-normalized enhancement weight into a second enhancement normalization subcomponent, and perform weight normalization on the second to-be-normalized enhancement weight through the second enhancement normalization subcomponent, to obtain a second feature enhancement weight.

[0287] The first enhancement subunit 13145 is configured to input the second feature enhancement weight and the to-be-enhanced image semantic feature into a first semantic enhancement subcomponent, respectively.

[0288] The first enhancement subunit 13145 is further configured to perform feature enhancement on the to-be-enhanced image semantic feature based on the second feature enhancement weight in the first semantic enhancement subcomponent, to obtain the first image semantic feature.

[0289] The specific function implementation manners of the first transposition subunit 13141, the first product subunit 13142, the first scaling subunit 13143, the first normalization subunit 13144, and the first enhancement subunit 13145 can be referred to the above Figure 8 Corresponding to step S207 in the above

[0290] Please refer to Figure 9The intra-modal cross-attention fusion component includes a fourth full connection layer subcomponent, a second full connection layer subcomponent, a third full connection layer subcomponent, and an intra-modal cross-attention enhancement subcomponent.

[0291] The second fusion unit 132 can include a sixth transformation subunit 1321, a seventh transformation subunit 1322, an eighth transformation subunit 1323, and an eighth input subunit 1324.

[0292] The sixth transformation subunit 1321 is configured to perform feature transformation on the image semantic initial feature through the fourth full connection layer subcomponent to obtain first to-be-fused image semantic features.

[0293] The seventh transformation subunit 1322 is configured to perform feature transformation on the image attribute feature through the second full connection layer subcomponent to obtain second to-be-fused image attribute features.

[0294] The eighth transformation subunit 1323 is configured to perform feature transformation on the image attribute feature through the third full connection layer subcomponent to obtain to-be-enhanced image attribute features.

[0295] The eighth input subunit 1324 is configured to input the first to-be-fused image semantic features, the second to-be-fused image attribute features, and the to-be-enhanced image attribute features into the intra-modal cross-attention enhancement subcomponent.

[0296] The eighth input subunit 1324 is further configured to perform feature enhancement on the to-be-enhanced image attribute features through the first to-be-fused image semantic features and the second to-be-fused image attribute features in the intra-modal cross-attention enhancement subcomponent to obtain second image semantic features.

[0297] The specific function implementation manners of the sixth transformation subunit 1321, the seventh transformation subunit 1322, the eighth transformation subunit 1323, and the eighth input subunit 1324 can be referred to the above Figure 8 The step S208 in the corresponding embodiment will not be repeated here.

[0298] Please refer to Figure 9 The intra-modal cross-attention enhancement subcomponent includes a first feature transpose subcomponent, a third enhancement weight subcomponent, a third enhancement scaling subcomponent, a third enhancement normalization subcomponent, and a second semantic enhancement subcomponent.

[0299] The eighth input subunit 1324 can include a second transpose subunit 13241, a second product subunit 13242, a second scaling subunit 13243, a second normalization subunit 13244, and a second enhancement subunit 13245.

[0300] The second transposition subunit 13241 is configured to perform feature transposition on the second to-be-fused image attribute feature by using the first feature transposition subassembly, to obtain a transposed image attribute feature corresponding to the second to-be-fused image attribute feature.

[0301] The second product subunit 13242 is configured to input the transposed image attribute feature and the first to-be-fused image semantic feature into the third enhanced weight subassembly respectively.

[0302] The second product subunit 13242 is further configured to perform feature multiplication on the first to-be-fused image semantic feature and the transposed image attribute feature by using the third enhanced weight subassembly, to obtain a third to-be-scaled enhanced weight for the to-be-enhanced image attribute feature.

[0303] The second scaling subunit 13243 is configured to input the third to-be-scaled enhanced weight into the third enhanced scaling subassembly, and perform weight scaling on the third to-be-scaled enhanced weight by using the third enhanced scaling subassembly, to obtain a third to-be-normalized enhanced weight.

[0304] The second normalization subunit 13244 is configured to input the third to-be-normalized enhanced weight into the third enhanced normalization subassembly, and perform weight normalization on the third to-be-normalized enhanced weight by using the third enhanced normalization subassembly, to obtain a third feature enhanced weight.

[0305] The second enhancement subunit 13245 is configured to input the third feature enhanced weight and the to-be-enhanced image attribute feature into the second semantic enhancement subassembly respectively.

[0306] The second enhancement subunit 13245 is further configured to perform feature enhancement on the to-be-enhanced image attribute feature based on the third feature enhanced weight in the second semantic enhancement subassembly, to obtain a second image semantic feature.

[0307] The specific function implementation manners of the second transposition subunit 13241, the second product subunit 13242, the second scaling subunit 13243, the second normalization subunit 13244, and the second enhancement subunit 13245 can be referred to the above Figure 3 The step S208 in the corresponding embodiment will not be repeated here.

[0308] Please refer to Figure 9 The image recognition model further includes a label enhancement network layer, and the label enhancement network layer includes a cross-modal semantic enhancement component, a feature pooling component, and a feature classification component.

[0309] The determination label module 14 can include a second weighting unit 141, a second acquisition unit 142, a third acquisition unit 143, and a fourth fusion unit 144.

[0310] The second weighting unit 141 is configured to input the label word vector matrix and the image semantic enhanced feature into the cross-modal semantic enhancement component respectively.

[0311] The second weighting unit 141 is further configured to perform feature weighting on the image semantic enhanced feature based on the label word vector matrix in the cross-modal semantic enhancement component, to obtain image semantic weighted features.

[0312] The second acquisition unit 142 is configured to input the image semantic weighted features into the feature pooling component, and obtain a semantic feature classification result corresponding to the image semantic weighted features through the feature pooling component.

[0313] The third acquisition unit 143 is configured to input the image attribute enhanced feature into the feature classification component, and obtain an attribute feature classification result corresponding to the image attribute enhanced feature through the feature classification component.

[0314] The fourth fusion unit 144 is configured to perform result fusion on the semantic feature classification result and the attribute feature classification result, to obtain a target classification result.

[0315] The fourth fusion unit 144 is further configured to determine a target class label of the image in the class label set according to the target classification result.

[0316] The specific function implementation manners of the second weighting unit 141, the second acquisition unit 142, the third acquisition unit 143 and the fourth fusion unit 144 can be referred to the above Figure 3 The step S104 in the corresponding embodiment will not be repeated here.

[0317] Please refer to Figure 9 The label word vector matrix includes at least two label word vectors; the at least two label word vectors include a label word vector A b and a label word vector A c b and c are positive integers, b is less than or equal to the total number of the at least two label word vectors, and c is less than or equal to the total number of the at least two label word vectors; the cross-modal semantic enhancement component includes a label similarity sub-component, a seventh full connection layer sub-component and a semantic weighting sub-component.

[0318] The second weighting unit 141 can include a sixth generation sub-unit 1411, a ninth transformation sub-unit 1412 and a seventh generation sub-unit 1413.

[0319] The sixth generation sub-unit 1411 is configured to obtain a vector similarity between the label word vector A b and the label word vector A c through the label similarity sub-component, and generate a label similarity matrix based on the obtained vector similarity.

[0320] The ninth transformation subunit 1412 is configured to perform feature transformation on the image semantic enhanced feature by the seventh full connection layer subcomponent to obtain a to-be-weighted image semantic enhanced feature;

[0321] The seventh generation subunit 1413 is configured to input the label similarity matrix and the to-be-weighted image semantic enhanced feature into the semantic weighting subcomponent respectively.

[0322] The seventh generation subunit 1413 is further configured to perform feature weighting on the to-be-weighted image semantic enhanced feature based on the label similarity matrix in the semantic weighting subcomponent to obtain an image semantic weighted feature.

[0323] The specific function implementation manners of the sixth generation subunit 1411, the ninth transformation subunit 1412, and the seventh generation subunit 1413 can be referred to the above Figure 3 The step S104 in the corresponding embodiment will not be repeated here.

[0324] Please refer to Figure 10 The data acquisition module 11 can include a fourth acquisition unit 111, a fifth acquisition unit 112, a sixth acquisition unit 113, and a seventh acquisition unit 114.

[0325] The fourth acquisition unit 111 is configured to acquire a sample image set; each sample image in the sample image set carries a category label.

[0326] The fourth acquisition unit 111 is further configured to generate a category label set according to the category label carried by each sample image respectively.

[0327] The fifth acquisition unit 112 is configured to acquire a word vector model, and acquire a label word vector corresponding to each category label in the category label set respectively according to the word vector model.

[0328] The sixth acquisition unit 113 is configured to acquire an image, perform cropping processing on the image to acquire D sub-images, and acquire image attribute initial features corresponding to the D sub-images respectively; D is a positive integer, the image sizes corresponding to the D sub-images are all the same, and the image sizes corresponding to the D sub-images are all less than or equal to an image size corresponding to the image.

[0329] The seventh acquisition unit 114 is configured to acquire an image feature extraction model, and acquire an image semantic initial feature corresponding to the image according to the image feature extraction model.

[0330] The specific function implementation manners of the fourth acquisition unit 111, the fifth acquisition unit 112, the sixth acquisition unit 113, and the seventh acquisition unit 114 can be referred to the above Figure 10 The step S101 in the corresponding embodiment will not be repeated here.

[0331] It can be known from the above that the application not only performs cross-modal feature fusion on the text modal feature (the label word vector matrix) and the image modal feature (the image attribute initial feature) to enhance the image modal feature and obtain the image attribute enhanced feature, but also performs intra-modal feature fusion on the image attribute feature and the image semantic initial feature to enhance the image semantic initial feature and obtain the image semantic enhanced feature. Through the cross-modal feature fusion and the intra-modal feature fusion, multi-modal information can be comprehensively fused to generate a feature including the multi-modal information, and then the feature representation capability of the image can be improved, so that the recognition accuracy of the target class label of the image can be improved.

[0332] Further, please refer to Figure 10 , Figure 10 is a structural schematic diagram of a computer device provided by an embodiment of the application. As shown in Figure 10 , the computer device 1000 can include at least one processor 1001 such as a CPU, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between the components. The user interface 1003 can include a display screen (Display) and a keyboard (Keyboard), and the network interface 1004 can optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 can be a high-speed RAM memory or a non-volatile memory such as at least one disk memory. The memory 1005 can also be at least one storage device located away from the aforementioned processor 1001. As shown in Figure 3 , the memory 1005 as a computer storage medium can include an operating system, a network communication module, a user interface module, and a device control application program.

[0333] In the computer device 1000 shown in Figure 8 , the network interface 1004 can provide network communication functions; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application program stored in the memory 1005 to realize the following functions.

[0334] obtain the label word vector corresponding to each class label in the class label set, obtain the image attribute initial feature corresponding to the image, and obtain the image semantic initial feature corresponding to the image;

[0335] perform feature fusion on the label word vector matrix and the image attribute initial feature to obtain the image attribute feature, perform self-attention enhancement on the image attribute feature to obtain the image attribute enhanced feature, and generate the image attribute enhanced feature based on the label word vector corresponding to each class label.

[0336] perform feature fusion on the image attribute features and the image semantic initial features to obtain image semantic enhanced features;

[0337] According to the label word vector matrix, the image semantic enhanced features, and the image attribute enhanced features, a target class label of the image is determined in a class label set.

[0338] It should be understood that the computer device 1000 described in the embodiments of the present application can perform the foregoing Figure 9 and Figure 3 the description of the data processing method in the respective corresponding embodiments, and can also perform the foregoing Figure 8 the description of the data processing apparatus 1 in the respective corresponding embodiments, which will not be described here again. In addition, the beneficial effects of using the same method will not be described again.

[0339] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the data processing method provided by each step in Figure 3 and Figure 8 will be implemented. For details, please refer to the implementation manners provided by each step in Figure 3 and Figure 8 which will not be described here again. In addition, the beneficial effects of using the same method will not be described again.

[0340] The computer readable storage medium described above can be an internal storage unit of the data processing apparatus or the computer device provided in any of the foregoing embodiments, such as a hard disk or a memory of the computer device. The computer readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the computer readable storage medium can include both the internal storage unit and the external storage device of the computer device. The computer readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer readable storage medium can also be used to temporarily store data that has been output or will be output.

[0341] The embodiments of the present application also provide a computer program product or a computer program. The computer program product or the computer program includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium. The processor executes the computer instructions, so that the computer device can perform the foregoingFigure 1 and Figure 1 The description of the data processing method in the respective embodiments will not be repeated here. In addition, the beneficial effects of using the same method will not be repeated.

[0342] The terms "first", "second", and the like in the description and claims of the present application and the accompanying drawings are used to distinguish different objects, and are not used to describe a particular order. In addition, the term "comprising" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, device, product, or apparatus.

[0343] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software, or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, each example has been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0344] The method and related apparatus provided by the embodiments of the present application are described with reference to the method flowchart and / or structural schematic diagram provided by the embodiments of the present application. Each flow and / or block in the method flowchart and / or structural schematic diagram, and the combination of the flows and / or blocks in the flowchart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a means for implementing the functions specified in the flowchart and / or block diagram. Figure 1 The functions specified in one flow or multiple flows and / or structural schematic Figure 1 The functions specified in one flow or multiple flows and / or structural schematic Figure 1 The functions specified in one flow or multiple flows and / or structural schematic ​The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the operations specified in the flowchart block or blocks. ​ The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the operations specified in the flowchart block or blocks.

[0345] The above disclosure is merely preferred embodiments of the present application and is not intended in any way to limit the scope of the present application. Equivalents are therefore intended which are within the scope of the following claims, so that any equivalent variations are still within the scope of the present application.

Claims

1. A data processing method, characterized by, The method comprises the following steps: obtaining a label word vector corresponding to each category label in a category label set, obtaining an initial image attribute feature corresponding to an image, and an initial image semantic feature corresponding to the image; performing feature fusion on the label word vector matrix and the initial image attribute feature to obtain an image attribute feature, and performing self-attention enhancement on the image attribute feature to obtain an image attribute enhanced feature; the label word vector matrix is generated based on the label word vector corresponding to each category label; performing feature fusion on the image attribute feature and the initial image semantic feature to obtain an image semantic enhanced feature; determining a target category label of the image in the category label set according to the label word vector matrix, the image semantic enhanced feature, and the image attribute enhanced feature.

2. The method of claim 1, wherein, the feature fusion on the label word vector matrix and the initial image attribute feature to obtain an image attribute feature comprises: obtaining an image recognition model, and inputting the label word vector matrix and the initial image attribute feature into the image recognition model respectively; the image recognition model comprises a cross-modal fusion network layer, and the cross-modal fusion network layer comprises a cross-modal cross-attention fusion component and a feature weighting component; performing feature fusion on the label word vector matrix and the initial image attribute feature through the cross-modal cross-attention fusion component to obtain a cross-modal association matrix associated with the label word vector matrix; inputting the cross-modal association matrix and the initial image attribute feature into the feature weighting component respectively; in the feature weighting component, the initial image attribute feature is weighted according to the cross-modal association matrix to obtain the image attribute feature.

3. The method of claim 2, wherein, the cross-modal cross-attention fusion component comprises a full connection layer subcomponent, a feature point product subcomponent, a matrix scaling subcomponent, and a matrix normalization subcomponent; the feature fusion on the label word vector matrix and the initial image attribute feature through the cross-modal cross-attention fusion component to obtain the cross-modal association matrix associated with the label word vector matrix comprises: performing feature transformation on the initial image attribute feature through the full connection layer subcomponent to obtain a to-be-fused image attribute feature; inputting the to-be-fused image attribute feature and the label word vector matrix into the feature point product subcomponent respectively; performing feature point product on the to-be-fused image attribute feature and the label word vector matrix through the feature point product subcomponent to obtain a to-be-scaled cross-modal association matrix; inputting the to-be-scaled cross-modal association matrix into the matrix scaling subcomponent, and performing feature scaling on the to-be-scaled cross-modal association matrix through the matrix scaling subcomponent to obtain a to-be-normalized cross-modal association matrix; inputting the to-be-normalized cross-modal association matrix into the matrix normalization subcomponent, and performing feature normalization on the to-be-normalized cross-modal association matrix through the matrix normalization subcomponent to obtain the cross-modal association matrix associated with the label word vector matrix.

4. The method of claim 2, wherein, The feature weighting component includes a full connection layer subcomponent, a weighted weight subcomponent, a weighted normalization subcomponent, and an attribute weighting subcomponent; In the feature weighting component, the image attribute initial features are subjected to feature weighting according to the cross-modal correlation matrix to obtain the image attribute features. The image attribute initial features are subjected to feature transformation by the full connection layer subcomponent to obtain to-be-fused image attribute features; The cross-modal correlation matrix and the label word vector matrix are subjected to feature multiplication by the weighted weight subcomponent to obtain to-be-normalized weighted weights for the to-be-fused image attribute features; The to-be-normalized weighted weights are input into the weighted normalization subcomponent, and the to-be-normalized weighted weights are subjected to weight normalization by the weighted normalization subcomponent to obtain feature weighting weights for the to-be-fused image attribute features; The feature weighting weights and the to-be-fused image attribute features are input into the attribute weighting subcomponent, respectively; In the attribute weighting subcomponent, the to-be-fused image attribute features are subjected to feature weighting based on the feature weighting weights to obtain the image attribute features.

5. The method of claim 2, wherein, The cross-modal fusion network layer further includes a cross-modal self-attention enhancement component, and the cross-modal self-attention enhancement component includes a first full connection layer subcomponent, a second full connection layer subcomponent, a third full connection layer subcomponent, and a cross-modal self-attention enhancement subcomponent; The image attribute features are subjected to self-attention enhancement to obtain image attribute enhanced features, including: The image attribute features are input into the first full connection layer subcomponent, and the image attribute features are subjected to feature transformation by the first full connection layer subcomponent to obtain first to-be-fused image attribute features; The image attribute features are input into the second full connection layer subcomponent, and the image attribute features are subjected to feature transformation by the second full connection layer subcomponent to obtain second to-be-fused image attribute features; The image attribute features are input into the third full connection layer subcomponent, and the image attribute features are subjected to feature transformation by the third full connection layer subcomponent to obtain to-be-enhanced image attribute features; The first to-be-fused image attribute features, the second to-be-fused image attribute features, and the to-be-enhanced image attribute features are input into the cross-modal self-attention enhancement subcomponent, respectively; In the cross-modal self-attention enhancement subcomponent, the to-be-enhanced image attribute features are subjected to feature enhancement by the first to-be-fused image attribute features and the second to-be-fused image attribute features to obtain the image attribute enhanced features.

6. The method of claim 5, wherein, The cross-modal self-attention enhancement subcomponent includes a first feature transpose subcomponent, a first enhancement weight subcomponent, a first enhancement scaling subcomponent, a first enhancement normalization subcomponent, and an attribute enhancement subcomponent; In the cross-modal self-attention enhancement subcomponent, the to-be-enhanced image attribute features are subjected to feature enhancement by the first to-be-fused image attribute features and the second to-be-fused image attribute features to obtain the image attribute enhanced features, including: The first feature transposition subcomponent is configured to perform feature transposition on the second image attribute feature to be fused to obtain a transposed image attribute feature corresponding to the second image attribute feature to be fused; The transposed image attribute feature and the first image attribute feature to be fused are input into the first enhanced weight subcomponent respectively; The first enhanced weight subcomponent is configured to perform feature multiplication on the first image attribute feature to be fused and the transposed image attribute feature to obtain first to-be-scaled enhanced weights for the image attribute feature to be enhanced; The first to-be-scaled enhanced weights are input into the first enhanced scaling subcomponent, and the first enhanced scaling subcomponent is configured to perform weight scaling on the first to-be-scaled enhanced weights to obtain first to-be-normalized enhanced weights; The first to-be-normalized enhanced weights are input into the first enhanced normalization subcomponent, and the first enhanced normalization subcomponent is configured to perform weight normalization on the first to-be-normalized enhanced weights to obtain first feature enhanced weights; The first feature enhanced weights and the image attribute feature to be enhanced are input into the attribute enhancement subcomponent respectively; In the attribute enhancement subcomponent, the image attribute feature to be enhanced is enhanced based on the first feature enhanced weights to obtain the image attribute enhanced feature.

7. The method of claim 2, wherein, The image recognition model further includes an intra-modal fusion network layer, and the intra-modal fusion network layer includes an intra-modal self-attention enhancement component, an intra-modal cross-attention fusion component, and a feature fusion component; The image attribute feature and the image semantic initial feature are fused to obtain an image semantic enhanced feature, including: The image semantic initial feature is input into the intra-modal self-attention enhancement component, and the intra-modal self-attention enhancement component is configured to enhance the image semantic initial feature to obtain a first image semantic feature; The image semantic initial feature and the image attribute feature are input into the intra-modal cross-attention enhancement component respectively; The intra-modal cross-attention fusion component is configured to fuse the image semantic initial feature and the image attribute feature to obtain a second image semantic feature; The first image semantic feature and the second image semantic feature are input into the feature fusion component respectively; The feature fusion component is configured to fuse the first image semantic feature and the second image semantic feature to obtain the image semantic enhanced feature.

8. The method of claim 7, wherein, The intra-modal self-attention enhancement component includes a fourth full connection layer subcomponent, a fifth full connection layer subcomponent, a sixth full connection layer subcomponent, and an intra-modal self-attention enhancement subcomponent; The intra-modal self-attention enhancement component is configured to enhance the image semantic initial feature to obtain a first image semantic feature, including: The fourth full connection layer subcomponent is configured to perform feature transformation on the image semantic initial feature to obtain a first image semantic feature to be fused; The fifth full connection layer subcomponent is configured to perform feature transformation on the image semantic initial feature to obtain a second image semantic feature to be fused; The sixth full connection layer subcomponent is configured to perform feature transformation on the image semantic initial feature to obtain a third image semantic feature to be fused. The image semantic initial feature is subjected to feature transformation by the sixth full connection layer subassembly to obtain an image semantic feature to be enhanced; The first image semantic feature to be fused, the second image semantic feature to be fused and the image semantic feature to be enhanced are input into the intra-modal self-attention enhancement subassembly respectively; In the intra-modal self-attention enhancement subassembly, the image semantic feature to be enhanced is subjected to feature enhancement by the first image semantic feature to be fused and the second image semantic feature to be fused to obtain the first image semantic feature.

9. The method of claim 8, wherein, The intra-modal self-attention enhancement subassembly comprises a second feature transposition subassembly, a second enhancement weight subassembly, a second enhancement scaling subassembly, a second enhancement normalization subassembly and a first semantic enhancement subassembly; In the intra-modal self-attention enhancement subassembly, the image semantic feature to be enhanced is subjected to feature enhancement by the first image semantic feature to be fused and the second image semantic feature to be fused to obtain the first image semantic feature, comprising: The second image semantic feature to be fused is subjected to feature transposition by the second feature transposition subassembly to obtain a transposed image semantic feature corresponding to the second image semantic feature to be fused; The transposed image semantic feature and the first image semantic feature to be fused are input into the second enhancement weight subassembly respectively; The first image semantic feature to be fused and the transposed image semantic feature are subjected to feature multiplication by the second enhancement weight subassembly to obtain second scaling enhancement weights for the image semantic feature to be enhanced; The second scaling enhancement weights are input into the second enhancement scaling subassembly, and the second scaling enhancement weights are subjected to weight scaling by the second enhancement scaling subassembly to obtain second normalization enhancement weights; The second normalization enhancement weights are input into the second enhancement normalization subassembly, and the second normalization enhancement weights are subjected to weight normalization by the second enhancement normalization subassembly to obtain second feature enhancement weights; The second feature enhancement weights and the image semantic feature to be enhanced are input into the first semantic enhancement subassembly respectively; In the first semantic enhancement subassembly, the image semantic feature to be enhanced is subjected to feature enhancement based on the second feature enhancement weights to obtain the first image semantic feature.

10. The method of claim 7, wherein, The intra-modal cross-attention fusion assembly comprises a fourth full connection layer subassembly, a second full connection layer subassembly, a third full connection layer subassembly and an intra-modal cross-attention enhancement subassembly; The image semantic initial feature and the image attribute feature are subjected to feature fusion by the intra-modal cross-attention fusion assembly to obtain a second image semantic feature, comprising: The image semantic initial feature is subjected to feature transformation by the fourth full connection layer subassembly to obtain a first image semantic feature to be fused; The image attribute feature is subjected to feature transformation by the second full connection layer subassembly to obtain a second image attribute feature to be fused; The image attribute feature is subjected to feature transformation by the third full connection layer subcomponent to obtain an image attribute feature to be enhanced; The first image semantic feature to be fused, the second image attribute feature to be fused, and the image attribute feature to be enhanced are input into the intra-modal cross-attention enhancement subcomponent respectively; In the intra-modal cross-attention enhancement subcomponent, the image attribute feature to be enhanced is subjected to feature enhancement based on the first image semantic feature to be fused and the second image attribute feature to be fused to obtain the second image semantic feature.

11. The method of claim 10, wherein, The intra-modal cross-attention enhancement subcomponent comprises a first feature transposition subcomponent, a third enhancement weight subcomponent, a third enhancement scaling subcomponent, a third enhancement normalization subcomponent, and a second semantic enhancement subcomponent; In the intra-modal cross-attention enhancement subcomponent, the image attribute feature to be enhanced is subjected to feature enhancement based on the first image semantic feature to be fused and the second image attribute feature to be fused to obtain the second image semantic feature, comprising: The second image attribute feature to be fused is subjected to feature transposition by the first feature transposition subcomponent to obtain a transposed image attribute feature corresponding to the second image attribute feature to be fused; The transposed image attribute feature and the first image semantic feature to be fused are input into the third enhancement weight subcomponent respectively; The first image semantic feature to be fused and the transposed image attribute feature are subjected to feature multiplication by the third enhancement weight subcomponent to obtain third to-be-scaled enhancement weights for the image attribute feature to be enhanced; The third to-be-scaled enhancement weights are input into the third enhancement scaling subcomponent, and the third to-be-scaled enhancement weights are subjected to weight scaling by the third enhancement scaling subcomponent to obtain third to-be-normalized enhancement weights; The third to-be-normalized enhancement weights are input into the third enhancement normalization subcomponent, and the third to-be-normalized enhancement weights are subjected to weight normalization by the third enhancement normalization subcomponent to obtain third feature enhancement weights; The third feature enhancement weights and the image attribute feature to be enhanced are input into the second semantic enhancement subcomponent respectively; In the second semantic enhancement subcomponent, the image attribute feature to be enhanced is subjected to feature enhancement based on the third feature enhancement weights to obtain the second image semantic feature.

12. The method of claim 2, wherein, The image recognition model further comprises a label enhancement network layer, and the label enhancement network layer comprises a cross-modal semantic enhancement component, a feature pooling component, and a feature classification component; The target class label of the image is determined from the label word vector matrix, the image semantic enhanced feature, and the image attribute enhanced feature in the class label set, comprising: The label word vector matrix and the image semantic enhanced feature are input into the cross-modal semantic enhancement component respectively; In the cross-modal semantic enhancement component, the image semantic enhanced feature is subjected to feature weighting based on the label word vector matrix to obtain an image semantic weighted feature; input the image semantic weighted feature to the feature pooling component, and obtain a semantic feature classification result corresponding to the image semantic weighted feature through the feature pooling component; input the image attribute enhanced feature to the feature classification component, and obtain an attribute feature classification result corresponding to the image attribute enhanced feature through the feature classification component; fuse the semantic feature classification result and the attribute feature classification result to obtain a target classification result; determine the target category label of the image in the category label set according to the target classification result.

13. The method of claim 12, wherein, The label word vector matrix comprises at least two label word vectors; the at least two label word vectors comprise a label word vector A b and a label word vector A c , b and c are positive integers, b is less than or equal to the total number of the at least two label word vectors, and c is less than or equal to the total number of the at least two label word vectors. The cross-modal semantic enhancement component includes a label similarity sub-component, a seventh fully connected layer sub-component, and a semantic weighting sub-component. In the cross-modal semantic enhancement component, the image semantic enhanced feature is weighted based on the label word vector matrix to obtain an image semantic weighted feature, including: The label word vector A is obtained through the label similarity subcomponent b The vector similarity between the label word vector A and the label word vector B is obtained c A label similarity matrix is generated based on the obtained vector similarity. The seventh fully connected layer sub-component is used to perform feature transformation on the image semantic enhanced feature to obtain a to-be-weighted image semantic enhanced feature; The label similarity matrix and the to-be-weighted image semantic enhanced feature are input into the semantic weighting sub-component; In the semantic weighting sub-component, the to-be-weighted image semantic enhanced feature is weighted based on the label similarity matrix to obtain the image semantic weighted feature.

14. The method of claim 1, wherein, The acquisition of the label word vector corresponding to each category label in the category label set, the acquisition of the image attribute initial feature corresponding to the image, and the acquisition of the image semantic initial feature corresponding to the image include: A sample image set is acquired. Each sample image in the sample image set carries a category label. The category label set is generated according to the category label carried by each sample image. A word vector model is acquired, and the label word vector corresponding to each category label in the category label set is acquired according to the word vector model. The image is acquired, the image is cropped to obtain D sub-images, and the image attribute initial feature corresponding to each of the D sub-images is acquired; D is a positive integer, the image sizes corresponding to the D sub-images are the same, and the image sizes corresponding to the D sub-images are all less than or equal to the image size corresponding to the image. An image feature extraction model is acquired, and the image semantic initial feature corresponding to the image is acquired according to the image feature extraction model.

15. A computer device, comprising: It includes: a processor, a memory, and a network interface; The processor is connected with the memory and the network interface, wherein the network interface is configured to provide data communication function, the memory is configured to store computer program, and the processor is configured to call the computer program to enable the computer device to execute the method in any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is adapted to be loaded and executed by the processor to enable the computer device with the processor to execute the method in any one of claims 1 to 14.

17. A computer program product, characterised in that, The computer program product comprises computer instructions stored in a computer readable storage medium, said computer instructions being adapted to be read and executed by a processor to cause a computer device having said processor to perform the method of any of claims 1-14.

Images