Resource classification method, device, electronic device and storage medium
By performing two fusion processing of the text features and image features of the resource, using convolutional neural network and multi-head attention mechanism, the target fusion features are generated to improve the accuracy of resource classification, and the failure of multimodal features in the existing technology is solved. Make full use of dependencies.
Patent Information
- Application Number
- CN202110975771.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-24
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-08-24
AI Technical Summary
In the prior art In resource classification, the simple stacking method of multimodal features fails to fully utilize the dependence between different modal features, resulting in inaccurate classification results.
By obtaining the text features and image features of the pending resources, two fusion processing is performed, including first interactive fusion through the convolutional neural network, and then further fusion using the multi-head attention mechanism or the two-way transmission neural network to generate target fusion features to determine the classification result.
The accuracy of classification results is improved, and the characteristics of resources are expressed more accurately through deep interaction and expression of dependence between different modal features.
Smart Images

Figure CN113886572B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a resource classification method, device, electronic device and storage medium. Background Art
[0002] With the development of computer technology, more and more resources are available on the Internet. In some scenarios, it is necessary to classify the resources on the Internet for practical needs. For example, short video platforms need to classify uploaded videos, and e-commerce platforms need to accurately classify the products listed by merchants.
[0003] In the related technology, in the process of classifying resources, in order to fully obtain the relevant features of the resources, it is generally achieved by separately extracting multiple modal features of the resources (such as image features, text features, voice features, etc.), and then simply stacking the multiple modal features extracted separately to achieve resource classification.
[0004] However, the current method of simply stacking multiple modal features extracted separately actually regards the features of resources in different modalities as independent features. However, the features of resources in different modalities may be interdependent. Therefore, the current method does not truly achieve the multimodal effect, resulting in inaccurate classification results. Summary of the invention
[0005] The present disclosure provides a resource classification method, device, electronic device and storage medium to at least solve the problem of inaccurate classification results in related technologies. The technical solution of the present disclosure is as follows:
[0006] According to a first aspect of an embodiment of the present disclosure, a resource classification method is provided, comprising:
[0007] Obtain text features and image features of the resource to be processed;
[0008] Performing a first fusion process on the text features and the image features to obtain fusion features of the resource to be processed;
[0009] Performing a second fusion process on the fused features to obtain target fused features of the resource to be processed;
[0010] Based on the target fusion feature, a classification result of the resource to be processed is determined.
[0011] In one of the embodiments, the first fusion processing is performed on the text features and the image features to obtain the fusion features of the resource to be processed, including: splicing the feature matrix corresponding to the text features and the feature matrix corresponding to the image features in a set direction to obtain a spliced feature matrix; interactively fusing the text features and the image features in the spliced feature matrix to obtain the fusion features of the resource to be processed.
[0012] In one of the embodiments, interactively fusing the text features and image features in the splicing feature matrix to obtain the fused features of the resource to be processed includes: inputting the splicing feature matrix into a convolutional neural network, interactively fusing the text features and image features in the splicing feature matrix through the convolutional neural network to obtain the output of the convolutional neural network, and using the output as the fused features of the resource to be processed.
[0013] In one embodiment, the second fusion processing is performed on the fused features to obtain the target fused features of the resource to be processed, including: using a multi-head attention mechanism to obtain a first feature in the fused features and a second feature for characterizing the relationship between the first feature and the context; fusing the first feature and the second feature to obtain the target fused features of the resource to be processed, wherein the fused features include multiple first feature elements.
[0014] In one of the embodiments, the second fusion processing is performed on the fused features to obtain the target fused features of the resource to be processed, including: inputting the fused features into a bidirectional transmission neural network, obtaining a first feature in the fused features and a second feature for characterizing the relationship between the first feature and the context through the bidirectional transmission neural network, and fusing the first feature and the second feature to obtain the target fused features of the resource to be processed.
[0015] In one of the embodiments, determining the classification result of the resource to be processed based on the target fusion feature includes: classifying the target fusion feature to obtain multiple candidate classification results of the resource to be processed, each of the candidate classification results having a corresponding classification value; and taking the candidate classification result with the highest classification value among the multiple candidate classification results as the classification result of the resource to be processed.
[0016] In one of the embodiments, the obtaining of text features and image features of the resource to be processed includes: obtaining at least one frame of image of the resource to be processed, encoding the at least one frame of image to obtain encoded image features; obtaining text content of the resource to be processed, encoding the text content to obtain encoded text features.
[0017] In one embodiment, the encoding processing of the at least one frame of image to obtain encoded image features includes: performing two-dimensional encoding on the at least one frame of image through an image encoder to obtain encoded image features, and the image features have a corresponding feature matrix; the encoding processing of the text content to obtain encoded text features includes: performing one-dimensional encoding on the text content through a text encoder to obtain an encoded one-dimensional vector; based on the feature matrix corresponding to the image features, converting the one-dimensional vector into a corresponding two-dimensional feature matrix, and using the two-dimensional feature matrix as the text features of the text content.
[0018] According to a second aspect of an embodiment of the present disclosure, there is provided a resource classification device, including:
[0019] A feature acquisition module is configured to acquire text features and image features of a resource to be processed;
[0020] A first fusion processing module is configured to perform a first fusion processing on the text feature and the image feature to obtain a fusion feature of the resource to be processed;
[0021] A second fusion processing module is configured to perform a second fusion processing on the fusion feature to obtain a target fusion feature of the resource to be processed;
[0022] The classification result determination module is configured to determine the classification result of the resource to be processed based on the target fusion feature.
[0023] In one of the embodiments, the first fusion processing module includes: a splicing unit, configured to perform splicing of the feature matrix corresponding to the text feature and the feature matrix corresponding to the image feature in a set direction to obtain a spliced feature matrix; a fusion unit, configured to perform interactive fusion of the text features and the image features in the spliced feature matrix to obtain the fusion features of the resource to be processed.
[0024] In one embodiment, the fusion unit is configured to execute: inputting the splicing feature matrix into a convolutional neural network, interactively fusing the text features and image features in the splicing feature matrix through the convolutional neural network, obtaining the output of the convolutional neural network, and using the output as the fusion feature of the resource to be processed.
[0025] In one embodiment, the second fusion processing module is configured to perform: using a multi-head attention mechanism to obtain a first feature in the fusion feature and a second feature used to characterize the relationship between the first feature and the context; fusing the first feature and the second feature to obtain a target fusion feature of the resource to be processed.
[0026] In one of the embodiments, the second fusion processing module is configured to execute: inputting the fused feature into a bidirectional transmission neural network, obtaining a first feature in the fused feature and a second feature used to characterize the relationship between the first feature and the context through the bidirectional transmission neural network, and fusing the first feature and the second feature to obtain a target fused feature of the resource to be processed.
[0027] In one of the embodiments, the classification result determination module is configured to execute: classify the target fusion features to obtain multiple candidate classification results of the resources to be processed, each of the candidate classification results having a corresponding classification value; and use the candidate classification result with the highest classification value among the multiple candidate classification results as the classification result of the resources to be processed.
[0028] In one of the embodiments, the feature acquisition module includes: an image feature acquisition unit, configured to execute acquisition of at least one frame of image of the resource to be processed, encode the at least one frame of image, and obtain the encoded image features; a text feature acquisition unit, configured to execute acquisition of text content of the resource to be processed, encode the text content, and obtain the encoded text features.
[0029] In one of the embodiments, the image feature acquisition unit is configured to execute: two-dimensionally encode the at least one frame of image through an image encoder to obtain the encoded image features, and the image features have a corresponding feature matrix; the text feature acquisition unit is configured to execute: one-dimensionally encode the text content through a text encoder to obtain the encoded one-dimensional vector; based on the feature matrix corresponding to the image features, convert the one-dimensional vector into a corresponding two-dimensional feature matrix, and use the two-dimensional feature matrix as the text features of the text content.
[0030] According to a third aspect of an embodiment of the present disclosure, a computer device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions so that the computer device executes the resource classification method described in any one of the embodiments of the first aspect.
[0031] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of a computer device, the computer device can execute the resource classification method described in any one of the embodiments of the first aspect.
[0032] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, the program product comprising a computer program, the computer program being stored in a readable storage medium, at least one processor of a device reading and executing the computer program from the readable storage medium, so that the device executes the resource classification method described in any one of the embodiments of the first aspect.
[0033] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: by acquiring the text features and image features of the resource to be processed, performing a first fusion process on the text features and image features to obtain the fusion features of the resource to be processed, and performing a second fusion process on the fusion features to obtain the target fusion features of the resource to be processed, and then determining the classification result of the resource to be processed based on the target fusion features. Since the target fusion features for the final classification process are obtained after performing two fusion processes on the text features and image features of the resource to be processed, the text features and image features under different modalities can interact deeply, which can not only reflect the features of the resource to be processed in each modality, but also reflect the dependency between the features in different modalities. Therefore, the target fusion features after two fusions can better express the characteristics of the resource to be processed, thereby improving the accuracy of the classification results.
[0034] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0036] Figure 1 The present invention is a diagram showing an application environment of a resource classification method according to an exemplary embodiment.
[0037] Figure 2 The figure is a flow chart of a resource classification method according to an exemplary embodiment.
[0038] Figure 3 is a schematic diagram of a feature acquisition step according to an exemplary embodiment.
[0039] Figure 4 is a schematic diagram showing a first fusion processing step according to an exemplary embodiment.
[0040] Figure 5 is a schematic diagram of a second fusion processing step according to an exemplary embodiment.
[0041] Figure 6The figure is a schematic diagram showing a step of determining a classification result according to an exemplary embodiment.
[0042] Figure 7 The figure is a block diagram of a resource classification device according to an exemplary embodiment.
[0043] Figure 8 The invention is a block diagram of a computer device according to an exemplary embodiment.
[0044] Fig. 9 It is a block diagram of a computer device according to another exemplary embodiment. DETAILED DESCRIPTION
[0045] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.
[0046] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0047] The resource classification method provided by the present disclosure can be applied to Figure 1 In the application environment shown. The computer device 110 obtains the text features and image features of the resource to be processed; performs a first fusion process on the text features and image features of the resource to be processed to obtain the fusion features of the resource to be processed; then performs a second fusion process on the fusion features to obtain the target fusion features of the resource to be processed; and then determines the classification result of the resource to be processed based on the target fusion features. Specifically, the computer device 110 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. Of course, the computer device 110 can also be implemented using an independent server or a server cluster consisting of multiple servers.
[0048] Figure 2 is a flow chart of a resource classification method according to an exemplary embodiment. Figure 2 As shown, this method is used to Figure 1 The computer device 110 includes the following steps.
[0049] In step S210, text features and image features of the resource to be processed are obtained.
[0050] Among them, the resources to be processed refer to the resources to be classified. For example, it can be a video that needs to be classified in a short video platform, or it can be a commodity that needs to be classified in an e-commerce platform. Text features and image features are features used to describe the detailed content of the resources to be processed based on the resources to be processed. Specifically, if the resource to be processed is a video, its corresponding image features can be image features obtained based on the video frame, and its corresponding text features can be related text features extracted based on the subtitles in the video or based on the voice-to-text conversion in the video. If the resource to be processed is a commodity, its corresponding image features can be image features obtained based on the picture or photo of the commodity, and its corresponding text features can be related text features extracted based on the relevant description information of the commodity. In this embodiment, when it is necessary to classify the resources to be processed, the text features and image features of the resources to be processed are first obtained based on the relevant content of the resources to be processed.
[0051] In step S220, a first fusion process is performed on the text features and the image features to obtain fusion features of the resource to be processed.
[0052] The first fusion processing is a fusion processing based on convolution. The fusion feature is a feature obtained by performing the first fusion processing on the text feature and the image feature so as to interactively fuse the text feature and the image feature. Specifically, the first fusion processing is performed on the text feature and the image feature of the resource to be processed by using convolution, so as to obtain the fusion feature of the resource to be processed.
[0053] In step S230, a second fusion process is performed on the fused features to obtain target fused features of the resources to be processed.
[0054] The second fusion processing is a fusion processing based on a multi-head attention mechanism. The target fusion feature is a feature obtained by further interactive fusion of the features after the second fusion processing based on the above-mentioned fusion features. In this embodiment, the second fusion processing is further performed on the above-mentioned fusion features by adopting a multi-head attention mechanism, so as to obtain the target fusion feature of the resource to be processed.
[0055] In step S240, the classification result of the resource to be processed is determined based on the target fusion feature.
[0056] The classification result refers to the classification of the resources to be processed obtained after the classification process is performed on the resources to be processed, that is, determining what category the resources to be processed belong to. Specifically, the computer device performs classification processing based on the target fusion features obtained after two fusion processes of the text features and image features of the resources to be processed, thereby obtaining the classification result of the resources to be processed.
[0057] In the above resource classification method, by obtaining the text features and image features of the resource to be processed, the text features and image features are subjected to a first fusion process to obtain the fusion features of the resource to be processed, and the fusion features are subjected to a second fusion process to obtain the target fusion features of the resource to be processed, and then based on the target fusion features, the classification result of the resource to be processed is determined. Since the target fusion features for the final classification process are obtained after two fusion processes of the text features and image features of the resource to be processed, the text features and image features under different modalities can interact deeply, which can not only reflect the features of the resource to be processed in each modality, but also reflect the dependency between the features in different modalities. Therefore, the target fusion features after two fusions can better express the characteristics of the resource to be processed, thereby improving the accuracy of the classification results.
[0058] In an exemplary embodiment, if Figure 3 As shown, in step S210, the text features and image features of the resource to be processed are obtained, which can be specifically achieved by the following steps:
[0059] In step S212, at least one frame of image of the resource to be processed is obtained, and encoding processing is performed on the at least one frame of image to obtain encoded image features.
[0060] Among them, the frame image refers to an image that describes the resource to be processed. If the resource to be processed is a video, the frame image can be each frame image that constitutes the video, or it can be a partial frame image extracted based on each frame image of the video. If the resource to be processed is a commodity, the frame image can be the commodity image corresponding to the commodity. The image feature is the feature extracted based on the image of the resource to be processed, that is, the feature of the resource to be processed in the image mode. In this embodiment, by encoding at least one frame image of the resource to be processed, the image feature of the encoded resource to be processed is obtained, so as to realize the feature extraction of the resource to be processed in the image mode.
[0061] In an exemplary embodiment, when encoding the frame image of the resource to be processed, the input frame image can be two-dimensionally encoded by an image encoder, thereby outputting a corresponding two-dimensional feature matrix, and the two-dimensional feature matrix is the image feature after encoding the resource to be processed. Among them, the image encoder can be implemented based on a convolutional neural network (CNN) or a recurrent neural network (RNN). In this embodiment, the image features of the resource to be processed can be efficiently extracted by the image encoder, thereby improving the efficiency of resource classification.
[0062] In step S214, the text content of the resource to be processed is obtained, and the text content is encoded to obtain encoded text features.
[0063] Among them, text content refers to text information describing the resource to be processed. If the resource to be processed is a video, the text content can be the subtitles in the video or the relevant text information obtained after converting the speech in the video to text. If the resource to be processed is a commodity, the text content can be the relevant text information extracted based on the relevant description information of the commodity, such as the commodity name, commodity introduction, etc. The text feature is the feature extracted based on the text content of the resource to be processed, that is, the feature of the resource to be processed in the text mode.
[0064] In this embodiment, by encoding at least one frame of the resource to be processed, image features of the resource to be processed are obtained, and by obtaining the text content of the resource to be processed and encoding the text content, text features of the resource to be processed are obtained, so as to realize feature extraction of the resource to be processed in image modality and feature extraction in text modality.
[0065] In an exemplary embodiment, when the text content of the resource to be processed is encoded, the text content of the input can be specifically encoded in one dimension by a text encoder to obtain a one-dimensional vector after encoding, and based on the feature matrix corresponding to the image feature, the one-dimensional vector is converted into a corresponding two-dimensional feature matrix, and the two-dimensional feature matrix is used as the text feature of the text content. Wherein, the text encoder can be based on a deep bidirectional encoder model (BidirectionalEncoder Representations from Transformers, referred to as BERT) implementation. Specifically, the deep bidirectional encoder model outputs a one-dimensional vector. In this embodiment, based on the dimension of the two-dimensional feature matrix corresponding to the image feature, the one-dimensional vector after encoding the text content is converted to obtain a two-dimensional feature matrix with the same dimension as the image feature after the conversion, and the two-dimensional feature matrix after the conversion is the text feature after the resource to be processed is encoded. In this embodiment, the image features of the resource to be processed can be efficiently extracted by the image encoder, and the text features of the resource to be processed can be efficiently and comprehensively extracted by the text encoder. By converting the one-dimensional vector of the text content into a two-dimensional feature matrix with the same dimension as the image feature, it is convenient for the subsequent fusion of text features and image features, and then the accuracy of resource classification can be improved.
[0066] In an exemplary embodiment, if Figure 4 As shown, in step S220, a first fusion process is performed on the text features and the image features to obtain the fusion features of the resource to be processed, which can be specifically achieved by the following steps:
[0067] In step S222, the feature matrix corresponding to the text feature and the feature matrix corresponding to the image feature are spliced in a set direction to obtain a spliced feature matrix.
[0068] Due to the above processing, the feature matrix corresponding to the text features and the feature matrix corresponding to the image features are both two-dimensional feature matrices with the same dimension. Therefore, the set direction is the row direction or column direction of the pre-set feature matrix. For example, if the pre-set splicing direction is the row direction, assuming that the image feature after encoding the resource to be processed is a 14×64-dimensional feature matrix, if the one-dimensional vector after encoding the text content is 768-dimensional, then based on the dimension of the image feature, the one-dimensional vector of the text content is converted into the corresponding two-dimensional feature matrix, that is, converted into a two-dimensional feature matrix of 12×64 dimensions with the same column direction. Then, the 14×64-dimensional feature matrix corresponding to the image features and the 12×64-dimensional feature matrix corresponding to the text content are spliced in the row direction to obtain a spliced feature matrix of 26×64 dimensions.
[0069] In step S224, the text features and image features in the splicing feature matrix are interactively fused to obtain fused features of the resource to be processed.
[0070] Among them, interactive fusion can be processing such as dimensional expression and feature fusion. The fusion feature is a fusion feature matrix obtained by interactively fusing the text features and image features in the splicing feature matrix. In this embodiment, the computer device can interactively fuse the text features and image features in the splicing feature matrix to obtain the fusion features of the resource to be processed. Since the fusion features can reflect the relationship between the features of different modes of the resource to be processed, it can better express the characteristics of the resource to be processed.
[0071] In an exemplary embodiment, the interactive fusion of text features and image features in the splicing feature matrix can be implemented by a convolutional neural network (CNN). Specifically, the splicing feature matrix obtained above is input into the convolutional neural network, so that the text features and image features in the splicing feature matrix are interactively fused through the convolutional neural network, and the output of the convolutional neural network is obtained, and the output is the fusion feature of the resource to be processed. In this embodiment, the text features and image features in the splicing feature matrix are interactively fused through the convolutional neural network, so that the processing speed of the interactive fusion can be improved.
[0072] In an exemplary embodiment, if Figure 5 As shown, in step S230, the fused features are subjected to a second fusion process to obtain target fused features of the resources to be processed, which can be specifically achieved by the following steps:
[0073] In step S232, a multi-head attention mechanism is used to obtain a first feature in the fused feature and a second feature used to characterize the relationship between the first feature and the context.
[0074] Among them, the multi-head attention mechanism is implemented based on multiple independent attention mechanisms. The first feature in the fusion feature refers to the feature element in the fusion feature matrix, and the second feature is a feature used to characterize the relationship between the feature element in the fusion feature matrix and the context. In this embodiment, for the sake of convenience, it is described by the first feature and the second feature. Specifically, based on the multi-head attention mechanism, the feature elements in the fusion feature matrix and the features used to characterize the relationship between the feature elements and the context can be obtained.
[0075] In step S234, the first feature and the second feature are fused to obtain a target fused feature of the resource to be processed.
[0076] The target fusion feature is a feature used for classification obtained by fusing the image feature and the text feature of the resource to be processed twice. In this embodiment, the target fusion feature of the resource to be processed is obtained by fusing the first feature and the second feature.
[0077] Specifically, the computer device can calculate the attention weight of each feature element in the fusion feature matrix based on the attention mechanism, so as to obtain the weight of the relationship between each feature element and the context, and the weight is the corresponding second feature. Then, the corresponding feature elements are weighted based on the weights, so as to give higher importance to feature elements with high correlation and lower importance to feature elements with low correlation. The result after weighted processing is the target fusion feature of the resource to be processed. Since the target fusion feature is obtained after two fusions, it can better express the characteristics of the resource to be processed, thereby improving the accuracy of the classification result.
[0078] In an exemplary embodiment, the second fusion processing of the fused features can be implemented by using a bidirectional transmission neural network (Transformer). Specifically, the fused features obtained by the first fusion processing are input into the bidirectional transmission neural network, and the first feature in the fused features and the second feature used to characterize the relationship between the first feature and the context are obtained through the bidirectional transmission neural network, and the first feature and the second feature are fused and output, thereby obtaining the target fused features of the resources to be processed. In this embodiment, the fused features are subjected to the second fusion processing through the bidirectional transmission neural network, so that the speed of the fusion processing can be improved.
[0079] In an exemplary embodiment, if Figure 6 As shown, in step S240, based on the target fusion feature, the classification result of the resource to be processed is determined, which can be specifically achieved by the following steps:
[0080] In step S242, the target fusion feature is classified to obtain a plurality of candidate classification results of the resources to be processed, each candidate classification result having a corresponding classification value.
[0081] The classification value may be a corresponding probability value or a score value. Specifically, the classification process may be implemented using a pre-trained classification model. In this embodiment, the target fusion feature may be input into a pre-trained classification model to obtain multiple candidate classification results of the resource to be processed output by the classification model, that is, to obtain the classification value of each category of the resource to be processed.
[0082] In step S244, the candidate classification result with the highest classification value among the multiple candidate classification results is used as the classification result of the resource to be processed.
[0083] Specifically, in this embodiment, based on the multiple candidate classification results obtained above, the candidate classification result with the highest classification value is used as the final classification result of the resource to be processed, that is, the category with the highest classification value is used as the final category of the resource to be processed.
[0084] In the above embodiment, the target fusion features are classified and processed to obtain multiple candidate classification results of the resources to be processed, and the candidate classification result with the highest classification value is used as the classification result of the resources to be processed. Since the target fusion features for classification processing are obtained after two fusions, they can better express the characteristics of the resources to be processed, which is beneficial to improve the accuracy of the classification results.
[0085] It should be understood that although Figure 1-Figure 6 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1-Figure 6 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.
[0086] Figure 7 is a block diagram of a resource classification device according to an exemplary embodiment. Figure 7 The device includes a feature acquisition module 702, a first fusion processing module 704, a second fusion processing module 706 and a classification result determination module 708.
[0087] A feature acquisition module 702 is configured to acquire text features and image features of a resource to be processed;
[0088] A first fusion processing module 704 is configured to perform a first fusion processing on the text feature and the image feature to obtain a fusion feature of the resource to be processed;
[0089] The second fusion processing module 706 is configured to perform a second fusion processing on the fusion feature to obtain a target fusion feature of the resource to be processed;
[0090] The classification result determination module 708 is configured to determine the classification result of the resource to be processed based on the target fusion feature.
[0091] In an exemplary embodiment, the first fusion processing module includes: a splicing unit, configured to perform splicing of the feature matrix corresponding to the text feature and the feature matrix corresponding to the image feature in a set direction to obtain a spliced feature matrix; a fusion unit, configured to perform interactive fusion of the text features and the image features in the spliced feature matrix to obtain the fusion features of the resource to be processed.
[0092] In an exemplary embodiment, the fusion unit is configured to execute: inputting the splicing feature matrix into a convolutional neural network, interactively fusing the text features and image features in the splicing feature matrix through the convolutional neural network, obtaining the output of the convolutional neural network, and using the output as the fusion feature of the resource to be processed.
[0093] In an exemplary embodiment, the second fusion processing module is configured to perform: using a multi-head attention mechanism to obtain a first feature in the fusion feature and a second feature used to characterize the relationship between the first feature and the context; fusing the first feature and the second feature to obtain a target fusion feature of the resource to be processed.
[0094] In an exemplary embodiment, the second fusion processing module is configured to perform: inputting the fused feature into a bidirectional transmission neural network, obtaining a first feature in the fused feature and a second feature used to characterize the relationship between the first feature and the context through the bidirectional transmission neural network, and fusing the first feature and the second feature to obtain a target fused feature of the resource to be processed.
[0095] In an exemplary embodiment, the classification result determination module is configured to perform: classifying the target fusion features to obtain multiple candidate classification results of the resources to be processed, each of the candidate classification results having a corresponding classification value; and taking the candidate classification result with the highest classification value among the multiple candidate classification results as the classification result of the resources to be processed.
[0096] In an exemplary embodiment, the feature acquisition module includes: an image feature acquisition unit, configured to execute acquisition of at least one frame of image of the resource to be processed, encode the at least one frame of image, and obtain the encoded image features; a text feature acquisition unit, configured to execute acquisition of text content of the resource to be processed, encode the text content, and obtain the encoded text features.
[0097] In an exemplary embodiment, the image feature acquisition unit is configured to execute: performing two-dimensional encoding on the at least one frame of image through an image encoder to obtain encoded image features, and the image features have corresponding feature matrices; the text feature acquisition unit is configured to execute: performing one-dimensional encoding on the text content through a text encoder to obtain encoded one-dimensional vectors; based on the feature matrix corresponding to the image features, converting the one-dimensional vector into a corresponding two-dimensional feature matrix, and using the two-dimensional feature matrix as the text features of the text content.
[0098] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0099] Figure 8 1 is a block diagram of a device Z00 for resource classification according to an exemplary embodiment. For example, the device Z00 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0100] Reference Figure 8 , the device Z00 may include one or more of the following components: a processing component Z02, a memory Z04, a power component Z06, a multimedia component Z08, an audio component Z10, an input / output (I / O) interface Z12, a sensor component Z14, and a communication component Z16.
[0101] The processing component Z02 generally controls the overall operation of the device Z00, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component Z02 may include one or more processors Z20 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component Z02 may include one or more modules to facilitate interaction between the processing component Z02 and other components. For example, the processing component Z02 may include a multimedia module to facilitate interaction between the multimedia component Z08 and the processing component Z02.
[0102] The memory Z04 is configured to store various types of data to support operations on the device Z00. Examples of such data include instructions for any application or method operating on the device Z00, contact data, phone book data, messages, pictures, videos, etc. The memory Z04 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.
[0103] The power supply assembly Z06 provides power to the various components of the device Z00. The power supply assembly Z06 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device Z00.
[0104] The multimedia component Z08 includes a screen that provides an output interface between the device Z00 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component Z08 includes a front camera and / or a rear camera. When the device Z00 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have focal length and optical zoom capabilities.
[0105] The audio component Z10 is configured to output and / or input audio signals. For example, the audio component Z10 includes a microphone (MIC), which is configured to receive external audio signals when the device Z00 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory Z04 or sent via the communication component Z16. In some embodiments, the audio component Z10 also includes a speaker for outputting audio signals.
[0106] I / O interface Z12 provides an interface between processing component Z02 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.
[0107] The sensor assembly Z14 includes one or more sensors for providing various aspects of status assessment for the device Z00. For example, the sensor assembly Z14 can detect the open / closed state of the device Z00, the relative positioning of components, such as the display and keypad of the device Z00, and the sensor assembly Z14 can also detect the position change of the device Z00 or a component of the device Z00, the presence or absence of user contact with the device Z00, the orientation or acceleration / deceleration of the device Z00, and the temperature change of the device Z00. The sensor assembly Z14 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly Z14 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly Z14 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0108] The communication component Z16 is configured to facilitate wired or wireless communication between the device Z00 and other devices. The device Z00 can access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component Z16 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component Z16 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0109] In an exemplary embodiment, the device Z00 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.
[0110] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory Z04 including instructions, and the above instructions can be executed by a processor Z20 of the device Z00 to perform the above method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0111] In an exemplary embodiment, a computer program product is also provided. The computer program product includes a computer program. The computer program is stored in a readable storage medium. At least one processor of a device reads and executes the computer program from the readable storage medium, so that the device executes the above-mentioned resource classification method.
[0112] Fig. 9 FIG. 1 is a block diagram of a device S00 for resource classification according to an exemplary embodiment. For example, the device S00 may be a server. Fig. 9 The device S00 includes a processing component S20, which further includes one or more processors, and a memory resource represented by a memory S22 for storing instructions executable by the processing component S20, such as an application. The application stored in the memory S22 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component S20 is configured to execute instructions to perform the above-mentioned resource classification method.
[0113] The device S00 may also include a power supply component S24 configured to perform power management of the device S00, a wired or wireless network interface S26 configured to connect the device S00 to a network, and an input / output (I / O) interface S28. The device S00 may operate based on an operating system stored in the memory S22, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD or the like.
[0114] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory S22 including instructions, and the above instructions can be executed by a processor of the device S00 to complete the above method. The storage medium can be a computer-readable storage medium, for example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0115] In an exemplary embodiment, a computer program product is also provided. The computer program product includes a computer program. The computer program is stored in a readable storage medium. At least one processor of a device reads and executes the computer program from the readable storage medium, so that the device executes the above-mentioned resource classification method.
[0116] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0117] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A resource classification method, characterized in that: include: Obtain text features and image features of the resource to be processed; Performing a first fusion process on the text features and the image features to obtain fusion features of the resource to be processed; Performing a second fusion process on the fused features to obtain target fused features of the resource to be processed; Based on the target fusion feature, determining the classification result of the resource to be processed; The first fusion processing is performed on the text features and the image features to obtain the fusion features of the resource to be processed, including: Splicing the feature matrix corresponding to the text feature and the feature matrix corresponding to the image feature in a set direction to obtain a spliced feature matrix, wherein the set direction is a row direction or a column direction of a preset feature matrix; Inputting the splicing feature matrix into a convolutional neural network, interactively fusing text features and image features in the splicing feature matrix through the convolutional neural network, obtaining an output of the convolutional neural network, and using the output as a fusion feature of the resource to be processed; The second fusion processing is performed on the fused features to obtain the target fused features of the resource to be processed, including: using a multi-head attention mechanism to obtain a first feature in the fused features and a second feature for characterizing the relationship between the first feature and the context, wherein the fused features include multiple first feature elements; fusing the first feature and the second feature to obtain the target fused features of the resource to be processed.
2. The method according to claim 1, characterized in that The performing a second fusion process on the fusion feature to obtain a target fusion feature of the resource to be processed includes: The fused features are input into a bidirectional transmission neural network, a first feature in the fused features and a second feature used to characterize the relationship between the first feature and the context are obtained through the bidirectional transmission neural network, and the first feature and the second feature are fused to obtain a target fused feature of the resource to be processed.
3. The method according to claim 1, characterized in that The determining the classification result of the resource to be processed based on the target fusion feature includes: Classify the target fusion feature to obtain multiple candidate classification results of the resource to be processed, each of the candidate classification results having a corresponding classification value; Among the multiple candidate classification results, the candidate classification result with the highest classification value is used as the classification result of the resource to be processed.
4. The method according to any one of claims 1 to 3, characterized in that: The step of obtaining text features and image features of the resource to be processed includes: Acquire at least one frame of image of the resource to be processed, and perform encoding processing on the at least one frame of image to obtain encoded image features; The text content of the resource to be processed is obtained, and the text content is encoded to obtain encoded text features.
5. The method according to claim 4, characterized in that The encoding process of the at least one frame of image to obtain encoded image features includes: Performing two-dimensional encoding on at least one frame of image by an image encoder to obtain encoded image features, wherein the image features have a corresponding feature matrix; The encoding process of the text content to obtain encoded text features includes: Performing one-dimensional encoding on the text content by a text encoder to obtain an encoded one-dimensional vector; Based on the feature matrix corresponding to the image feature, the one-dimensional vector is converted into a corresponding two-dimensional feature matrix, and the two-dimensional feature matrix is used as the text feature of the text content.
6. A resource classification device, characterized in that: include: A feature acquisition module is configured to acquire text features and image features of a resource to be processed; A first fusion processing module is configured to perform a first fusion processing on the text feature and the image feature to obtain a fusion feature of the resource to be processed; A second fusion processing module is configured to perform a second fusion processing on the fusion feature to obtain a target fusion feature of the resource to be processed; A classification result determination module is configured to determine the classification result of the resource to be processed based on the target fusion feature; The first fusion processing module includes: A splicing unit is configured to perform splicing of the feature matrix corresponding to the text feature and the feature matrix corresponding to the image feature in a set direction to obtain a spliced feature matrix, wherein the set direction is a row direction or a column direction of a preset feature matrix; A fusion unit is configured to input the spliced feature matrix into a convolutional neural network, interactively fuse the text features and image features in the spliced feature matrix through the convolutional neural network, obtain an output of the convolutional neural network, and use the output as a fusion feature of the resource to be processed; The second fusion processing module is configured to execute: A multi-head attention mechanism is used to obtain a first feature in the fused feature and a second feature used to characterize the relationship between the first feature and the context, wherein the fused feature includes multiple first feature elements; the first feature and the second feature are fused to obtain a target fused feature of the resource to be processed.
7. The device according to claim 6, characterized in that The second fusion processing module is configured to execute: The fused features are input into a bidirectional transmission neural network, a first feature in the fused features and a second feature used to characterize the relationship between the first feature and the context are obtained through the bidirectional transmission neural network, and the first feature and the second feature are fused to obtain a target fused feature of the resource to be processed.
8. The device according to claim 6, characterized in that The classification result determination module is configured to execute: Classify the target fusion feature to obtain multiple candidate classification results of the resource to be processed, each of the candidate classification results having a corresponding classification value; Among the multiple candidate classification results, the candidate classification result with the highest classification value is used as the classification result of the resource to be processed.
9. The device according to any one of claims 6 to 8, characterized in that The feature acquisition module comprises: An image feature acquisition unit is configured to acquire at least one frame of the image of the resource to be processed, perform encoding processing on the at least one frame of the image, and obtain encoded image features; The text feature acquisition unit is configured to acquire the text content of the resource to be processed, encode the text content, and obtain the encoded text features.
10. The device according to claim 9, characterized in that The image feature acquisition unit is configured to perform: Performing two-dimensional encoding on the at least one frame of image by an image encoder to obtain encoded image features, wherein the image features have a corresponding feature matrix; The text feature acquisition unit is configured to perform: one-dimensional encoding of the text content through a text encoder to obtain an encoded one-dimensional vector; based on a feature matrix corresponding to the image feature, converting the one-dimensional vector into a corresponding two-dimensional feature matrix, and using the two-dimensional feature matrix as the text feature of the text content.
11. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the resource classification method as described in any one of claims 1 to 5.
12. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the resource classification method as claimed in any one of claims 1 to 5.
13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the resource classification method according to any one of claims 1 to 5 is implemented.
Citation Information
Patent Citations
Video classification method and device and electronic equipment
CN110399841A
Attention fusion-based online short video multi-modal emotion recognition method
CN111275085A
Attention mechanism-based intention recognition method and device, equipment and storage medium
CN111737458A
Interactive-scene-oriented automatic lie detection method and device, equipment and medium
CN112329748A
Social media sentiment analysis method and system based on multi-modal feature fusion
CN112508077A
