Content prediction method and device, equipment, storage medium and product
By dynamically selecting feature processing methods in multimodal data scenarios, the problem of information loss caused by fixed feature extraction is solved, the accuracy of content prediction is improved, and it is suitable for feature processing of different prediction tasks and data.
Patent Information
- Application Number
- CN202310558215.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-17
- Publication Date
- 2024-11-22
AI Technical Summary
In the content prediction of multimodal data scenarios, fixed-type feature extraction and feature splicing lead to missing information, resulting in low accuracy of prediction results.
By obtaining a variety of candidate feature processing methods, including feature filtering and feature dimensionality reduction, the most suitable feature processing method is dynamically selected based on the prediction task and data attributes, and feature processing is performed to improve the accuracy of the prediction results.
It improves the matching degree between feature processing methods and prediction tasks, improves the accuracy of content prediction, and is suitable for dynamic adjustment of different prediction tasks and data.
Smart Images

Figure CN119007228A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of machine learning, and in particular to a content prediction method, apparatus, device, storage medium and product. Background Art
[0002] In a multimodal data scenario, for content prediction tasks, it is necessary to extract features from the content data in different modalities corresponding to the content prediction task, and then perform content prediction after feature fusion, and finally obtain the prediction result corresponding to the content prediction task.
[0003] In related technologies, content data under different modalities are usually subjected to fixed feature extraction and then feature splicing, thereby analyzing the spliced multimodal fusion features to ultimately obtain a prediction result.
[0004] However, in the related art, a fixed method is selected for feature extraction and feature splicing during the content prediction process. Due to the differences in content prediction tasks, the features extracted by the fixed method may have missing information, making the fixed method mismatched with the input content data and the content prediction task, resulting in the final multimodal fusion feature having low accuracy, and thus making the final content prediction result less accurate. Summary of the invention
[0005] The embodiments of the present application provide a content prediction method, apparatus, device, storage medium and product, which can improve the adaptability between feature processing mode and prediction task and content data. The technical solution is as follows.
[0006] In one aspect, a content prediction method is provided, the method comprising:
[0007] Acquire multiple content data corresponding to the prediction task, and extract first feature representations respectively corresponding to the multiple content data, wherein the multiple content data correspond to at least two modalities;
[0008] Acquire multiple candidate feature processing methods, each candidate feature processing method including at least one of a feature screening method and a feature dimensionality reduction method for the first feature representation;
[0009] Determining a target feature processing method from the plurality of candidate feature processing methods based on a prediction target corresponding to the prediction task and a data attribute corresponding to the content data, wherein the data attribute is used to indicate a data category of the content data under the prediction task;
[0010] Performing feature processing on the plurality of first feature representations by using the target feature processing method to obtain a plurality of second feature representations;
[0011] Content prediction is performed on the multiple second feature representations to obtain a prediction result corresponding to the prediction task, where the prediction result is used to indicate an analysis result corresponding to the prediction task obtained through the multiple content data.
[0012] On the other hand, a content prediction method is provided, which is applied to an image recognition scenario, and includes:
[0013] Acquire a plurality of content data corresponding to the first image, and extract first feature representations respectively corresponding to the plurality of content data, wherein the plurality of content data correspond to at least two modalities;
[0014] Acquire multiple candidate feature processing methods, each candidate feature processing method including at least one of a feature screening method and a feature dimensionality reduction method for the first feature representation;
[0015] Determining a target feature processing method from the plurality of candidate feature processing methods based on the recognition object in the first image and a data attribute corresponding to the content data, the data attribute being used to indicate a data category of the content data under the image recognition task;
[0016] Performing feature processing on the plurality of first feature representations by using the target feature processing method to obtain a plurality of second feature representations;
[0017] Image recognition is performed on the multiple second feature representations to obtain a recognition result corresponding to the first image, where the recognition result is used to indicate a category corresponding to an identified object in the first image obtained through the multiple content data.
[0018] In another aspect, a content prediction device is provided, the device comprising:
[0019] A first acquisition module, configured to acquire a plurality of content data corresponding to the prediction task, and extract first feature representations respectively corresponding to the plurality of content data, wherein the plurality of content data correspond to at least two modes;
[0020] The first acquisition module is further used to acquire multiple candidate feature processing methods, each candidate feature processing method includes at least one of a feature screening method and a feature dimensionality reduction method for the first feature representation;
[0021] A first determination module, configured to determine a target feature processing method from the plurality of candidate feature processing methods based on a prediction target corresponding to the prediction task and a data attribute corresponding to the content data, wherein the data attribute is used to indicate a data category of the content data under the prediction task;
[0022] A first processing module, configured to perform feature processing on the plurality of first feature representations by using the target feature processing method to obtain a plurality of second feature representations;
[0023] A prediction module is used to perform content prediction on the multiple second feature representations to obtain a prediction result corresponding to the prediction task, wherein the prediction result is used to indicate an analysis result corresponding to the prediction task obtained through the multiple content data.
[0024] In some embodiments, the first determining module includes:
[0025] An output unit is used to input the prediction target corresponding to the prediction task, multiple first feature representations, and the multiple candidate feature processing methods into a pre-trained reinforcement learning model, and output the target feature processing method. The reinforcement learning model is used to perform fitness analysis on the multiple candidate feature processing methods in combination with the prediction target corresponding to the prediction task and the multiple first feature representations. The target feature processing method includes at least one of a target feature screening method and a target feature dimensionality reduction method.
[0026] In some embodiments, the reinforcement learning model includes a target policy network and an agent network;
[0027] The output unit is further used to input the prediction target, multiple first feature representations and the multiple candidate feature processing methods corresponding to the prediction task into the target strategy network, and output the prediction probabilities corresponding to the multiple candidate feature processing methods respectively. The target strategy network is used to perform the fitness analysis on the multiple candidate feature processing methods in combination with the prediction target and multiple first feature representations corresponding to the prediction task; determine the target feature screening method based on the prediction probabilities corresponding to the multiple candidate feature processing methods respectively; input the prediction target, multiple first feature representations and the prediction probabilities corresponding to the multiple candidate feature processing methods corresponding to the prediction task into the intelligent agent network, and output the target feature dimensionality reduction method. The intelligent agent network is used to predict the target feature dimensionality reduction method that meets the prediction target based on the prediction probabilities corresponding to the multiple first feature representations and the multiple candidate feature processing methods respectively.
[0028] In some embodiments, the prediction probabilities corresponding to the plurality of candidate feature processing methods include first prediction probabilities corresponding to a plurality of feature screening methods;
[0029] The output unit is further used to determine the feature screening method with the highest first prediction probability value as the target feature screening method from the first prediction probabilities respectively corresponding to the multiple feature screening methods.
[0030] In some embodiments, the prediction probabilities corresponding to the plurality of candidate feature processing methods include second prediction probabilities corresponding to a plurality of feature dimensionality reduction methods;
[0031] The output unit is also used to input the prediction target corresponding to the prediction task, multiple first feature representations, and the second prediction probabilities corresponding to the multiple feature dimensionality reduction methods into the intelligent agent network, and output the target feature dimensionality reduction method.
[0032] In some embodiments, the apparatus further comprises:
[0033] The first acquisition module is used to acquire a plurality of sample content data corresponding to the sample prediction task, and extract first feature representations respectively corresponding to the plurality of sample content data, wherein the plurality of sample content data correspond to at least two modes;
[0034] The first acquisition module is further used to acquire multiple sample feature processing methods, each sample feature processing method includes at least one of a sample feature screening method and a sample feature dimensionality reduction method;
[0035] An output module, used to input the prediction target corresponding to the sample prediction task, the multiple first feature representations and the multiple sample feature processing methods into the sample strategy network, and output the prediction probabilities corresponding to the multiple sample feature processing methods respectively;
[0036] The output module is further used to input the prediction target corresponding to the sample prediction task, the multiple first feature representations, and the prediction probabilities corresponding to the multiple sample feature processing methods into the agent network, and output the prediction result of the feature dimensionality reduction method;
[0037] A generation module generates a target reward function based on the prediction result of the feature dimensionality reduction method;
[0038] A training module is used to train the sample policy network based on the target reward function to obtain the target policy network.
[0039] In some embodiments, the first processing module is further used to perform correlation screening on the multiple first feature representations by using the target feature screening method to obtain multiple first sub-feature representations that meet the correlation condition, wherein the correlation condition is used to indicate the correlation between the first feature representation and the prediction target; and perform feature dimensionality reduction on the multiple first sub-feature representations by using the target feature dimensionality reduction method to obtain the multiple second feature representations.
[0040] In some embodiments, the output module is also used to input the multiple first feature representations into a pre-trained deep learning model, and output multiple high-dimensional feature representations, the feature dimensions corresponding to the high-dimensional feature representations are higher than the feature dimensions corresponding to the first feature representations, and the deep learning model is used to extract the high-dimensional feature representations corresponding to the first feature representations.
[0041] In some embodiments, the first acquisition module is used to obtain feature identifiers corresponding to the multiple first feature representations, and the feature identifiers are used to indicate feature dimensions corresponding to the first feature representations; first feature representations with the same feature identifier in the first feature representations corresponding to the multiple content data are feature spliced to obtain a multi-fusion feature representation.
[0042] On the other hand, a content prediction device is provided, which is applied to an image recognition scenario, and includes:
[0043] A second acquisition module is used to acquire a plurality of content data corresponding to the first image, and extract first feature representations respectively corresponding to the plurality of content data, wherein the plurality of content data corresponds to at least two modalities;
[0044] The second acquisition module is further used to acquire multiple candidate feature processing methods, each candidate feature processing method includes at least one of a feature screening method and a feature dimensionality reduction method for the first feature representation;
[0045] A second determination module, configured to determine a target feature processing method from the plurality of candidate feature processing methods based on a recognition object in the first image and a data attribute corresponding to the content data, wherein the data attribute is used to indicate a data category of the content data under the image recognition task;
[0046] A second processing module, configured to perform feature processing on the plurality of first feature representations by using the target feature processing method to obtain a plurality of second feature representations;
[0047] The recognition module is used to perform image recognition on the multiple second feature representations to obtain a recognition result corresponding to the first image, wherein the recognition result is used to indicate a category corresponding to the recognized object in the first image obtained through the multiple content data.
[0048] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a content prediction method as described in any of the above-mentioned embodiments of the present application.
[0049] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a content prediction method as described in any of the above-mentioned embodiments of the present application.
[0050] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs any of the content prediction methods described in the above embodiments.
[0051] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:
[0052] After obtaining multiple content data under at least two modes corresponding to the prediction task and extracting the first feature representations corresponding to the multiple content data, a plurality of candidate feature processing methods are obtained, and a target feature processing method is selected from them according to the prediction target corresponding to the prediction task and the data attribute corresponding to the content data, so as to perform feature processing on the multiple first feature representations according to the feature screening method and / or feature dimension reduction method in the target feature processing method, obtain multiple second feature representations, and finally perform content prediction according to the second feature representation to obtain the prediction result corresponding to the prediction task. That is, after extracting the first feature representations corresponding to the multiple content data corresponding to the prediction task, the most suitable feature processing method is selected according to the prediction task and the content data to perform feature processing on the first feature representation again, so that the processed second feature representation is more in line with the prediction task, thereby improving the accuracy of the final predicted content, and according to different prediction tasks and different content data, the selection result of the feature processing method can be dynamically adjusted. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0054] Figure 1 It is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;
[0055] Figure 2 is a flow chart of a content prediction method provided by an exemplary embodiment of the present application;
[0056] Figure 3 is a flow chart of a content prediction method provided by an exemplary embodiment of the present application;
[0057] Figure 4 is a flow chart of a content prediction method provided by another exemplary embodiment of the present application;
[0058] Figure 5 is a flow chart of a content prediction method provided by another exemplary embodiment of the present application;
[0059] Figure 6 is a structural block diagram of a content prediction device provided by an exemplary embodiment of the present application;
[0060] Figure 7 is a structural block diagram of a content prediction device provided by another exemplary embodiment of the present application;
[0061] Figure 8 is a structural block diagram of a content prediction device provided by another exemplary embodiment of the application;
[0062] Fig. 9 It is a schematic diagram of the structure of a server provided by an exemplary embodiment of the application. DETAILED DESCRIPTION
[0063] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.
[0064] First, a brief introduction is given to the terms involved in the embodiments of the present application.
[0065] Artificial Intelligence (AI): It is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.
[0066] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0067] Machine Learning (ML): It is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.
[0068] First, the implementation environment of this application is introduced. Figure 1 , which shows a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application, such as Figure 1 As shown, the implementation environment includes a terminal 110 and a server 120 , and the terminal 110 and the server 120 are connected via a communication network 130 .
[0069] The terminal 110 stores a plurality of content data corresponding to the prediction task, and generates a content prediction request by sending the prediction target and the content data corresponding to the prediction task to the server 120 .
[0070] After receiving the content prediction request sent by the terminal 110, the server 120 first extracts the first feature representations corresponding to the multiple content data respectively, and obtains the multiple pre-stored candidate feature processing methods from the corresponding file of the server 120, and determines the target feature processing method from the multiple candidate feature processing methods according to the prediction target corresponding to the prediction task in the content prediction request and the data attribute corresponding to the content data, and then performs feature processing on the multiple first feature representations according to the feature screening method and / or feature dimension reduction method in the target feature processing method, so as to obtain multiple second feature representations, and finally performs content prediction on the prediction task according to the multiple second feature representations, so as to obtain the prediction result corresponding to the prediction task, and feeds the prediction result back to the terminal 110, so that the terminal 110 displays the prediction result corresponding to the prediction task.
[0071] Among them, the terminal 110 includes at least one of the terminals such as smart phones, tablet computers, portable laptops, desktop computers, smart speakers, smart wearable devices, smart voice interaction devices, smart home appliances, and vehicle-mounted terminals. The server 120 can be used in scenarios where command operations are implemented in the fields of smart transportation, vehicle-mounted terminals, and the Internet of Things.
[0072] It is worth noting that the above-mentioned communication network 130 can be implemented as a wired network or a wireless network, and the communication network 130 can be implemented as any one of a local area network, a metropolitan area network or a wide area network, which is not limited in the embodiments of the present application.
[0073] It is worth noting that the above-mentioned server 120 can be implemented as a cloud server in the cloud, where cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network in a wide area network or a local area network to achieve data calculation, storage, processing, and sharing.
[0074] In some embodiments, the server 120 may also be implemented as a node in a blockchain system.
[0075] It is worth noting that the content prediction method provided in the embodiment of the present application can be implemented by the terminal 110 alone, can also be implemented by the server 120 alone, or can also be implemented by the terminal 110 and the server 120 in collaboration.
[0076] Among them, when the terminal 110 and the server 120 collaborate to implement the solution provided by the embodiment of the present application, the terminal 110 and the server 120 can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present application.
[0077] In the embodiment of the present application, the content prediction method and implementation provided in the server 120 are business services in the application service layer.
[0078] It should be noted that before collecting relevant data of users and during the process of collecting relevant data of users, this application can display a prompt interface, pop-up window or output voice prompt information, and the prompt interface, pop-up window or voice prompt information is used to prompt the user that its relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining relevant data of users after obtaining the confirmation operation issued by the user to the prompt interface or pop-up window, otherwise (that is, when the confirmation operation issued by the user to the prompt interface or pop-up window is not obtained), the relevant steps of obtaining relevant data of users are terminated, that is, the relevant data of users are not obtained. In other words, all user data collected by this application are collected with the consent and authorization of the user, and the collection, use and processing of relevant user data need to comply with the relevant laws, regulations and standards of the relevant regions.
[0079] In combination with the above-mentioned noun introduction and implementation environment, the content prediction method provided in the embodiment of the present application is described, and the method is performed by the server as an example for description. For schematic illustration, please refer to Figure 2 , which shows a flowchart of a content prediction method provided by an exemplary embodiment of the present application, the method includes the following steps.
[0080] Step 210: Acquire multiple content data corresponding to the prediction task, and extract first feature representations corresponding to the multiple content data respectively.
[0081] Among them, multiple content data correspond to at least two modes.
[0082] In an illustrative manner, a prediction task refers to predicting a specified input content, thereby obtaining a prediction result corresponding to the specified content. The types of prediction tasks are different for different scenarios, and therefore, the prediction targets corresponding to the prediction tasks are also different.
[0083] In one example, when the scene is an image recognition scene, the prediction task can be implemented as identifying a specified object in the input image to obtain a classification corresponding to the specified object. For example, when an animal image is input, the prediction task is to identify the animal category contained in the animal image.
[0084] In another example, when the scenario is a text recognition scenario, the prediction task can be implemented as identifying entity words in the input text content to obtain question and answer results corresponding to the text content. For example: input a paragraph of text, and the prediction task is to identify the entity words contained in the text content to determine the question and answer results corresponding to the text content.
[0085] In addition, in addition to the application scenarios mentioned above, the application scenarios corresponding to the prediction tasks also include at least one of the following scenarios: target detection, image segmentation, face recognition, sentiment analysis, named entity recognition, machine translation, speech recognition, medical image analysis, pathological diagnosis, gene sequence analysis, etc.
[0086] Illustratively, content data refers to data obtained from the specified content corresponding to the predicted content. For example, when the prediction task is to identify a specified object in an input image, the content data includes image data corresponding to the input image, text description data corresponding to the input image, and voice introduction data corresponding to the input image.
[0087] Illustratively, the modality is used to indicate the type of data. In one example, the image data corresponding to the input image is content data in the image modality, the text description data corresponding to the input image is content data in the text modality, and the voice description data corresponding to the input image is content data in the voice modality.
[0088] Optionally, for the same prediction task, at least one content data is included under the same modality. For example, for prediction task a, content data under multiple modalities are included, wherein the first modality includes content data a and content data b, the second modality includes content data c, and the third modality includes content data d and content data e, wherein the first modality, the second modality and the third modality belong to different modalities respectively.
[0089] In some embodiments, after acquiring multiple content data corresponding to the prediction task, first feature representations corresponding to the multiple content data are extracted, wherein the feature extraction method is different for content data of different modalities.
[0090] Optionally, the feature type corresponding to the first feature representation includes at least one of a one-hot encoding representation, an embedding feature representation, and a matrix vector representation.
[0091] Step 220, obtaining multiple candidate feature processing methods.
[0092] Each candidate feature processing method includes at least one of a feature screening method and a feature dimension reduction method for the first feature representation.
[0093] Illustratively, the feature processing method refers to a method of performing feature processing again on the first feature representation corresponding to the content data on the basis of obtaining the first feature representation, so as to obtain the second feature representation.
[0094] In some embodiments, feature screening refers to screening first feature representations corresponding to a plurality of content data to obtain a portion of the first feature representations, wherein the screening condition includes at least one of the following conditions: the relevance of the first feature representation to the prediction task, the feature importance corresponding to the first feature representation, and the like.
[0095] In one example, the correlation between the first feature representation and the prediction task refers to the degree of information sharing between the first feature representation and the prediction target corresponding to the prediction task, that is, the mutual information (MI) between the first feature representation and the prediction target corresponding to the prediction task. For example: the amount of information corresponding to the prediction target A included in the first feature representation a is the mutual information between the first feature representation a and the prediction target A.
[0096] Illustratively, the prediction target corresponding to the prediction task refers to the specified object that needs to be predicted in the input content. For example, when the input content is an image, the prediction target is the specified object that needs to be identified in the image. Optionally, the specified object refers to all objects contained in the input content; or, the specified object is at least one object specified among the objects contained in the input content.
[0097] In another exemplary embodiment, the feature importance corresponding to the first feature representation is used to measure the contribution of the first feature representation in the prediction process of the prediction task. Optionally, the feature importance corresponding to the first feature representation can be obtained by at least one of the following methods: Information Gain (IG), Gini Index, etc.
[0098] The information gain method refers to the reduction in entropy corresponding to the predicted target under the condition of a given first feature representation.
[0099] The Gini index refers to the impurity corresponding to the predicted target under the condition of a given first feature representation.
[0100] In some embodiments, the feature dimension reduction method refers to reducing the feature dimension corresponding to the first feature representation after screening, so as to reduce the redundant features and noise features between the first feature representations, and realize the feature optimization of the first feature representation after screening. Optionally, the feature dimension reduction method includes at least one of the principal component analysis (PCA) method, T-distributed stochastic neighbor embedding (T-SNE) method, and the like.
[0101] Optionally, the multiple candidate feature processing methods are all candidate feature processing methods stored in the server; or, the multiple candidate feature processing methods are selectable candidate feature processing methods corresponding to the prediction task, which is not limited.
[0102] Optionally, when each candidate feature processing method includes both a feature screening method and a feature dimensionality reduction method, the feature screening method and the feature dimensionality reduction method are a fixed combination of feature processing methods; or, the feature screening method and the feature dimensionality reduction method are a randomly matched combination of feature processing methods, and there is no limitation on this.
[0103] In the present application, the types of methods included in the multiple candidate feature processing methods may be the same or different. For example, candidate feature processing method 1 includes feature screening method a and feature dimensionality reduction method b, candidate feature processing method 2 includes feature screening method b, and candidate feature processing method 3 includes feature dimensionality reduction method c.
[0104] Step 230 , determining a target feature processing method from a plurality of candidate feature processing methods based on a prediction target corresponding to the prediction task and a data attribute corresponding to the content data.
[0105] The data attribute is used to indicate the data category of the content data under the prediction task.
[0106] Illustratively, the data attribute corresponding to the content data refers to the data type under the prediction task corresponding to the content data. For example, if the prediction task is an input image and the content data a is a text description corresponding to the image, then the data attribute corresponding to the content data a is a text type.
[0107] In some embodiments, the target feature processing method refers to a feature processing method that has the highest degree of adaptability to the prediction target of the prediction task and the data attributes of the content data among multiple candidate feature processing methods.
[0108] Optionally, the target feature processing method may include at least one of the following methods:
[0109] 1. Pre-train a reinforcement learning model, input the prediction target corresponding to the prediction task, the first feature representation corresponding to the content data, and multiple candidate feature processing methods into the reinforcement learning model, and then output the target feature processing method, that is, automatically select the target feature processing method through the reinforcement learning model;
[0110] 2. Each candidate feature processing method is annotated with a first matching score corresponding to each prediction task, and a second matching score corresponding to the first feature representation of the content data under each modality, so that the total matching score corresponding to each candidate feature processing method is determined according to the prediction task and the content data, and the target feature processing method is selected according to the total matching score.
[0111] It is worth noting that the above-mentioned target feature processing method is only an illustrative example and is not limited to this in the embodiments of the present application.
[0112] For the second determination method of the above-mentioned target feature processing method, in one example, the current method includes candidate feature processing method 1 and candidate feature processing method 2, wherein the first matching score of candidate feature processing method 1 for prediction task a is 80, the first matching score for prediction task b is 20, the second matching score for the first feature representation corresponding to content data a is 30, the second matching score for the first feature representation corresponding to content data b is 50, and the second matching score for the first feature representation corresponding to content data c is 20; the first matching score of candidate feature processing method 2 for prediction task a is 30, and the second matching score for prediction task b is 20. The first matching score of prediction task b is 70, the second matching score of the first feature representation corresponding to content data a is 20, the second matching score of the first feature representation corresponding to content data b is 60, and the second matching score of the first feature representation corresponding to content data c is 20. Therefore, after obtaining prediction task a and content data b and content data c, the total matching score corresponding to candidate feature processing method 1 is calculated to be 80+50+20=150, and the total matching score corresponding to candidate feature processing method 2 is 30+60+20=110. Therefore, candidate feature processing method 1 is selected as the target feature processing method.
[0113] Step 240 , performing feature processing on the plurality of first feature representations by using a target feature processing method to obtain a plurality of second feature representations.
[0114] Illustratively, after determining the target feature processing method, feature screening and / or feature dimensionality reduction are performed on multiple first feature representations respectively through the target feature screening method and / or feature dimensionality reduction method in the target feature processing method, so as to obtain multiple second feature representations.
[0115] Illustratively, the second feature representation is used to represent the feature representation that is most relevant to the prediction target of the prediction task.
[0116] The number of features represented by the second feature is equal to or less than the number of features represented by the first feature.
[0117] Optionally, the first feature representation and the second feature representation may belong to the same feature representation; or may belong to different feature representations.
[0118] Step 250 , performing content prediction on the plurality of second feature representations to obtain a prediction result corresponding to the prediction task.
[0119] The prediction result is used to indicate the analysis result corresponding to the prediction task obtained through multiple content data.
[0120] Schematically, a content prediction model is pre-trained, and after obtaining multiple second feature representations, the multiple second feature representations are input into the content prediction model, and the prediction result corresponding to the prediction task, that is, the analysis result corresponding to the prediction target, is output.
[0121] In one example, when the prediction task is to perform image recognition on an input image, after obtaining multiple second feature representations, the multiple second feature representations are input into a pre-trained image recognition model, and the categories corresponding to each object in the input image are output.
[0122] In summary, the content prediction method provided in the embodiment of the present application, after obtaining multiple content data under at least two modes corresponding to the prediction task and extracting the first feature representations respectively corresponding to the multiple content data, obtains multiple candidate feature processing methods, and selects the target feature processing method from them according to the prediction target corresponding to the prediction task and the data attribute corresponding to the content data, thereby performing feature processing on the multiple first feature representations according to the feature screening method and / or feature dimension reduction method in the target feature processing method, obtaining multiple second feature representations, and finally performing content prediction according to the second feature representation to obtain the prediction result corresponding to the prediction task. That is, after extracting the first feature representations respectively corresponding to the multiple content data corresponding to the prediction task, the most suitable feature processing method is selected according to the prediction task and the content data to perform feature processing on the first feature representation again, so that the processed second feature representation is more in line with the prediction task, thereby improving the accuracy of the final predicted content, and according to different prediction tasks and different content data, the selection result of the feature processing method can be dynamically adjusted.
[0123] In an optional embodiment, the target feature processing method is determined by a pre-trained reinforcement learning model. For example, please refer to Figure 3 , which shows a flow chart of a content prediction method provided by an exemplary embodiment of the present application, that is, step 230 includes step 231, step 240 includes step 241 and step 242, step 220 also includes step 2201 before step 220, and step 210 also includes step 2101 and step 2102 after step 210. Figure 3 As shown, the method includes the following steps.
[0124] Step 210: Acquire multiple content data corresponding to the prediction task, and extract first feature representations corresponding to the multiple content data respectively.
[0125] Among them, multiple content data correspond to at least two modes.
[0126] Illustratively, after the prediction task is determined, a plurality of content data in at least two modes corresponding to the prediction task is obtained.
[0127] After acquiring a plurality of content data, data preprocessing is first performed on the plurality of content data to obtain a plurality of preprocessed data, wherein the preprocessed data is used to ensure that the content data can be correctly represented in a subsequent process.
[0128] In one example, when the content data is data in text mode, preprocessing operations such as word segmentation, stop word removal, stem extraction, and word form restoration are performed on the content data.
[0129] Among them, word segmentation refers to the process of recombining continuous word sequences into semantically independent word sequences according to certain specifications. For example, if the text sentence "Today's weather is very good" is input and segmented according to the part of speech, the segmentation result is "Today / of / weather / very good", where "weather" and "today" are nouns, "of" is a particle, and "very good" is an adjective. Stop word removal refers to the process of automatically filtering out certain characters or words before or after processing natural language data in information retrieval to save storage space and improve search efficiency. Stemming refers to the process of removing the affixes of words in the text content to obtain the root word. For example, when the text data is English text, the general writing of words in the English text is obtained by extracting stems. Lemma restoration refers to converting a word of any form into a grammatical basic form according to dictionary requirements.
[0130] In another example, when the content data is data in an image modality, preprocessing operations such as image scaling, image normalization, image correction, and data enhancement are performed on the content data.
[0131] Among them, image scaling refers to reducing or enlarging the size of an image; image normalization refers to the process of performing a series of standard processing transformations on an image to transform it into a fixed standard form; image correction refers to the restoration processing of a distorted image; and data enhancement refers to changing an image to generate a new image.
[0132] In other examples, when the content data is data in a mode other than the above two modes, corresponding preprocessing operations are performed on the content data according to the modality type.
[0133] In some embodiments, after obtaining a plurality of pre-processed data corresponding to the prediction task, first feature representations respectively corresponding to the plurality of pre-processed data are extracted.
[0134] In this embodiment, a plurality of preprocessed data are input into the deep learning model through a pre-trained deep learning model, and embedded feature representations (embedding) corresponding to the plurality of preprocessed data are output as the first feature representation.
[0135] Below, the deep learning model used to extract the first feature representation of content data in several different modalities is introduced in detail.
[0136] (I) Data in text mode
[0137] For text data of words or word classes, a variety of different pre-trained word embedding models can be used to map the words or words in the text data into a continuous vector space. The pre-trained word embedding models include at least one of the word vector model (Word To Vector, Word2Vec), the global logarithmic bilinear regression model (Global Vectors, GloVe), the BERT model, etc.
[0138] For text data such as sentences or paragraphs, recurrent neural network models such as Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), and Gate Recurrent Unit (GRU) can be used to extract the embedded feature representation corresponding to the text data.
[0139] (1) Word2Vec model
[0140] The Word2Vec model is a neural network-based word embedding representation learning method that maps each word in text data to a continuous vector space. Word2Vec includes two models: the Skip-Gram model and the Continuous Bag of Words model (CBOW).
[0141] For the Skip-Gram model, first a word w is given as the central word. The prediction goal of the Skip-Gram model is to predict the context word c corresponding to the central word, that is, the surrounding words corresponding to the central word. The optimization goal of the Skip-Gram model is to maximize the log-likelihood function. For details, please refer to Formula 1.
[0142] Formula 1: L = ∑ w∈W ∑ -n≤j≤n,j≠0 logP(c w+j |w)
[0143] Where W represents all words in the text data, n represents the context window size, and P(c w+j |w) means predicting the context word c given the central word w w+j probability.
[0144] Probability P(c w+j |w) is calculated by the Softmax function. For details, please refer to the following formula 2.
[0145] Formula 2:
[0146] Here, v w and v c are the embedding vectors of words w and c respectively, and V represents the vocabulary.
[0147] (2) GloVe Model
[0148] The GloVe model learns word embedding representation by minimizing the objective function. For details, please refer to the following formula 3.
[0149] Formula 3:
[0150] Among them, |V| is the word size of the text data, P ij is the probability of word i and word j appearing together, w i and w j are the embedding vectors of word i and word j, respectively, and b i and b j is a bias term, and f(x) is a weight function used to balance the contribution of different words.
[0151] 2. Data in image mode
[0152] A pre-trained convolutional neural network is used to extract image features from the input image and generate an embedded feature representation corresponding to the image data. Optionally, the convolutional neural network includes at least one of a 16-layer Visual Geometry Group Network (VGG-16), a Residual Network (ResNet), an Inception model, and the like.
[0153] (1) Convolutional Neural Networks (CNN)
[0154] Convolutional neural network is a deep learning model for image data, which can automatically learn the local and global features corresponding to image data. The basic structure of CNN model includes convolution layer, activation function layer and pooling layer.
[0155] For the convolution layer, given the input feature map X and the convolution kernel W, the output feature map Y is given. For details, please refer to the following formula 4.
[0156] Formula 4:
[0157] Among them, X i+a,j+b Represents the value of the input feature map X at the position (i+a, j+self), W a,b represents the value of the convolution kernel W at the position (a, b), Y ij Represents the value of the output feature map Y at position (i, j).
[0158] For the activation function layer, commonly used activation functions include ReLU function, tanh function and sigmoid function, etc. Taking the ReLU function as an example, refer to the following formula 5.
[0159] Formula 5: f(x) = max(0, x)
[0160] The goal of the pooling layer (such as the maximum pooling layer) is to reduce the spatial resolution of the feature map while retaining key information. For the maximum pooling layer, refer to the following formula 6.
[0161] Formula 6:
[0162] Among them, stride represents the step size of pooling.
[0163] (2) Transfer Learning
[0164] By leveraging pre-trained CNN models (such as VGG, ResNet, etc.), embedded feature representations of image data can be obtained.
[0165] Given a pre-trained CNN model M and an input image I, we can obtain the embedded feature representation corresponding to the input image in the following way:
[0166] 1. Input image I into the pre-trained CNN model M and calculate the output feature map of each layer;
[0167] 2. Select the output feature map of a specified layer (usually a fully connected layer or a pooling layer) as the embedded feature representation corresponding to image I.
[0168] (3) Content data in other modes
[0169] Select the corresponding deep learning model according to the modality type. For example, the content data under the audio modality can use a deep acoustic model, such as a neural network-hidden Markov model (Deep Neural Networks-Hidden Markov Model, DNN-HMM) to extract features from the content data and obtain the embedded feature representation corresponding to the content data.
[0170] In some embodiments, after obtaining the first feature representations respectively corresponding to the multiple content data, it is also necessary to perform feature fusion on the first feature representations respectively corresponding to the multiple content data, so as to obtain a unified feature vector representation as a multimodal fusion feature.
[0171] Optionally, the feature fusion method is at least one of the following methods:
[0172] 1. Weighted average: The first feature representations of different modes are weighted averaged according to pre-set weights. The weights can be optimized through methods such as cross-validation;
[0173] 2. Feature concatenation: directly concatenate the embedded feature representations of different modalities to form a longer vector representation;
[0174] 3. Self-attention fusion: Use the self-attention mechanism (such as Transformer) to weightedly combine the embedded feature representations of different modalities, so as to learn a more discriminative fused feature representation.
[0175] The following is a detailed description of the feature splicing method.
[0176] Step 2101, obtaining feature identifiers corresponding to a plurality of first feature representations respectively.
[0177] The feature identifier is used to indicate the feature dimension corresponding to the first feature representation.
[0178] Illustratively, the feature identifier refers to the feature dimension corresponding to the first feature representation, that is, different feature representations are configured for different feature dimensions. For example, for the first feature representation of 256 dimensions, the feature identifier is configured as 1, and for the first feature representation of 512 dimensions, the feature identifier is configured as 2.
[0179] After configuring the feature identifiers corresponding to different feature dimensions, the corresponding feature identifier is configured for the first feature representation according to the feature dimension corresponding to each first feature representation.
[0180] Step 2102 , performing feature concatenation on first feature representations with the same feature identifier among first feature representations corresponding to a plurality of content data, to obtain a multi-fusion feature representation.
[0181] Illustratively, after obtaining feature identifiers corresponding to multiple first feature representations, first feature representations with the same feature identifier (that is, first feature representations with the same feature dimension) are feature concatenated to obtain a multimodal feature representation corresponding to the prediction task.
[0182] Illustratively, after obtaining the multimodal fusion feature representation, the multimodal fusion features are feature processed through subsequent adaptive feature engineering technology to obtain a second feature representation most relevant to the prediction task. The adaptive feature engineering technology is described in detail below.
[0183] Step 2201, input multiple first feature representations into a pre-trained deep learning model, and output multiple high-dimensional feature representations.
[0184] Among them, the feature dimension corresponding to the high-dimensional feature representation is higher than the feature dimension corresponding to the first feature representation, and the deep learning model is used to extract the high-dimensional feature representation corresponding to the first feature representation.
[0185] Illustratively, after obtaining multimodal feature representations corresponding to multiple first feature representations, they are input into a pre-trained deep learning model based on the multimodal feature representations, thereby outputting high-dimensional feature representations corresponding to multimodal fusion features.
[0186] Among them, different deep learning models are used for different modality types. The deep learning models corresponding to different modality types are introduced below.
[0187] It is worth noting that the deep learning model in this step is different from the deep learning model in the above step. The deep learning model in this step is used to capture more abstract and complex patterns in the content data based on the multimodal fusion features. High-dimensional feature representation can more effectively express the key information and data structure in the content data, which helps to improve the generalization ability and model performance of subsequent models.
[0188] (I) Content data in text mode
[0189] Models such as Text Convolutional Neural Network (TextCNN) and Bidirectional Long Short-Term Memory Network (BidirectionalLSTM, BiLSTM) can be used to extract features from multi-fusion feature representations to capture the contextual information corresponding to the text data.
[0190] 2. Content data in image mode
[0191] A deeper convolutional neural network (such as ResNet) can be used to extract features from multi-fusion feature representations to obtain semantic information corresponding to higher-level images.
[0192] (III) Content data in other modes
[0193] For different types of data, select the corresponding deep learning model for feature extraction.
[0194] Step 220, obtaining multiple candidate feature processing methods.
[0195] Each candidate feature processing method includes at least one of a feature screening method and a feature dimension reduction method for the first feature representation.
[0196] Illustratively, different types of candidate feature processing methods are described in detail below.
[0197] (I) Feature screening method
[0198] 1. Mutual Information
[0199] In the feature screening process, the mutual information between the high-dimensional feature representation and the prediction target can be calculated to screen out the feature representation most relevant to the target variable corresponding to the prediction target. The calculation formula of the mutual information can refer to the following formula 7.
[0200] Formula 7: MI(X, Y) = ∑ x∈X ∑ y∈Y P(x,y)*log(P(x,y) / (P(x)*P(y)))
[0201] Among them, X represents the multimodal fusion feature, Y represents the target variable corresponding to the predicted target, P(x, y) represents the joint probability distribution of X and Y, and P(x) and P(y) represent the marginal probability distributions of X and Y respectively.
[0202] 2. Feature Importance
[0203] In some machine learning models (such as decision trees, random forests, etc.), feature importance can be evaluated based on the information gain or Gini index of the input multimodal fusion feature representation, thereby screening based on feature importance.
[0204] (1) Information gain
[0205] Information gain represents the reduction in entropy of the target variable Y given feature X. The calculation formula of information gain can be referred to as the following formula 8.
[0206] Formula 8: IG(Y|X)=H(Y)-H(Y|X)
[0207] Among them, H(Y) represents the entropy of the target variable Y, and H(Y|X) represents the conditional entropy of the target variable Y under the condition of given feature X.
[0208] (2) Gini Index
[0209] The Gini index measures the impurity of the target variable Y under the condition of a given feature X. The calculation formula of the Gini index can be referred to as the following formula 9.
[0210] Formula 9: Gini(Y|X)=∑ x∈X P(x)*(1-∑ y∈Y P(y|x) 2 )
[0211] (II) Feature Dimensionality Reduction Method
[0212] The selected features are further optimized, including principal component analysis (PCA), t-SNE and other methods to reduce feature dimensions and reduce redundancy and noise between features.
[0213] (1) PCA
[0214] PCA maps the original feature space to a new feature space through linear transformation, making the covariance between the new features zero.
[0215] The main calculation steps of PCA are:
[0216] 1. Calculate the covariance matrix C of the original feature matrix X;
[0217] 2. Calculate the eigenvalues and eigenvectors of C;
[0218] 3. Select the eigenvectors corresponding to the first k largest eigenvalues to form the transformation matrix W;
[0219] 4. Multiply the original feature matrix X by the transformation matrix W to obtain the new feature matrix Y.
[0220] The specific calculation process can be referred to the following formula 10:
[0221] Formula 10: Y=X*W
[0222] Among them, X is the original feature matrix, W is the transformation matrix, and Y is the new feature matrix.
[0223] (2) t-SNE
[0224] The main calculation steps of t-SNE are:
[0225] 1. Calculate the similarity probability distribution P between samples in the original feature space (that is, multimodal fusion features);
[0226] 2. Calculate the similarity probability distribution Q between samples in low-dimensional space;
[0227] 3. Minimize the KL divergence (Kullback-Leibler Divergence) between P and Q to obtain a low-dimensional feature representation.
[0228] The volume calculation process can refer to the following formula 11:
[0229] Formula 11: KL(P||Q)=∑ i ∑ j P i j*log(P i j / Q i j)
[0230] Step 231, input the prediction target corresponding to the prediction task, multiple first feature representations, and multiple candidate feature processing methods into a pre-trained reinforcement learning model, and output the target feature processing method.
[0231] Among them, the reinforcement learning model is used to perform fitness analysis on multiple candidate feature processing methods in combination with the prediction target corresponding to the prediction task and multiple first feature representations, and the target feature processing method includes at least one of a target feature screening method and a target feature dimensionality reduction method.
[0232] Schematically, reinforcement learning includes four parameters, namely state, action, reward and policy.
[0233] In this embodiment, the prediction target and multiple high-dimensional feature representations corresponding to the prediction task are used as the state of reinforcement learning, and multiple candidate feature processing methods are input into a pre-trained reinforcement learning model. The reinforcement learning model is combined with the prediction target and the high-dimensional feature representation to analyze the multiple candidate processing methods, and the target feature screening method and target feature dimensionality reduction method with the highest adaptability to the prediction task are determined as the target feature processing method output.
[0234] In an illustrative manner, by adopting a reinforcement learning model, the selection result of the target feature processing method is dynamically adjusted according to different content data and prediction tasks.
[0235] In some embodiments, the reinforcement learning model includes a target policy network and an agent network; the prediction target, multiple first feature representations, and multiple candidate feature processing methods corresponding to the prediction task are input into the target policy network, and the prediction probabilities corresponding to the multiple candidate feature processing methods are output. The target policy network is used to perform fitness analysis on the multiple candidate feature processing methods in combination with the prediction target and multiple first feature representations corresponding to the prediction task; the target feature screening method is determined based on the prediction probabilities corresponding to the multiple candidate feature processing methods; the prediction target, multiple first feature representations, and the prediction probabilities corresponding to the multiple candidate feature processing methods corresponding to the prediction task are input into the agent network, and the target feature dimensionality reduction method is output.
[0236] Optionally, the target policy network includes at least one of network types such as a deep Q network and a policy gradient network.
[0237] The target policy network is used to estimate the Q value corresponding to the action and the state. The Q value represents the long-term reward value when performing an action in a specified state.
[0238] Multiple high-dimensional feature representations and prediction targets corresponding to the prediction tasks are input into the target strategy network, and the Q value corresponding to each candidate feature processing method is output, that is, the prediction probability corresponding to the multiple candidate feature processing methods.
[0239] Among them, the prediction probabilities corresponding to the multiple candidate feature processing methods include the first prediction probabilities (Q1 values) corresponding to the multiple feature screening methods, and the second prediction probabilities (Q2 values) corresponding to the multiple feature dimensionality reduction methods.
[0240] Among them, after obtaining the first prediction probabilities corresponding to multiple feature screening methods, the feature screening method with the highest first prediction probability value is selected as the target feature screening method. That is, the prediction probabilities corresponding to multiple candidate feature processing methods include the first prediction probabilities corresponding to multiple feature screening methods; from the first prediction probabilities corresponding to multiple feature screening methods, the feature screening method with the highest first prediction probability value is determined as the target feature screening method.
[0241] After obtaining the second prediction probabilities corresponding to the multiple feature dimensionality reduction methods, the second prediction probabilities corresponding to the multiple feature dimensionality reduction methods, the multiple high-dimensional feature representations, and the prediction targets corresponding to the prediction tasks are input into the intelligent agent network, and the target feature dimensionality reduction method is output. That is, the prediction probabilities corresponding to the multiple candidate feature processing methods include the second prediction probabilities corresponding to the multiple feature dimensionality reduction methods; the prediction targets corresponding to the prediction tasks, the multiple first feature representations, and the second prediction probabilities corresponding to the multiple feature dimensionality reduction methods are input into the intelligent agent network, and the target feature dimensionality reduction method is output, and the intelligent agent network is used to predict the target feature dimensionality reduction method that meets the prediction target based on the prediction probabilities corresponding to the multiple first feature representations and the multiple candidate feature processing methods.
[0242] Step 241 , performing correlation screening on a plurality of first feature representations by a target feature screening method, and obtaining a plurality of first sub-feature representations meeting a correlation condition.
[0243] The correlation condition is used to indicate the correlation between the first feature representation and the prediction target.
[0244] Illustratively, after determining a target feature screening method from the above-mentioned feature screening methods through a reinforcement learning model, multiple high-dimensional feature representations are screened for correlation through the target feature screening method, thereby obtaining multiple first sub-feature representations that meet the correlation conditions.
[0245] Step 242 , performing feature dimensionality reduction on the plurality of first sub-feature representations by using a target feature dimensionality reduction method to obtain a plurality of second feature representations.
[0246] Illustratively, after determining a target feature dimensionality reduction method from the above-mentioned feature dimensionality reduction methods through a reinforcement learning model, feature dimensionality reduction is performed on the selected multiple first sub-feature representations through the target feature dimensionality reduction method, thereby obtaining multiple second feature representations.
[0247] Step 250 , performing content prediction on the plurality of second feature representations to obtain a prediction result corresponding to the prediction task.
[0248] The prediction result is used to indicate the analysis result corresponding to the prediction task obtained through multiple content data.
[0249] The process of content prediction in step 250 has been described in detail in the above embodiment, and will not be repeated in this embodiment.
[0250] In summary, the content prediction method provided in the embodiment of the present application, after obtaining multiple content data under at least two modes corresponding to the prediction task and extracting the first feature representations respectively corresponding to the multiple content data, obtains multiple candidate feature processing methods, and selects the target feature processing method from them according to the prediction target corresponding to the prediction task and the data attribute corresponding to the content data, thereby performing feature processing on the multiple first feature representations according to the feature screening method and / or feature dimension reduction method in the target feature processing method, obtaining multiple second feature representations, and finally performing content prediction according to the second feature representation to obtain the prediction result corresponding to the prediction task. That is, after extracting the first feature representations respectively corresponding to the multiple content data corresponding to the prediction task, the most suitable feature processing method is selected according to the prediction task and the content data to perform feature processing on the first feature representation again, so that the processed second feature representation is more in line with the prediction task, thereby improving the accuracy of the final predicted content, and according to different prediction tasks and different content data, the selection result of the feature processing method can be dynamically adjusted.
[0251] In this embodiment, a pre-trained reinforcement learning model is used to combine the prediction target corresponding to the prediction task and multiple first feature representations to perform a fitness analysis on multiple candidate feature processing methods, so as to select the target feature processing method with the highest fitness for the prediction task, thereby improving the accuracy of the subsequent second feature representation and ultimately improving the accuracy of content prediction.
[0252] In an optional embodiment, the reinforcement learning model needs to be trained. For example, please refer to Figure 4 , which shows a flow chart of a content prediction method provided by an exemplary embodiment of the present application, such as Figure 4 As shown, the method includes the following steps.
[0253] Step 410: Acquire multiple sample content data corresponding to the sample prediction task, and extract first feature representations corresponding to the multiple sample content data respectively.
[0254] Among them, the multiple sample content data correspond to at least two modalities.
[0255] Schematically, a sample prediction task refers to a task used to train a sample reinforcement learning model.
[0256] In this embodiment, after the sample prediction task is obtained, a plurality of sample content data in at least two modes corresponding to the sample prediction task are obtained.
[0257] In this embodiment, the first feature representations respectively corresponding to the plurality of sample content data are extracted by the feature extraction method mentioned in the above embodiment.
[0258] After obtaining the first feature representations respectively corresponding to the plurality of sample content data, the first feature representations having the same feature representation are feature concatenated to obtain a multimodal feature representation.
[0259] The multimodal feature representation is input into a pre-trained deep learning model, and the high-dimensional feature representation corresponding to the multimodal features is output.
[0260] The following is a detailed description of the training process of the sample reinforcement learning model. It is worth noting that the following training process only represents a single round of training process, and the training process of the entire reinforcement learning model includes multiple rounds of training processes as described below.
[0261] Step 420, obtaining multiple sample feature processing methods.
[0262] Among them, each sample feature processing method includes at least one of a sample feature screening method and a sample feature dimensionality reduction method.
[0263] Illustratively, a plurality of pre-stored feature processing methods are obtained as sample feature processing methods.
[0264] Step 430, the prediction target corresponding to the sample prediction task, multiple first feature representations and multiple sample feature processing methods are input into the sample strategy network, and the prediction probabilities corresponding to the multiple sample feature processing methods are output.
[0265] In this embodiment, the prediction target corresponding to the sample prediction task and multiple high-dimensional feature representations are used as states in the sample reinforcement learning model.
[0266] The sample reinforcement learning model includes a sample policy network and a sample agent network. The prediction target corresponding to the sample prediction task and multiple high-dimensional feature representations are input into the sample policy network, and the prediction probabilities corresponding to multiple sample feature processing methods are output.
[0267] Step 440, input the prediction target corresponding to the sample prediction task, multiple first feature representations, and prediction probabilities corresponding to multiple sample feature processing methods into the sample agent network, and output the prediction result of the feature dimensionality reduction method.
[0268] In this embodiment, the prediction target corresponding to the sample prediction task, multiple first feature representations, and the prediction probabilities corresponding to multiple sample feature processing methods are input into the sample agent network, and the prediction result corresponding to the feature dimensionality reduction method is output, that is, the target feature dimensionality reduction method.
[0269] Step 450, generating a target reward function based on the prediction result of the feature dimensionality reduction method.
[0270] In this embodiment, the feature processing method selection process is used as the environment in reinforcement learning, and the state of the next round and the target reward function are generated by the prediction results corresponding to the feature dimensionality reduction method output by the sample intelligent agent network.
[0271] Step 460: Train the sample policy network based on the target reward function to obtain a target policy network.
[0272] The target reward function is used to update the network parameters of the sample policy network of this round, and the training termination condition is set. When the network parameters of the sample policy network meet the training termination condition, the training process ends.
[0273] Among them, the training termination condition includes setting at least one of the condition types such as a training round number threshold, a model performance threshold, etc.
[0274] In summary, the content prediction method provided in the embodiment of the present application, after obtaining multiple content data under at least two modes corresponding to the prediction task and extracting the first feature representations respectively corresponding to the multiple content data, obtains multiple candidate feature processing methods, and selects the target feature processing method from them according to the prediction target corresponding to the prediction task and the data attribute corresponding to the content data, thereby performing feature processing on the multiple first feature representations according to the feature screening method and / or feature dimension reduction method in the target feature processing method, obtaining multiple second feature representations, and finally performing content prediction according to the second feature representation to obtain the prediction result corresponding to the prediction task. That is, after extracting the first feature representations respectively corresponding to the multiple content data corresponding to the prediction task, the most suitable feature processing method is selected according to the prediction task and the content data to perform feature processing on the first feature representation again, so that the processed second feature representation is more in line with the prediction task, thereby improving the accuracy of the final predicted content, and according to different prediction tasks and different content data, the selection result of the feature processing method can be dynamically adjusted.
[0275] The beneficial effects provided by this application include the following:
[0276] (1) Efficient implementation: When implementing multimodal adaptive feature engineering technology, efficient programming languages and optimization algorithms can be used to improve the operating efficiency of the system.
[0277] (2) Parallel computing: Use multi-core CPU, GPU and other hardware resources to implement parallel computing and accelerate the feature engineering process.
[0278] (3) Model fusion: Multiple basic models are fused to improve the overall performance. For example, stacking, bagging, and other methods can be used for model fusion.
[0279] (4) Online learning: In order to adapt to the dynamic changes of data and tasks, online learning methods can be used to update the model in real time.
[0280] In some embodiments, the content prediction method is applied to an image recognition scenario as an example for illustration. For example, please refer to Figure 5 , which shows a flow chart of a content prediction method provided by an exemplary embodiment of the present application, such as Figure 5 As shown, the method includes the following steps.
[0281] Step 510: Acquire a plurality of content data corresponding to the first image, and extract first feature representations respectively corresponding to the plurality of content data.
[0282] Among them, multiple content data correspond to at least two modes.
[0283] Illustratively, a first image is acquired, and the prediction task is to identify categories corresponding to all objects contained in the first image.
[0284] Acquire content data in an image mode, content data in a text mode, and content data in an audio mode corresponding to the first image.
[0285] First, the plurality of content data are preprocessed respectively by using different data preprocessing methods to obtain a plurality of preprocessed data.
[0286] The first feature representations corresponding to the plurality of preprocessed data are extracted through a deep learning model.
[0287] After obtaining multiple first feature representations, feature concatenation is performed on first feature representations with the same feature representation to obtain multiple multimodal fusion feature representations.
[0288] Step 520, obtaining multiple candidate feature processing methods.
[0289] Each candidate feature processing method includes at least one of a feature screening method and a feature dimension reduction method for the first feature representation.
[0290] A plurality of pre-stored feature processing methods are obtained as candidate feature processing methods, including a feature screening method and a feature dimensionality reduction method.
[0291] Step 530 : determining a target feature processing method from a plurality of candidate feature processing methods based on the recognized object in the first image and the data attributes corresponding to the content data.
[0292] The data attribute is used to indicate the data category of the content data in the image recognition task.
[0293] First, the high-dimensional feature representation corresponding to the multimodal fusion feature representation is extracted through a pre-trained deep learning model.
[0294] The first image, the high-dimensional feature representation, and a plurality of candidate feature representations are input into a pre-trained reinforcement learning model, and a target feature processing method is output.
[0295] Step 540 , performing feature processing on the plurality of first feature representations by using a target feature processing method to obtain a plurality of second feature representations.
[0296] The high-dimensional feature representation is successively subjected to feature screening and feature dimensionality reduction through a target feature screening method and a target feature dimensionality reduction method corresponding to the target feature processing method to obtain a plurality of second feature representations.
[0297] Step 550: Perform image recognition on the plurality of second feature representations to obtain a recognition result corresponding to the first image, where the recognition result is used to indicate a category corresponding to the recognized object in the first image obtained through the plurality of content data.
[0298] The plurality of second feature representations are input into a pre-trained image recognition model for image recognition, and the categories corresponding to the objects in the first image are obtained as output.
[0299] In another embodiment, the adaptive feature selection engineering provided by the present application can also be applied to the training process of the content prediction model. First, according to the deep learning model adapted to the prediction task, such as support vector machine (SVM), neural network model, decision tree model, etc., the feature representation optimized by the adaptive feature selection engineering is input into the content prediction model for training, thereby obtaining a trained content prediction model, and finally the input data is predicted by the trained content prediction model to obtain the prediction result. Finally, the performance of the model is evaluated using indicators such as accuracy, F1 value, and ROC curve, so as to perform subsequent model optimization.
[0300] In summary, the content prediction method provided in the embodiment of the present application, after obtaining multiple content data under at least two modes corresponding to the prediction task and extracting the first feature representations respectively corresponding to the multiple content data, obtains multiple candidate feature processing methods, and selects the target feature processing method from them according to the prediction target corresponding to the prediction task and the data attribute corresponding to the content data, thereby performing feature processing on the multiple first feature representations according to the feature screening method and / or feature dimension reduction method in the target feature processing method, obtaining multiple second feature representations, and finally performing content prediction according to the second feature representation to obtain the prediction result corresponding to the prediction task. That is, after extracting the first feature representations respectively corresponding to the multiple content data corresponding to the prediction task, the most suitable feature processing method is selected according to the prediction task and the content data to perform feature processing on the first feature representation again, so that the processed second feature representation is more in line with the prediction task, thereby improving the accuracy of the final predicted content, and according to different prediction tasks and different content data, the selection result of the feature processing method can be dynamically adjusted.
[0301] Figure 6 is a structural block diagram of a content prediction device provided by an exemplary embodiment of the present application. Figure 6 As shown, the device comprises:
[0302] A first acquisition module 610 is used to acquire a plurality of content data corresponding to the prediction task and extract first feature representations respectively corresponding to the plurality of content data, wherein the plurality of content data corresponds to at least two modalities;
[0303] The first acquisition module 610 is further used to acquire multiple candidate feature processing methods, each candidate feature processing method includes at least one of a feature screening method and a feature dimensionality reduction method for the first feature representation;
[0304] A first determination module 620, configured to determine a target feature processing method from the plurality of candidate feature processing methods based on a prediction target corresponding to the prediction task and a data attribute corresponding to the content data, wherein the data attribute is used to indicate a data category of the content data under the prediction task;
[0305] A first processing module 630 is used to perform feature processing on the plurality of first feature representations by using the target feature processing method to obtain a plurality of second feature representations;
[0306] The prediction module 640 is used to perform content prediction on the multiple second feature representations to obtain a prediction result corresponding to the prediction task, wherein the prediction result is used to indicate an analysis result corresponding to the prediction task obtained through the multiple content data.
[0307] In some embodiments, Figure 7As shown, the first determination module 620 includes:
[0308] The output unit 621 is used to input the prediction target corresponding to the prediction task, multiple first feature representations, and the multiple candidate feature processing methods into a pre-trained reinforcement learning model, and output the target feature processing method. The reinforcement learning model is used to perform fitness analysis on the multiple candidate feature processing methods in combination with the prediction target corresponding to the prediction task and the multiple first feature representations. The target feature processing method includes at least one of a target feature screening method and a target feature dimensionality reduction method.
[0309] In some embodiments, the reinforcement learning model includes a target policy network and an agent network;
[0310] The output unit 621 is also used to input the prediction target corresponding to the prediction task, multiple first feature representations and the multiple candidate feature processing methods into the target strategy network, and output the prediction probabilities corresponding to the multiple candidate feature processing methods. The target strategy network is used to perform the fitness analysis on the multiple candidate feature processing methods in combination with the prediction target and multiple first feature representations corresponding to the prediction task; determine the target feature screening method based on the prediction probabilities corresponding to the multiple candidate feature processing methods; input the prediction target corresponding to the prediction task, multiple first feature representations and the prediction probabilities corresponding to the multiple candidate feature processing methods into the intelligent agent network, and output the target feature dimensionality reduction method. The intelligent agent network is used to predict the target feature dimensionality reduction method that meets the prediction target based on the prediction probabilities corresponding to the multiple first feature representations and the multiple candidate feature processing methods.
[0311] In some embodiments, the prediction probabilities corresponding to the plurality of candidate feature processing methods include first prediction probabilities corresponding to a plurality of feature screening methods;
[0312] The output unit 621 is further configured to determine, from among the first prediction probabilities respectively corresponding to the multiple feature screening methods, a feature screening method having the highest first prediction probability value as the target feature screening method.
[0313] In some embodiments, the prediction probabilities corresponding to the plurality of candidate feature processing methods include second prediction probabilities corresponding to a plurality of feature dimensionality reduction methods;
[0314] The output unit 621 is also used to input the prediction target corresponding to the prediction task, multiple first feature representations, and the second prediction probabilities corresponding to the multiple feature dimensionality reduction methods into the intelligent agent network, and output the target feature dimensionality reduction method.
[0315] In some embodiments, the apparatus further comprises:
[0316] The first acquisition module 610 is used to acquire a plurality of sample content data corresponding to the sample prediction task, and extract first feature representations respectively corresponding to the plurality of sample content data, wherein the plurality of sample content data correspond to at least two modalities;
[0317] The first acquisition module 610 is further used to acquire multiple sample feature processing methods, each of which includes at least one of a sample feature screening method and a sample feature dimensionality reduction method;
[0318] An output module 650 is used to input the prediction target corresponding to the sample prediction task, the multiple first feature representations, and the multiple sample feature processing methods into a sample strategy network, and output the prediction probabilities corresponding to the multiple sample feature processing methods respectively;
[0319] The output module 650 is further used to input the prediction target corresponding to the sample prediction task, the multiple first feature representations, and the prediction probabilities corresponding to the multiple sample feature processing methods into the agent network, and output the prediction result obtained by the feature dimensionality reduction method;
[0320] A generating module 660 generates a target reward function based on the prediction result of the feature dimensionality reduction method;
[0321] The training module 670 trains the sample policy network based on the target reward function to obtain the target policy network.
[0322] In some embodiments, the first processing module 630 is further used to perform correlation screening on the multiple first feature representations by using the target feature screening method to obtain multiple first sub-feature representations that meet the correlation condition, wherein the correlation condition is used to indicate the correlation between the first feature representation and the prediction target; and perform feature dimensionality reduction on the multiple first sub-feature representations by using the target feature dimensionality reduction method to obtain the multiple second feature representations.
[0323] In some embodiments, the output module 650 is also used to input the multiple first feature representations into a pre-trained deep learning model, and output multiple high-dimensional feature representations, wherein the feature dimensions corresponding to the high-dimensional feature representations are higher than the feature dimensions corresponding to the first feature representations, and the deep learning model is used to extract the high-dimensional feature representations corresponding to the first feature representations.
[0324] In some embodiments, the first acquisition module 610 is used to obtain feature identifiers corresponding to the multiple first feature representations, where the feature identifiers are used to indicate feature dimensions corresponding to the first feature representations; and to perform feature splicing on the first feature representations having the same feature identifier in the first feature representations corresponding to the multiple content data to obtain a multi-fusion feature representation.
[0325] Figure 8 is a structural block diagram of a content prediction device provided by an exemplary embodiment of the present application. Figure 6 As shown, the device comprises:
[0326] A second acquisition module 810 is used to acquire a plurality of content data corresponding to the first image, and extract first feature representations respectively corresponding to the plurality of content data, wherein the plurality of content data corresponds to at least two modalities;
[0327] The second acquisition module 810 is further used to acquire multiple candidate feature processing methods, each candidate feature processing method includes at least one of a feature screening method and a feature dimensionality reduction method for the first feature representation;
[0328] A second determination module 820, configured to determine a target feature processing method from the plurality of candidate feature processing methods based on a recognition object in the first image and a data attribute corresponding to the content data, wherein the data attribute is used to indicate a data category of the content data under the image recognition task;
[0329] A second processing module 830 is used to perform feature processing on the multiple first feature representations by using the target feature processing method to obtain multiple second feature representations;
[0330] The recognition module 840 is used to perform image recognition on the multiple second feature representations to obtain a recognition result corresponding to the first image, wherein the recognition result is used to indicate a category corresponding to the recognized object in the first image obtained through the multiple content data.
[0331] In summary, the content prediction device provided in the embodiment of the present application, after obtaining multiple content data under at least two modes corresponding to the prediction task and extracting the first feature representations respectively corresponding to the multiple content data, obtains multiple candidate feature processing methods, and selects the target feature processing method from them according to the prediction target corresponding to the prediction task and the data attribute corresponding to the content data, thereby performing feature processing on the multiple first feature representations according to the feature screening method and / or feature dimension reduction method in the target feature processing method, obtaining multiple second feature representations, and finally performing content prediction according to the second feature representation to obtain the prediction result corresponding to the prediction task. That is, after extracting the first feature representations respectively corresponding to the multiple content data corresponding to the prediction task, the most suitable feature processing method is selected according to the prediction task and the content data to perform feature processing on the first feature representation again, so that the processed second feature representation is more in line with the prediction task, thereby improving the accuracy of the final predicted content, and according to different prediction tasks and different content data, the selection result of the feature processing method can be dynamically adjusted.
[0332] It should be noted that the content prediction device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the content prediction device provided in the above embodiment belongs to the same concept as the content prediction method embodiment. The specific implementation process is detailed in the method embodiment and will not be repeated here.
[0333] Fig. 9 The following is a schematic diagram showing the structure of a server provided by an exemplary embodiment of the present application. Specifically:
[0334] The server 900 includes a central processing unit (CPU) 901, a system memory 904 including a random access memory (RAM) 902 and a read only memory (ROM) 903, and a system bus 905 connecting the system memory 904 and the central processing unit 901. The server 900 also includes a mass storage device 906 for storing an operating system 913, application programs 914, and other program modules 915.
[0335] The mass storage device 906 is connected to the central processing unit 901 through a mass storage controller (not shown) connected to the system bus 905. The mass storage device 906 and its associated computer readable media provide non-volatile storage for the server 900. That is, the mass storage device 906 may include a computer readable medium (not shown) such as a hard disk or a compact disc read only memory (CD-ROM) drive.
[0336] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other solid-state storage technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory 904 and mass storage device 906 can be collectively referred to as memory.
[0337] According to various embodiments of the present application, the server 900 can also be connected to a remote computer on the network through a network such as the Internet. That is, the server 900 can be connected to the network 912 through the network interface unit 911 connected to the system bus 905, or the network interface unit 911 can be used to connect to other types of networks or remote computer systems (not shown).
[0338] The above-mentioned memory also includes one or more programs, and the one or more programs are stored in the memory and configured to be executed by the CPU.
[0339] An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the content prediction method provided by the above-mentioned method embodiments.
[0340] An embodiment of the present application also provides a computer-readable storage medium, on which is stored at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the content prediction method provided by the above-mentioned method embodiments.
[0341] The embodiment of the present application also provides a computer program product, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs any of the content prediction methods described in the above embodiments.
[0342] Optionally, the computer readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.
[0343] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0344] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.
Claims
1. A content prediction method, characterized in that: The method comprises: Acquire multiple content data corresponding to the prediction task, and extract first feature representations respectively corresponding to the multiple content data, wherein the multiple content data correspond to at least two modalities; Acquire multiple candidate feature processing methods, each candidate feature processing method including at least one of a feature screening method and a feature dimensionality reduction method for the first feature representation; Determining a target feature processing method from the plurality of candidate feature processing methods based on a prediction target corresponding to the prediction task and a data attribute corresponding to the content data, wherein the data attribute is used to indicate a data category corresponding to the content data under the prediction task; Performing feature processing on the plurality of first feature representations by using the target feature processing method to obtain a plurality of second feature representations; Content prediction is performed on the multiple second feature representations to obtain a prediction result corresponding to the prediction task, where the prediction result is used to indicate an analysis result corresponding to the prediction task obtained through the multiple content data.
2. The method according to claim 1, characterized in that The determining of a target feature processing method from the plurality of candidate feature processing methods based on the prediction target corresponding to the prediction task and the data attribute corresponding to the content data includes: The prediction target corresponding to the prediction task, multiple first feature representations, and the multiple candidate feature processing methods are input into a pre-trained reinforcement learning model, and the target feature processing method is output. The reinforcement learning model is used to perform fitness analysis on the multiple candidate feature processing methods in combination with the prediction target corresponding to the prediction task and the multiple first feature representations. The target feature processing method includes at least one of a target feature screening method and a target feature dimensionality reduction method.
3. The method according to claim 2, characterized in that The reinforcement learning model includes a target strategy network and an agent network; The step of inputting the prediction target corresponding to the prediction task, the plurality of first feature representations, and the plurality of candidate feature processing methods into a pre-trained reinforcement learning model, and outputting the target feature processing method, comprises: Inputting the prediction target corresponding to the prediction task, the multiple first feature representations, and the multiple candidate feature processing methods into the target strategy network, outputting the prediction probabilities corresponding to the multiple candidate feature processing methods, respectively, the target strategy network being used to perform the fitness analysis on the multiple candidate feature processing methods in combination with the prediction target corresponding to the prediction task and the multiple first feature representations; Determining the target feature screening method based on the prediction probabilities corresponding to the multiple candidate feature processing methods; The prediction target corresponding to the prediction task, multiple first feature representations, and the prediction probabilities corresponding to the multiple candidate feature processing methods are input into the intelligent agent network, and the target feature dimensionality reduction method is output. The intelligent agent network is used to predict the target feature dimensionality reduction method that meets the prediction target based on the prediction probabilities corresponding to the multiple first feature representations and the multiple candidate feature processing methods.
4. The method according to claim 3, characterized in that The prediction probabilities corresponding to the plurality of candidate feature processing methods include first prediction probabilities corresponding to the plurality of feature screening methods; The determining the target feature screening method based on the prediction probabilities respectively corresponding to the multiple candidate feature processing methods includes: The feature screening method having the highest first prediction probability value among the first prediction probabilities respectively corresponding to the multiple feature screening methods is determined as the target feature screening method.
5. The method according to claim 3, characterized in that: The prediction probabilities corresponding to the plurality of candidate feature processing methods include second prediction probabilities corresponding to the plurality of feature dimensionality reduction methods; The step of inputting the prediction target corresponding to the prediction task, the plurality of first feature representations, and the prediction probabilities respectively corresponding to the plurality of candidate feature processing methods into the agent network, and outputting the target feature dimensionality reduction method, comprises: The prediction target corresponding to the prediction task, multiple first feature representations, and second prediction probabilities corresponding to the multiple feature dimensionality reduction methods are input into the intelligent agent network, and the target feature dimensionality reduction method is output.
6. The method according to claim 3, characterized in that Before inputting the prediction target corresponding to the prediction task, the plurality of first feature representations and the plurality of candidate feature processing methods into the feature processing network and outputting the prediction probabilities respectively corresponding to the plurality of candidate feature processing methods, the method further includes: Acquire multiple sample content data corresponding to the sample prediction task, and extract first feature representations respectively corresponding to the multiple sample content data, wherein the multiple sample content data correspond to at least two modalities; Acquire multiple sample feature processing methods, each of which includes at least one of a sample feature screening method and a sample feature dimensionality reduction method; Inputting the prediction target corresponding to the sample prediction task, the multiple first feature representations, and the multiple sample feature processing methods into the sample strategy network, and outputting the prediction probabilities corresponding to the multiple sample feature processing methods respectively; Inputting the prediction target corresponding to the sample prediction task, the multiple first feature representations, and the prediction probabilities corresponding to the multiple sample feature processing methods into the agent network, and outputting the prediction result of the feature dimensionality reduction method; Generate a target reward function based on the prediction results of the feature dimensionality reduction method; The sample policy network is trained based on the target reward function to obtain the target policy network.
7. The method according to any one of claims 2 to 6, characterized in that: The step of performing feature processing on the plurality of first feature representations by using the target feature processing method to obtain a plurality of second feature representations includes: Performing correlation screening on the plurality of first feature representations by the target feature screening method to obtain a plurality of first sub-feature representations that meet a correlation condition, wherein the correlation condition is used to indicate the correlation between the first feature representation and the prediction target; The plurality of first sub-feature representations are subjected to feature dimensionality reduction by using the target feature dimensionality reduction method to obtain the plurality of second feature representations.
8. The method according to any one of claims 1 to 6, characterized in that: Before obtaining multiple candidate feature processing methods, the method further includes: The multiple first feature representations are input into a pre-trained deep learning model, and multiple high-dimensional feature representations are output, wherein the feature dimensions corresponding to the high-dimensional feature representations are higher than the feature dimensions corresponding to the first feature representations, and the deep learning model is used to extract the high-dimensional feature representations corresponding to the first feature representations.
9. The method according to any one of claims 1 to 6, characterized in that: After extracting the first feature representations respectively corresponding to the plurality of content data, the method further includes: Acquire feature identifiers corresponding to the plurality of first feature representations respectively, where the feature identifiers are used to indicate feature dimensions corresponding to the first feature representations; The first feature representations having the same feature identifier in the first feature representations respectively corresponding to the multiple content data are feature-concatenated to obtain a multi-fusion feature representation.
10. A content prediction method, characterized in that: The method is applied to an image recognition scenario, and the method comprises: Acquire a plurality of content data corresponding to the first image, and extract first feature representations respectively corresponding to the plurality of content data, wherein the plurality of content data correspond to at least two modalities; Acquire multiple candidate feature processing methods, each candidate feature processing method including at least one of a feature screening method and a feature dimensionality reduction method for the first feature representation; Determining a target feature processing method from the plurality of candidate feature processing methods based on the recognition object in the first image and a data attribute corresponding to the content data, the data attribute being used to indicate a data category of the content data under the image recognition task; Performing feature processing on the plurality of first feature representations by using the target feature processing method to obtain a plurality of second feature representations; Image recognition is performed on the multiple second feature representations to obtain a recognition result corresponding to the first image, where the recognition result is used to indicate a category corresponding to an identified object in the first image obtained through the multiple content data.
11. A content prediction device, characterized in that: The device comprises: A first acquisition module, configured to acquire a plurality of content data corresponding to the prediction task, and extract first feature representations respectively corresponding to the plurality of content data, wherein the plurality of content data correspond to at least two modes; The first acquisition module is further used to acquire multiple candidate feature processing methods, each candidate feature processing method includes at least one of a feature screening method and a feature dimensionality reduction method for the first feature representation; A first determination module, configured to determine a target feature processing method from the plurality of candidate feature processing methods based on a prediction target corresponding to the prediction task and a data attribute corresponding to the content data, wherein the data attribute is used to indicate a data category of the content data under the prediction task; A first processing module, configured to perform feature processing on the plurality of first feature representations by using the target feature processing method to obtain a plurality of second feature representations; A prediction module is used to perform content prediction on the multiple second feature representations to obtain a prediction result corresponding to the prediction task, wherein the prediction result is used to indicate an analysis result corresponding to the prediction task obtained through the multiple content data.
12. A content prediction device, characterized in that: The device is applied to an image recognition scenario, and comprises: A second acquisition module is used to acquire a plurality of content data corresponding to the first image, and extract first feature representations respectively corresponding to the plurality of content data, wherein the plurality of content data corresponds to at least two modalities; The second acquisition module is further used to acquire multiple candidate feature processing methods, each candidate feature processing method includes at least one of a feature screening method and a feature dimensionality reduction method for the first feature representation; A second determination module, configured to determine a target feature processing method from the plurality of candidate feature processing methods based on a recognition object in the first image and a data attribute corresponding to the content data, wherein the data attribute is used to indicate a data category of the content data under the image recognition task; A second processing module, configured to perform feature processing on the plurality of first feature representations by using the target feature processing method to obtain a plurality of second feature representations; The recognition module is used to perform image recognition on the multiple second feature representations to obtain a recognition result corresponding to the first image, wherein the recognition result is used to indicate a category corresponding to the recognized object in the first image obtained through the multiple content data.
13. A computer device, characterized in that: The computer device includes a processor and a memory, wherein at least one program is stored in the memory, and the at least one program is loaded and executed by the processor to implement the content prediction method according to any one of claims 1 to 10.
14. A computer-readable storage medium, characterized in that: The storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the content prediction method according to any one of claims 1 to 10.
15. A computer program product, characterized in that The invention comprises a computer program, which implements the content prediction method according to any one of claims 1 to 10 when being executed by a processor.
