A content push method, feature vector determination method and related devices

Abstract

The embodiments of the present application disclose a content push method, a feature vector determination method and related devices, which at least involve machine learning in artificial intelligence, obtain target content to be processed, and the target content includes first information from a first modality and second information from a second modality; generate a first undetermined vector corresponding to the first information, and a second undetermined vector corresponding to the second information; determine a first activation vector based on the first undetermined vector, and the first activation vector includes a first weight parameter determined by the first undetermined vector; adjust the second undetermined vector based on the first activation vector to obtain a second modal vector, wherein the first weight parameter is used to enhance key information in the second undetermined vector and suppress non-key information in the second undetermined vector; generate a feature vector corresponding to the target content based on the first undetermined vector and the second modal vector, thereby achieving better fusion of undetermined vectors of different modalities and improving the accuracy of the feature vector.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a content pushing method, a feature vector determination method, and related devices. Background Art

[0002] All kinds of content can be disseminated on the Internet, and a vectorized expression of the content can be generated based on the content: a feature vector. This feature vector can quantitatively express the relevant information of the corresponding content, so the feature vector can be applied to a large number of content processing scenarios, such as content classification and content push through the feature vector of the content.

[0003] It can be seen that the accuracy of feature vectors directly affects subsequent content processing scenarios. However, currently, much content contains related information from multiple dimensions. For example, the relevant information of a video can come from two dimensions: the video title and the video frame. The text vector is extracted based on the video title, while the image vector is extracted based on the video frame. Both the text vector and the image vector are needed to construct the feature vector that represents the video.

[0004] Vectors generated based on different sources (such as text and video frames) like the one in the above example are multimodal vectors. How to reasonably construct a feature vector corresponding to the content through multimodal vectors is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] In order to solve the above technical problems, the present application provides a content push method, a feature vector determination method and related devices for reasonably constructing a feature vector corresponding to a content through a multimodal vector.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] In one aspect, the present application provides a method for determining a feature vector, the method comprising:

[0008] Acquire target content to be processed, where the target content includes first information from a first modality and second information from a second modality;

[0009] Generate a first undetermined vector corresponding to the first information and a second undetermined vector corresponding to the second information;

[0010] determining a first activation vector according to the first undetermined vector, wherein the first activation vector includes a first weight parameter determined by the first undetermined vector;

[0011] Adjusting the second undetermined vector by using the first activation vector to obtain a second modal vector, wherein the first weight parameter is used to enhance key information in the second undetermined vector and suppress non-key information in the second undetermined vector;

[0012] A feature vector corresponding to the target content is generated according to the first undetermined vector and the second modal vector.

[0013] On the other hand, the present application provides a feature vector determination device, the device comprising: an acquisition unit, a first generation unit, a first determination unit, an adjustment unit, and a second generation unit;

[0014] The acquisition unit is configured to acquire target content to be processed, wherein the target content includes first information from a first modality and second information from a second modality;

[0015] The first generating unit is configured to generate a first undetermined vector corresponding to the first information and a second undetermined vector corresponding to the second information;

[0016] The first determining unit is configured to determine a first activation vector according to the first undetermined vector, where the first activation vector includes a first weight parameter determined by the first undetermined vector;

[0017] The adjustment unit is configured to adjust the second undetermined vector by using the first activation vector to obtain a second modal vector, wherein the first weight parameter is used to enhance key information in the second undetermined vector and suppress non-key information in the second undetermined vector;

[0018] The second generating unit is configured to generate a feature vector corresponding to the target content according to the first undetermined vector and the second modal vector.

[0019] On the other hand, the present application provides a content push method, the method comprising:

[0020] Determine the object characteristics of the target object;

[0021] Determining target content associated with the object feature from the pending content based on a feature vector of the pending content; the target content includes first information from a first modality and second information from a second modality; the feature vector of the target content is generated based on the first pending vector and a second modality vector; the first pending vector corresponds to the first information; the second pending vector corresponds to the second information; the second modality vector is obtained by adjusting the second pending vector using a first activation vector; the first activation vector includes a first weight parameter determined based on the first pending vector; the first weight parameter is used to enhance key information in the second pending vector and suppress non-key information in the second pending vector;

[0022] The target content is returned based on the target object.

[0023] On the other hand, the present application provides a content push device, the device comprising: a determination unit and a return unit;

[0024] The determining unit is configured to determine an object feature of a target object; and determine, from the to-be-determined content, target content associated with the object feature based on a feature vector of the to-be-determined content; the target content includes first information from a first modality and second information from a second modality, the feature vector of the target content is generated based on a first to-be-determined vector and a second modality vector, the first to-be-determined vector corresponds to the first information, the second to-be-determined vector corresponds to the second information, the second modality vector is obtained by adjusting the second to-be-determined vector by a first activation vector, the first activation vector includes a first weight parameter determined based on the first to-be-determined vector, the first weight parameter being used to enhance key information in the second to-be-determined vector and suppress non-key information in the second to-be-determined vector;

[0025] The returning unit is configured to return the target content based on the target object.

[0026] In another aspect, the present application provides a computer device, comprising a processor and a memory:

[0027] The memory is used to store program code and transmit the program code to the processor;

[0028] The processor is configured to execute the method described above according to the instructions in the program code.

[0029] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the method described in the above aspects.

[0030] In another aspect, embodiments of the present application provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described in the above aspects.

[0031] It can be seen from the above technical solution that the object characteristics of the target object are determined, and the target content is determined from the pending content based on the association between the feature vector of the pending content and the object characteristics. The target content for which the feature vector is to be generated has the first information from the first modality and the second information from the second modality. It is necessary to generate the feature vector of the target content by integrating the first information and the second information. To this end, a first pending vector corresponding to the first information and a second pending vector corresponding to the second information are generated, and a feature vector of the target content is generated based on the first pending vector and the second pending vector. Since information from different modalities has an expression method corresponding to the modality in which it is located, different expressions will differ in expression characteristics and emphasis. In order to avoid the adverse effect of different expressions on the generated feature vectors due to the dispersion of expression emphasis, it is necessary to forward fuse the pending vectors from different modalities. The first activation vector can be determined based on the first pending vector. The first activation vector includes a first weight parameter determined by the first pending vector. The first weight parameter is determined by the criticality of the information in the first pending vector, which can reflect the focus of the information in the first pending vector. The information included in the second pending feature vector can be adjusted in a targeted manner based on the focus of the first pending feature vector, thereby enhancing the key information in the second pending vector and suppressing non-key information. The second modal vector obtained by adjustment gradually converges with the expression focus of the first pending vector, achieving a better fusion of the pending vectors of different modalities, strengthening the key information related to the target content theme in the generated feature vector, and improving the accuracy of the feature vector. Therefore, the target content corresponding to the feature vector is more likely to meet the needs of the target object, and returning the target content to the target object can ensure the accuracy of the push of the target content. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0033] Figure 1 A schematic diagram of an application scenario of the feature vector determination method provided in an embodiment of the present application;

[0034] Figure 2 A flowchart of a method for determining a feature vector provided in an embodiment of the present application;

[0035] Figure 3 Schematic diagram of a multimodal vector fusion method;

[0036] Figure 4 A schematic diagram of an application scenario of a method for determining a feature vector provided in an embodiment of the present application;

[0037] Figure 5 A flowchart of a content push method provided in an embodiment of the present application;

[0038] Figure 6 A schematic diagram of a feature vector determination device provided in an embodiment of the present application;

[0039] Figure 7 A schematic diagram of a content push device provided in an embodiment of the present application;

[0040] Figure 8 A schematic diagram of the structure of the server provided in the embodiment of the present application;

[0041] Figure 9 A schematic diagram of the structure of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The embodiments of the present application are described below with reference to the accompanying drawings.

[0043] Regarding the problem of constructing a feature vector corresponding to a content based on multimodal vectors, the relevant technology generally performs a simple splicing (concat method) of the multimodal vectors. However, this simple splicing method results in less interaction between the multimodal vectors, which leads to poor accuracy, making the method of constructing a feature vector corresponding to a content based on multimodal vectors unreasonable.

[0044] Based on this, embodiments of the present application provide a content push method, a feature vector determination method, and related devices for improving the accuracy of feature vectors generated based on multimodal vectors.

[0045] The feature vector determination method provided in the embodiment of the present application is based on artificial intelligence. Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0046] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0047] In the embodiments of this application, the main artificial intelligence software technologies involved include the above-mentioned natural language processing, machine learning, etc. For example, it may involve image semantic understanding (ISU) in computer vision (CV), semantic understanding in natural language processing (NLP), deep learning in machine learning (ML), etc.

[0048] The feature vector determination method provided in this application can be applied to a feature vector determination device with data processing capabilities, and the content push method provided in this application can be applied to a content push device with data processing capabilities. The feature vector determination device or the content push device can be a terminal device or a server. Among them, the terminal device can specifically be a smart phone, a desktop computer, a laptop computer, a tablet computer, a smart speaker, a smart watch, etc., but is not limited to these; the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The terminal device and the server can be directly or indirectly connected through wired or wireless communication, and this application does not limit this.

[0049] This feature vector determines whether the device or content delivery device is capable of implementing computer vision technology. Computer vision is the study of how machines can "see." More specifically, it refers to the use of cameras and computers to replace the human eye in identifying, detecting, and measuring objects. Further image processing is performed to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems that can extract information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0050] The feature vector determines that the device or content push device may have the ability to implement natural language processing. Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can enable effective communication between humans and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology generally includes text processing, semantic understanding, machine translation, robot question and answer, knowledge graph and other technologies. In an embodiment of the present application, the text processing device can process the text through technologies such as text preprocessing and semantic understanding in natural language processing.

[0051] This feature vector determines whether the device or content push device possesses machine learning capabilities. Machine learning is a multidisciplinary discipline that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of artificial intelligence. Machine learning and deep learning typically include technologies such as artificial neural networks.

[0052] The artificial intelligence model used in the feature vector determination method provided in the embodiment of the present application mainly involves the application of machine learning. The first to-be-determined vector and the second to-be-determined vector are extracted through deep learning in machine learning, so as to adjust the second to-be-determined vector through the first activation vector determined by the first to-be-determined vector, thereby improving the accuracy of the generated feature vector without increasing the amount of calculation.

[0053] In order to facilitate understanding of the technical solution of the present application, the feature vector determination method provided in the embodiment of the present application is introduced below in combination with actual application scenarios.

[0054] See also Figure 1 , which is a schematic diagram of an application scenario of the feature vector determination method provided in the embodiment of the present application. Figure 1 In the application scenario shown, the aforementioned feature vector determination device is a server 100, which is used to generate a feature vector corresponding to the target content.

[0055] The server 100 obtains the target content to be processed, which includes first information from a first modality and second information from a second modality, i.e., the first information and the second information come from different aspects of the target content, and it is necessary to combine the first information and the second information to generate a feature vector corresponding to the target content. The target content to be processed is the content for which a feature vector needs to be generated, and the content can be, for example, video, audio, user relationships based on a knowledge graph, etc. Figure 1 In the application scenario shown, the target content is a video, the first information may be text information from a title description, and the second information may be image information from a video frame.

[0056] The server 100 generates a first undetermined vector corresponding to the first information and a second undetermined vector corresponding to the second information, so as to generate a feature vector of the target content based on the first undetermined vector and the second undetermined vector. Figure 1 In the scene shown, the first undetermined vector is a text undetermined vector, and the second undetermined vector is an image undetermined vector.

[0057] Information from different modalities is expressed in different ways, and different expressions differ in their characteristics and emphasis. The first pending vector and the second pending vector are derived from the vectors corresponding to information of different modalities of the target content. They focus on different aspects of the target content. For example, the text pending vector focuses on the title content of the video, and the image pending vector focuses on the video frame content.

[0058] In order to avoid the adverse effect of different expressions on the generated feature vectors due to the dispersion of expression emphasis, it is necessary to forward fuse the undetermined vectors from different modalities. Therefore, the emphasis of the first undetermined vector can be used to guide the adjustment of the second undetermined vector, so that the expression emphasis of the two gradually tends to be unified.

[0059] The server 100 determines the first activation vector based on the first pending vector. The first activation vector includes a first weight parameter determined by the first pending vector. The first weight parameter is determined by the criticality of the information in the first pending vector and can reflect the emphasis of the information in the first pending vector. When the second pending feature vector is adjusted with the first activation vector, the first weight parameter included therein can play a role in guiding the adjustment, so as to adjust the information included in the second pending feature vector based on the emphasis of the first pending feature vector, thereby enhancing the key information in the second pending vector and suppressing the non-critical information of the second pending vector, and obtaining the second modal vector. Figure 1 In the application scenario shown, a first activation vector can be determined from the text vector to be determined, and the image vector to be determined can be adjusted using the first activation vector to obtain a second modal vector.

[0060] The expression emphasis of the second modal vector obtained through adjustment and the first quantity to be oriented gradually tends to be unified, reducing the adverse effect of different expression methods on the dispersion of expression emphasis on the feature vector corresponding to the generated target content, so that the feature vector corresponding to the target content generated based on the two can more accurately represent the relevant information of the target content.

[0061] Therefore, in the process of generating the feature vector corresponding to the target content, a better fusion of the undetermined vectors of different modalities is achieved, and the key information related to the target content theme in the generated feature vector is strengthened, thereby improving the accuracy.

[0062] In the following, in conjunction with the accompanying drawings, a feature vector determination method provided by an embodiment of the present application is introduced with a server as a feature vector determination device.

[0063] See also Figure 2 , Figure 2 A flow chart of a method for determining a feature vector provided in an embodiment of the present application is shown as follows: Figure 2 As shown, the feature vector determination method includes the following steps:

[0064] S201: Acquire target content to be processed.

[0065] The target content is content that needs to be processed, such as classification and push. The content is the information or experience presented by the creator to the user. For example, the content can be video, audio, user relationships based on knowledge graphs, text, etc.

[0066] The target content typically contains information from different modalities, including first information from a first modality and second information from a second modality. The first and second modalities are different modalities, and information from different modalities is expressed in different ways, with different expressive characteristics and key points.

[0067] For example, if the target content is video content, the first information can be text from the title description, and the second modal information can be image information from the video frame. The text information focuses on the title description of the video content, used to attract users to click and watch, and is generally expressed through text. The image information focuses on the video frame displaying the video content, used to fully explain the video content, and is generally expressed through images. Obviously, the expression characteristics and emphasis of text information and image information from different modalities are different.

[0068] S202: Generate a first undetermined vector corresponding to the first information and a second undetermined vector corresponding to the second information.

[0069] The first information and the second information come from different modes of the target content. It is necessary to combine the first information and the second information to generate a feature vector of the target content. To this end, a first undetermined vector corresponding to the first information and a second undetermined vector corresponding to the second information can be generated, and a feature vector corresponding to the target content can be generated based on the first undetermined vector and the second undetermined vector.

[0070] This application does not specifically limit the method for generating the first undetermined vector and the second undetermined vector. For example, the first information can be input into a first feature module to generate a first undetermined vector corresponding to the first information, and the second information can be input into a second feature module to generate a second undetermined vector corresponding to the second information. The first feature module can be the same as or different from the second feature module and can be configured based on the requirements of the first and second information.

[0071] Taking video content as an example, text information can be input into a trained Bidirectional Encoder Representations from Transformers (BERT) model to generate a text vector corresponding to the text information. Image information can be input into a trained residual neural network (ResNet) to generate an image vector corresponding to the image information.

[0072] S203: Determine a first active vector according to the first undetermined vector.

[0073] Since the first information and the second information come from different modalities and are expressed in different ways, the information carried by the first undetermined vector and the second undetermined vector will also reflect the target content in different ways of expression.

[0074] In related technologies, a simple concatenation (concat method) of multimodal vectors is used to generate a feature vector corresponding to the target content based on the first undetermined vector and the second undetermined vector. Figure 3This figure shows a multimodal vector fusion method. The first information, "A quick and easy way to cook braised pork," is fed into the BERT model to obtain the first undetermined vector. The second information, a braised pork video frame, is fed into the ResNet model to obtain the second undetermined vector. The first and second undetermined vectors are then fused using the concat method, and the classification model is used to obtain the classification result.

[0075] However, the above method will result in less interaction between the first undetermined vector and the second undetermined vector, resulting in the generation of feature vectors corresponding to the target content based on undetermined vectors with different expressions, which will have the adverse effect of dispersing the expression focus. Therefore, it is necessary to perform forward fusion of undetermined vectors from different modalities.

[0076] Among them, forward fusion means that as the degree of fusion of the first to-be-determined vector and the second to-be-determined vector becomes higher, the accuracy of the feature vector corresponding to the target content generated thereby becomes higher. In other words, the higher degree of fusion of the first to-be-determined vector and the second to-be-determined vector has a positive impact on the feature vector, so that the information carried in the feature vector is more consistent with the key information related to the theme of the target content, that is, the information that is highly relevant to the central idea expressed by the target content.

[0077] Therefore, to improve the degree of fusion between the first undetermined vector and the second undetermined vector, the first undetermined vector and the second undetermined vector can be made to express the target content in the same way, thereby reducing the adverse effect of distracting the focus of expression on subsequently generated feature vectors. Based on the unified goal of making the feature vectors more accurately reflect the key information of the target content, the first undetermined vector and the second undetermined vector can be made to express the target content in the same way, and the second undetermined feature vector can be adjusted based on the first undetermined vector.

[0078] Specifically, the first activation vector is determined based on the first undetermined vector. The first activation vector includes a first weight parameter determined by the first undetermined vector. The first weight parameter is determined by the criticality of the information carried by the first undetermined vector, which can reflect the emphasis of the information in the first undetermined vector. In other words, under the first mode, the first undetermined vector reflects the expression focus of the target content theme.

[0079] For example, the more critical the information carried by the first undetermined vector, the more it reflects the theme of the target content, and the larger the value of the corresponding first weight parameter. Similarly, the less critical the information carried by the first undetermined vector, the smaller the value of the first weight parameter. Therefore, the first weight parameter in the first activation vector can be used to guide the adjustment of the second undetermined vector. That is, in the second mode, the second undetermined vector can be adjusted to reflect the focus of the target content theme.

[0080] S204: Adjust the second undetermined vector using the first activation vector to obtain a second modal vector.

[0081] From the above, it can be seen that the first weight parameter included in the first activation vector can reflect the emphasis of the information carried by the first undetermined vector. Based on the emphasis of the first undetermined vector, the information carried by the second undetermined vector can be adjusted in a targeted manner, that is, based on the first weight parameter, the key information in the second undetermined vector is enhanced, and the non-key information in the second undetermined vector is suppressed, thereby obtaining the second modal vector.

[0082] For example, a larger value for the first weight parameter indicates that the information carried by the first undetermined vector corresponding to the first weight parameter is more critical and more representative of the theme of the target content. Based on the emphasis of the first undetermined vector reflected by the first weight parameter, the information in the second undetermined vector is enhanced to obtain a second modal vector, such that the expression of the second modal vector is more similar to that of the first undetermined vector in reflecting the theme of the target content.

[0083] As a possible implementation, the first activation vector can be a gate function. The principle of the gate function is to output a threshold vector and adjust the corresponding output neuron state through the threshold vector. The lower the threshold vector value, the more severely suppressed the neuron is, and its influence will be weakened, thereby suppressing non-critical information in the second undetermined vector. Similarly, the higher the threshold vector value, the more strongly enhanced the neuron is, and its influence will be increased, thereby enhancing the critical information in the second undetermined vector.

[0084] The embodiment of the present application does not specifically limit the threshold function. For example, the first activation vector can be a sigmoid function (a logical function). The sigmoid function controls the threshold vector value in the interval [0,1]. If the threshold vector value is closer to 0, it means that the neuron is severely suppressed and the influence will be weakened, thereby achieving the suppression of non-critical information in the second undetermined vector. If the threshold vector value is closer to 1, the information represented by the neuron is more important. It can be understood that only suppressing the non-critical information of the second undetermined vector can also highlight the critical information of the second undetermined vector, which is equivalent to enhancing the critical information. For example, the first activation vector can also be a tanh function (a hyperbolic function).

[0085] Thus, although the first undetermined vector and the second undetermined vector have different expressions, the second undetermined vector can be adjusted using the first weight parameter included in the first activation vector, so that the adjusted second modal vector gradually converges with the expression focus of the first undetermined vector, better reflecting the theme of the target content. Moreover, since the second modal vector and the first undetermined vector do not have exactly the same expression focus, the two can also complement each other, so that the feature vector corresponding to the target content generated based on the two can more accurately represent the relevant information of the target content.

[0086] S205: Generate a feature vector corresponding to the target content according to the first undetermined vector and the second modal vector.

[0087] Generating the feature vector corresponding to the target content no longer involves a simple concatenation of the first and second undetermined vectors to achieve fusion. Instead, the second undetermined vector is adjusted by the first weight parameter included in the first activation vector determined based on the first undetermined vector, allowing the second undetermined vector to interact with the first undetermined vector during fusion, resulting in a second modal vector. Through this adjustment, the second modal vector and the first undetermined vector gradually converge in terms of their expression emphasis, reducing the adverse effects of the fragmented expression emphasis of the feature vector corresponding to the target content. This allows the feature vector corresponding to the target content generated from the two to more accurately represent the relevant information of the target content.

[0088] After obtaining the feature vector, the target content is processed accordingly, for example, the target content's content category is determined. The present embodiment does not specifically limit the method for classifying the target content. For example, the feature vector corresponding to the target content can be input into a pre-trained classification model to obtain the target content's content category, thereby enabling recommendations, searches, and the like for the target content.

[0089] For example, in a target content recommendation scenario, the user characteristics of the user to be recommended can be obtained. If the degree of matching between the feature vector of the target content and the user characteristics of the user to be recommended meets the first preset matching condition, it means that the target content is more likely to meet the needs of the target user. In this case, the target content will be recommended to the user to be recommended, thereby improving the user experience.

[0090] For another example, in the search scenario of the target content, the semantic features of the question to be answered can be obtained. If the degree of match between the feature vector of the target content and the semantic features of the question to be answered satisfies the second preset matching condition, it means that the target content is more likely to be the answer to the question to be answered, and the target content is included in the search results and recommended as an answer to the question to be answered, thereby improving the accuracy of the search results. It can be seen from the above technical solution that the target content for which the feature vector is to be generated has first information from the first modality and second information from the second modality, and it is necessary to generate the feature vector of the target content by integrating the first information and the second information. To this end, a first undetermined vector corresponding to the first information and a second undetermined vector corresponding to the second information are generated, and a feature vector of the target content is generated based on the first undetermined vector and the second undetermined vector. Since information from different modalities has an expression method corresponding to the modality in which it is located, different expressions will differ in expression characteristics and emphasis. In order to avoid the adverse effect of different expressions on the generated feature vectors due to the dispersion of expression emphasis, it is necessary to forward fuse the undetermined vectors from different modalities. A first activation vector can be determined based on the first undetermined vector. The first activation vector includes a first weight parameter determined by the first undetermined vector. The first weight parameter is determined by the criticality of the information in the first undetermined vector and can reflect the emphasis of the information in the first undetermined vector. Based on the emphasis of the first undetermined feature vector, the information included in the second undetermined feature vector can be adjusted in a targeted manner, thereby enhancing the key information in the second undetermined vector and suppressing non-key information. The second modal vector obtained through adjustment gradually converges with the expression emphasis of the first undetermined vector, achieving a better fusion of undetermined vectors of different modalities, strengthening the key information related to the target content theme in the generated feature vector, and improving accuracy.

[0091] As a possible implementation method, not only can the second undetermined vector be adjusted according to the first undetermined vector, but on this basis, the first undetermined vector can also be adjusted according to the second undetermined vector to achieve mutual adjustment between the first undetermined vector and the second undetermined vector.

[0092] Specifically, a second activation vector is determined based on the second pending vector. The second activation vector includes a second weight parameter determined by the second pending vector. The second weight parameter is determined by the criticality of the information carried by the second pending vector, which can reflect the emphasis of the information in the second pending vector. In other words, under the second mode, the second pending vector reflects the expression focus of the target content theme.

[0093] Based on the emphasis of the second undetermined vector, the information carried by the first undetermined vector can be adjusted in a targeted manner, that is, through the second weight parameter included in the second activation vector, the key information in the first undetermined vector is enhanced, and the non-key information in the first undetermined vector is suppressed to obtain the first modal vector.

[0094] Therefore, even if the first undetermined vector and the second undetermined vector have different expressions, the second undetermined vector can be adjusted according to the first weight parameter included in the first activation vector, and the first undetermined vector can be adjusted according to the second weight parameter included in the second activation function, so that the expression focus of the first modal vector and the second modal vector obtained after adjustment gradually tends to be unified, that is, each other's expression method reflects the expression focus of the target theme, so that the degree of fusion of the first modal vector and the second modal vector is higher, and it can better reflect the theme of the target content, and the feature vector corresponding to the target content is generated according to the first modal vector and the second modal vector, which is more accurate.

[0095] As a possible implementation manner, the second activation vector may also be a gate function to enhance key information and suppress non-key information in the second undetermined vector.

[0096] As a possible implementation manner, the first activation limit and the second activation vector may be the same threshold function, so that the feature vector generated based on the first modal vector and the second modal vector is more accurate.

[0097] Because the first information and the second information come from different modalities of the target content, the dimensions of the first undetermined vector and the second undetermined vector may be different. For example, the aforementioned text undetermined vector may be 768-dimensional, and the image undetermined vector may be 1024-dimensional. In related technologies, the first undetermined vector and the second undetermined vector can be fused through tensor multiplication. Although this can improve the degree of fusion between the first undetermined vector and the second undetermined vector, the fusion process introduces a three-dimensional vector W. The three-dimensional vector W contains too many parameters, which increases the computational complexity.

[0098] For example, if the dimension of the first undetermined vector V1 is M, and the dimension of the second undetermined vector V2 is N, a three-dimensional vector W is introduced by tensor multiplication, and the dimension of the three-dimensional vector W is R. M×T×N dimensional, then the first undetermined vector V1 and the second undetermined vector V2 are fused through the three-dimensional vector W to obtain a feature vector with a dimension of T, which can be expressed as: FusionV = V1 × W × V2. From the dimension of the introduced three-dimensional vector W, that is, R M×T×N It can be seen that the three-dimensional vector W contains too many parameters, which will increase the amount of calculation.

[0099] Based on this, the following introduces a method to reduce the amount of fusion calculation while improving the subsequent fusion level.

[0100] If the dimension of the first vector to be determined is the first dimension, and the dimension of the second vector to be determined is the second dimension, when determining the first activation vector based on the first vector to be determined, a first mapping matrix can be introduced. The dimension of the first activation vector can be adjusted through the first mapping matrix, and the first vector to be determined can be mapped from the first dimension to the second dimension to obtain the first activation vector with the dimension of the second dimension, so that the dimension of the first activation vector is equal to the dimension of the second vector to be determined.

[0101] For example, if the first dimension is M, the second dimension is N, the first mapping matrix is ​​W1, and the dimension of the first mapping matrix W1 is R M×N dimension, the first undetermined vector can be mapped from M dimension to N dimension to obtain the first activation vector with N dimension.

[0102] Similarly, when determining the second activation vector based on the second undetermined vector, a second mapping matrix can be introduced. The dimension of the second activation vector can be adjusted through the second mapping matrix, and the second undetermined vector can be mapped from the second dimension to the first dimension to obtain a second activation vector with the dimension of the first dimension, so that the dimension of the second activation vector is equal to the dimension of the first undetermined vector.

[0103] For example, if the first dimension is M, the second dimension is N, the second mapping matrix is ​​W2, and the dimension of W2 of the second mapping matrix is ​​R N×M dimension, the second undetermined vector can be mapped from N dimension to M dimension to obtain a second activation vector with M dimension.

[0104] It should be noted that the first dimension and the second dimension may be equal or unequal, and this application does not make any specific limitation on this.

[0105] Thus, by introducing the first mapping matrix and / or the second mapping matrix, the dimensions of the first activation vector and / or the second activation vector can be adjusted, thereby enabling better adjustment of the second undetermined vector and / or the first undetermined vector, improving the subsequent degree of fusion, and thus obtaining a more accurate feature vector. Furthermore, both the first mapping matrix and the second mapping matrix are two-dimensional vectors. Compared to three-dimensional vectors, achieving fusion through two-dimensional vectors can reduce the amount of computation, especially when the dimensional difference between the first undetermined vector and the second undetermined vector is large, which can greatly reduce the amount of computation and save computational costs.

[0106] As a possible implementation, if the target content is used for content classification, the aforementioned first feature module, second feature module, and first mapping matrix can constitute a vector generation model. The following describes a training method for the vector generation model, which includes the following steps:

[0107] S1: Obtain training samples associated with the target content.

[0108] The training samples are associated with the target content. For example, if the target content is a video, then the training samples are also videos. Similar to the target content, the training samples also contain information from different modalities, including information from the first modality and information from the second modality. Furthermore, the training samples have sample labels that identify the category of the content they belong to. For example, the training samples may be labeled as belonging to categories such as gaming videos and food videos.

[0109] S2: Determine the first undetermined sample vector and the second undetermined sample vector of the training sample through the initial vector generation model.

[0110] The initial vector generation model includes a first feature module, a second feature module and a first mapping matrix. The first feature module can be used to determine the first undetermined sample vector of the training sample, and the second feature module can be used to determine the second undetermined sample vector of the training sample.

[0111] S3: Determine a first pending activation vector according to the first mapping matrix and the first pending sample vector in the initial vector generation model, and obtain a second modal sample vector based on the first pending activation vector and the second pending sample vector.

[0112] The first pending sample vector is mapped from the current dimension to the dimension of the second pending sample vector through the first mapping matrix to obtain a first pending activation vector, where the dimension of the first pending activation vector is the dimension of the second pending sample vector.

[0113] The second undetermined sample vector is adjusted by the first undetermined activation vector, as shown in S204 for details, to obtain a second modal sample vector.

[0114] S4: Using the first undetermined sample vector and the second modal sample vector as sample vectors of training samples, and obtaining corresponding classification results through a classifier.

[0115] A sample vector corresponding to the training sample is generated according to the first undetermined sample vector and the second modal sample vector, and the sample vector is input into the classifier to obtain a classification result corresponding to the training sample.

[0116] The embodiment of the present application does not specifically limit the classifier, and it can be, for example, a linear classifier (linear regression), a logistic regression classifier (logistic regression), a support vector machine (SVM), etc.

[0117] S5: According to the difference between the sample label and the classification result, the parameters of the first feature module and the first mapping matrix are adjusted to obtain a vector generation model.

[0118] The difference between the sample label and the classification result is determined. Based on the difference, the relevant parameters of the first feature module and the first mapping matrix can be adjusted so that the classification result output by the initial vector generation model is close to the sample label. When the training of the initial vector generation model is completed, that is, the relevant parameters in the initial vector generation model are adjusted, the initial vector generation model with the relevant parameters adjusted can be a vector generation model.

[0119] As a possible implementation, the vector generation model may include not only the first feature module, the second feature module, and the first mapping matrix, but also the second mapping matrix. The specific training process is as follows:

[0120] S11: Acquire training samples associated with the target content.

[0121] S22: Determine a first pending sample vector and a second pending sample vector of the training sample through the initial vector generation model.

[0122] S33: Determine a first pending activation vector based on the first mapping matrix and the first pending sample vector in the initial vector generation model, and obtain a second modal sample vector based on the first pending activation vector and the second pending sample vector. Determine a second pending activation vector based on the second mapping matrix and the second pending sample vector in the initial vector generation model, and obtain a first modal sample vector based on the second pending activation vector and the first pending sample vector.

[0123] S44: Using the first modality sample vector and the second modality sample vector as sample vectors of training samples, and obtaining corresponding classification results through a classifier.

[0124] S55: According to the difference between the sample label and the classification result, the parameters of the first feature module, the first mapping matrix and the second mapping matrix are adjusted to obtain a vector generation model.

[0125] The relevant details can be found in S1-S5 and will not be elaborated here.

[0126] Next, we will combine Figure 4 , the feature vector determination method provided by the embodiment of the present application is described by taking the target content to be processed as the target video as an example. Figure 4 , which is a schematic diagram of an application scenario of a feature vector determination method provided in an embodiment of the present application.

[0127] exist Figure 4 In the application scenario shown, the vector generation model is generated through the aforementioned S11-S55 training, including a first feature module, a second feature module, a first mapping matrix, and a second mapping matrix. The first feature module is a BERT model, the second feature module is a ResNet model, the first mapping matrix is ​​ω1, and the second mapping matrix is ​​ω2.

[0128] The target video includes text information from the title description "Home-cooked braised pork recipe that can be learned in one go" and image information from the video frame. The image information can be extracted through key frames to obtain one of the key frames.

[0129] The text "Homemade Braised Pork Recipe You Can Learn in One Go" is fed into the BERT model to obtain the text vector x1. The extracted keyframes are fed into the ResNet model to obtain the image vector x2. The text vector x1 has 4 dimensions, and the image vector x2 has 3 dimensions.

[0130] According to the first mapping matrix ω1, the text vector x1 is mapped from 4 dimensions to 3 dimensions to obtain the first activation vector with a dimension of 3 dimensions. Among them, b1 is an adjustable parameter used to adjust the first activation vector. Similarly, according to the second mapping matrix ω2, the image vector x2 is mapped from 3D to 4D to obtain the second activation vector with a dimension of 4D Wherein, b2 is an adjustable parameter used to adjust the second activation vector.

[0131] The first activation vector obtained Multiply the image vector x2 to get the second modal vector, which is the new expression of the image side. Multiply it with the text vector x1 to get the first modal vector, which is the new expression on the text side.

[0132] The first modal vector and the second modal vector are concatenated to generate a feature vector corresponding to the target video. This feature vector is then fed into the SVM model to classify the target video as a food video.

[0133] Therefore, the interaction between the modal features in the model is strengthened through the first activation function and the second activation function, that is, the image pending vector corresponding to the image side is strengthened and suppressed by the text pending vector corresponding to the text side, and the image pending vector is strengthened and suppressed by the text pending vector corresponding to the text side, so as to achieve the purpose of searching for text from image and searching for image from text, that is, to suppress the vector feature dimensions that are irrelevant to the subject content of the target video and strengthen the feature information that is more relevant to the subject content of the target video.

[0134] At the same time, the first mapping matrix and the second mapping matrix are both two-dimensional vectors. Compared with the three-dimensional vectors introduced in tensor multiplication, fusion through two-dimensional vectors can reduce the amount of calculation, especially when the dimension gap between the text vector to be determined and the image vector to be determined is large, which can greatly reduce the amount of calculation and save computing costs.

[0135] In addition to the feature vector push method provided in the above embodiment, the embodiment of the present application also provides a content push method. Figure 5 , using the server as the content push device, a content push method provided in an embodiment of the present application is introduced.

[0136] See also Figure 5 , which is a flow chart of a content push method provided by an embodiment of the present application. Figure 5 As shown, the content push method includes the following steps:

[0137] S501: Determine the object features of the target object.

[0138] The target object is the person to whom the target content is pushed. This can be a target user, a target keyword, or other similar object. To push appropriate content to the target object, you first determine the target object's vectorized representation, namely, its object features. These features can include user features corresponding to the target user, word features corresponding to the target keyword, and other similar features. These features can help you identify the target object's needs.

[0139] S502: Determine target content associated with the object feature from the content to be determined according to the feature vector of the content to be determined.

[0140] After clarifying the needs of the target audience, content that meets these needs can be selected from the pending content as the target content. To improve the accuracy of determining the target content, a feature vector of the pending content can be determined. The target content can be determined by determining the degree of correlation between the feature vector and the object's characteristics. For example, the higher the degree of correlation between the feature vector and the object's characteristics, the more the pending content corresponding to the feature vector meets the needs of the target audience, and the pending content can be selected as the target content.

[0141] The method of determining the feature vector of the content to be determined can refer to the feature vector determination method described in S201-S205 above, and will not be repeated here.

[0142] S503: Return target content based on the target object.

[0143] The target content is more likely to meet the needs of the target object, so the target content can be returned to the target object to ensure the accuracy of the push of the target content.

[0144] For example, if the target object is a target user and the object feature is a user feature corresponding to the target user, target content can be pushed to the target user, so that the target content better meets the needs of the target user and improves the target user's experience.

[0145] For another example, if the target object is a target keyword, and the object feature is a word feature corresponding to the target keyword, the corresponding search result is returned based on the target keyword, and the search result includes the target content, so that the target content is more matched with the target keyword, thereby improving the accuracy of the search result. The content push method provided in the embodiment of the present application determines the object feature of the target object, and determines the target content from the pending content based on the association between the feature vector of the pending content and the object feature. The target content to be generated with the feature vector has the first information from the first modality and the second information from the second modality, and it is necessary to generate the feature vector of the target content by synthesizing the first information and the second information. For this purpose, a first pending vector corresponding to the first information and a second pending vector corresponding to the second information are generated, and a feature vector of the target content is generated based on the first pending vector and the second pending vector. Since information from different modalities has an expression method corresponding to the modality in which it is located, different expressions will differ in expression characteristics and emphasis. In order to avoid the adverse effect of different expressions on the generated feature vectors due to the dispersion of expression emphasis, it is necessary to forward fuse the pending vectors from different modalities. The first activation vector can be determined based on the first pending vector. The first activation vector includes a first weight parameter determined by the first pending vector. The first weight parameter is determined by the criticality of the information in the first pending vector, which can reflect the focus of the information in the first pending vector. The information included in the second pending feature vector can be adjusted in a targeted manner based on the focus of the first pending feature vector, thereby enhancing the key information in the second pending vector and suppressing non-key information. The second modal vector obtained by adjustment gradually converges with the expression focus of the first pending vector, achieving a better fusion of the pending vectors of different modalities, strengthening the key information related to the target content theme in the generated feature vector, and improving the accuracy of the feature vector. Therefore, the target content corresponding to the feature vector is more likely to meet the needs of the target object, and returning the target content to the target object can ensure the accuracy of the push of the target content.

[0146] In response to the feature vector determination method provided in the above embodiment, an embodiment of the present application also provides a feature vector determination device.

[0147] See also Figure 6 , which is a schematic diagram of a feature vector determination device provided by an embodiment of the present application. Figure 6 As shown, the feature vector determination device 600 includes: an acquisition unit 601, a first generation unit 602, a first determination unit 603, an adjustment unit 604 and a second generation unit 605;

[0148] The acquisition unit 601 is configured to acquire target content to be processed, where the target content includes first information from a first modality and second information from a second modality;

[0149] The first generating unit 602 is configured to generate a first undetermined vector corresponding to the first information and a second undetermined vector corresponding to the second information;

[0150] The first determining unit 603 is configured to determine a first activation vector according to the first undetermined vector, where the first activation vector includes a first weight parameter determined by the first undetermined vector;

[0151] The adjusting unit 604 is configured to adjust the second undetermined vector using the first activation vector to obtain a second modal vector, wherein the first weight parameter is used to enhance key information in the second undetermined vector and suppress non-key information in the second undetermined vector;

[0152] The second generating unit 605 is configured to generate a feature vector corresponding to the target content according to the first undetermined vector and the second modal vector.

[0153] As a possible implementation manner, the first determining unit 603 is further configured to:

[0154] determining a second activation vector according to the second undetermined vector, wherein the second activation vector includes a second weight parameter determined by the second undetermined vector;

[0155] The second generating unit is configured to:

[0156] Adjusting the first undetermined vector by using the second activation vector to obtain a first modal vector, wherein the second weight parameter is used to enhance key information in the first undetermined vector and suppress non-key information in the first undetermined vector;

[0157] A feature vector corresponding to the target content is generated according to the first modal vector and the second modal vector.

[0158] As a possible implementation manner, the dimension of the first quantity to be oriented is a first dimension, the dimension of the second quantity to be oriented is a second dimension, and the first determining unit 603 is configured to:

[0159] According to a first mapping matrix, the first undetermined vector is mapped from the first dimension to the second dimension to obtain the first activation vector with the second dimension.

[0160] As a possible implementation manner, the dimension of the first quantity to be oriented is a first dimension, the dimension of the second quantity to be oriented is a second dimension, and the first determining unit 603 is configured to:

[0161] According to the second mapping matrix, the second undetermined vector is mapped from the second dimension to the first dimension to obtain the second activation vector with the first dimension.

[0162] As a possible implementation, the vector generation model includes a first feature module, a second feature module, and a first mapping matrix; the first generation unit 602 is configured to:

[0163] Generate a first undetermined vector corresponding to the first information through the first feature module, and generate a second undetermined vector corresponding to the second information through the second feature module;

[0164] The apparatus further comprises a training unit, configured to:

[0165] Acquire a training sample associated with the target content, where the training sample includes information from the first modality and information from the second modality, and has a sample label for identifying a content category to which the training sample belongs;

[0166] Determining a first undetermined sample vector and a second undetermined sample vector of the training sample by using an initial vector generation model;

[0167] Determining a first pending activation vector according to a first mapping matrix in the initial vector generation model and the first pending sample vector, and obtaining a second modality sample vector based on the first pending activation vector and the second pending sample vector;

[0168] Using the first undetermined sample vector and the second modality sample vector as sample vectors of the training sample, and obtaining corresponding classification results through a classifier;

[0169] According to the difference between the sample label and the classification result, parameters of the first feature module and the first mapping matrix are adjusted to obtain the vector generation model.

[0170] As a possible implementation manner, the apparatus further includes a second determining unit, configured to:

[0171] The content category of the target content is determined according to the feature vector.

[0172] As a possible implementation, the target content is video content, and the video content includes text information from a title description and image information from a video frame. The first generating unit 602 is configured to:

[0173] A text vector to be determined corresponding to the text information and an image vector to be determined corresponding to the image information are generated.

[0174] An embodiment of the present application provides a feature vector determination device, and the target content of the feature vector to be generated has first information from a first modality and second information from a second modality, and it is necessary to generate a feature vector of the target content by integrating the first information and the second information. To this end, a first undetermined vector corresponding to the first information and a second undetermined vector corresponding to the second information are generated, and a feature vector of the target content is generated based on the first undetermined vector and the second undetermined vector. Since information from different modalities has an expression method corresponding to the modality in which it is located, different expressions will differ in expression characteristics and emphasis. The information carried in the first undetermined vector and the second undetermined vector generated by the first information and the second information will also reflect the above-mentioned expression method. In order to avoid the adverse effect of different expressions on the generated feature vectors due to the dispersion of expression emphasis, it is necessary to forward fuse the undetermined vectors from different modalities. The first activation vector can be determined based on the first undetermined vector. Among them, the first activation vector includes a first weight parameter determined by the first undetermined vector, and the first weight parameter is determined by the criticality of the information in the first undetermined vector, which can reflect the emphasis of the information in the first undetermined vector. When the second undetermined feature vector is adjusted by the first activation vector, the first weight parameter included therein can play a role in guiding the adjustment, so as to adjust the information included in the second undetermined feature vector based on the emphasis of the first undetermined feature vector, thereby enhancing the key information in the second undetermined vector and suppressing non-key information. The expression emphasis of the second modal vector obtained by adjustment gradually tends to be unified with that of the first undetermined vector, so that the feature vector corresponding to the target content generated based on the two can more accurately represent the relevant information of the target content. Therefore, in the process of generating the feature vector corresponding to the target content, a better fusion of the undetermined vectors of different modes is achieved, and the key information related to the target content theme in the generated feature vector is strengthened, thereby improving accuracy.

[0175] In response to the content push method provided in the above embodiment, an embodiment of the present application also provides a content push device.

[0176] See also Figure 7 , which is a schematic diagram of a content push device provided by an embodiment of the present application. Figure 7 As shown, the content push device 700 includes: a determining unit 701 and a returning unit 702;

[0177] The determining unit 701 is configured to determine an object feature of a target object; determine target content associated with the object feature from the undetermined content based on a feature vector of the undetermined content; the target content includes first information from a first modality and second information from a second modality; the feature vector of the target content is generated based on a first undetermined vector and a second modal vector; the first undetermined vector corresponds to the first information; the second undetermined vector corresponds to the second information; the second modal vector is obtained by adjusting the second undetermined vector using a first activation vector; the first activation vector includes a first weight parameter determined based on the first undetermined vector; the first weight parameter is used to enhance key information in the second undetermined vector and suppress non-key information in the second undetermined vector;

[0178] The returning unit 702 is configured to return the target content based on the target object.

[0179] As a possible implementation manner, if the target object is a target user and the object feature is a user feature corresponding to the target user, the returning unit 702 is configured to:

[0180] Pushing the target content to the target user;

[0181] If the target object is a target keyword and the object feature is a word feature corresponding to the target keyword, the returning unit 702 is configured to:

[0182] The corresponding search results are returned based on the target keyword, and the search results include the target content.

[0183] The content push device provided in the embodiment of the present application determines the object characteristics of the target object, and determines the target content from the pending content based on the association between the feature vector of the pending content and the object characteristics. The target content of the to-be-determined feature vector has first information from the first modality and second information from the second modality, and it is necessary to generate the feature vector of the target content by integrating the first information and the second information. To this end, a first to-be-determined vector corresponding to the first information and a second to-be-determined vector corresponding to the second information are generated, and a feature vector of the target content is generated based on the first to-be-determined vector and the second to-be-determined vector. Since information from different modalities has an expression method corresponding to the modality in which it is located, different expressions will differ in expression characteristics and emphasis. In order to avoid the adverse effect of different expressions on the generated feature vectors due to the dispersion of expression emphasis, it is necessary to forward fuse the to-be-determined vectors from different modalities. The first activation vector can be determined based on the first pending vector. The first activation vector includes a first weight parameter determined by the first pending vector. The first weight parameter is determined by the criticality of the information in the first pending vector, which can reflect the focus of the information in the first pending vector. The information included in the second pending feature vector can be adjusted in a targeted manner based on the focus of the first pending feature vector, thereby enhancing the key information in the second pending vector and suppressing non-key information. The second modal vector obtained by adjustment gradually converges with the expression focus of the first pending vector, achieving a better fusion of the pending vectors of different modalities, strengthening the key information related to the target content theme in the generated feature vector, and improving the accuracy of the feature vector. Therefore, the target content corresponding to the feature vector is more likely to meet the needs of the target object, and returning the target content to the target object can ensure the accuracy of the push of the target content.

[0184] The aforementioned feature vector determination device and content push device can also be a computer device, which can be a server or a terminal device. The following will introduce the computer device provided in the embodiment of the present application from the perspective of hardware entity. Figure 8 The following is a schematic diagram of the server structure. Figure 9 Shown is a schematic diagram of the structure of the terminal equipment.

[0185] See also Figure 8 , Figure 8This is a schematic diagram of a server structure provided by an embodiment of the present application. The server 1400 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1422 (for example, one or more processors) and memory 1432, and one or more storage media 1430 (for example, one or more mass storage devices) for storing application programs 1442 or data 1444. Among them, the memory 1432 and the storage medium 1430 can be short-term storage or persistent storage. The program stored in the storage medium 1430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1422 can be configured to communicate with the storage medium 1430 to execute a series of instruction operations in the storage medium 1430 on the server 1400.

[0186] The server 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input and output interfaces 1458, and / or one or more operating systems 1441, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0187] The steps performed by the server in the above embodiment can be based on the Figure 8 The server structure shown.

[0188] The CPU 1422 is configured to execute the following steps:

[0189] Acquire target content to be processed, where the target content includes first information from a first modality and second information from a second modality;

[0190] Generate a first undetermined vector corresponding to the first information and a second undetermined vector corresponding to the second information;

[0191] determining a first activation vector according to the first undetermined vector, wherein the first activation vector includes a first weight parameter determined by the first undetermined vector;

[0192] Adjusting the second undetermined vector by using the first activation vector to obtain a second modal vector, wherein the first weight parameter is used to enhance key information in the second undetermined vector and suppress non-key information in the second undetermined vector;

[0193] A feature vector corresponding to the target content is generated according to the first undetermined vector and the second modal vector.

[0194] Or perform the following steps:

[0195] Determine the object characteristics of the target object;

[0196] Determining target content associated with the object feature from the pending content based on a feature vector of the pending content; the target content includes first information from a first modality and second information from a second modality; the feature vector of the target content is generated based on the first pending vector and a second modality vector; the first pending vector corresponds to the first information; the second pending vector corresponds to the second information; the second modality vector is obtained by adjusting the second pending vector using a first activation vector; the first activation vector includes a first weight parameter determined based on the first pending vector; the first weight parameter is used to enhance key information in the second pending vector and suppress non-key information in the second pending vector;

[0197] The target content is returned based on the target object.

[0198] Optionally, the CPU 1422 may also execute the method steps of any specific implementation of the feature vector determination method or the content push method in the embodiments of the present application.

[0199] See also Figure 9 , Figure 9 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application. Figure 9 The block diagram shows a partial structure of a smartphone related to the terminal device provided in an embodiment of the present application. The smartphone includes: a radio frequency (RF) circuit 1510, a memory 1520, an input unit 1530, a display unit 1540, a sensor 1550, an audio circuit 1560, a wireless fidelity (WiFi) module 1570, a processor 1580, and a power supply 1590. Those skilled in the art will understand that Figure 9 The structure of the smartphone shown in the figure does not constitute a limitation on the smartphone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0200] The following combination Figure 9 A detailed introduction to the various components of a smartphone:

[0201] The RF circuit 1510 can be used to receive and send signals during information transmission or calls. In particular, after receiving downlink information from the base station, it is sent to the processor 1580 for processing. In addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1510 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 1510 can also communicate with the network and other devices via wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0202] The memory 1520 can be used to store software programs and modules. The processor 1580 implements various functional applications and data processing of the smartphone by running the software programs and modules stored in the memory 1520. The memory 1520 may mainly include a program storage area and a data storage area. The program storage area may store an operating system, at least one application required for a function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created based on the use of the smartphone (such as audio data, a phone book, etc.). In addition, the memory 1520 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0203] The input unit 1530 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the smartphone. Specifically, the input unit 1530 may include a touch panel 1531 and other input devices 1532. The touch panel 1531, also known as a touch screen, can collect user touch operations on or near it (such as operations performed by the user using a finger, stylus, or any other suitable object or accessory on or near the touch panel 1531) and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1531 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch direction and detects the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device and converts it into touch point coordinates, which are then sent to the processor 1580. It can also receive commands sent by the processor 1580 and execute them. In addition, the touch panel 1531 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1531, the input unit 1530 may further include other input devices 1532. Specifically, the other input devices 1532 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, power keys, etc.), a trackball, a mouse, and a joystick.

[0204] The display unit 1540 can be used to display information input by the user or information provided to the user and various menus of the smartphone. The display unit 1540 may include a display panel 1541. Optionally, the display panel 1541 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 1531 may cover the display panel 1541. When the touch panel 1531 detects a touch operation on or near it, it is transmitted to the processor 1580 to determine the type of touch event. Subsequently, the processor 1580 provides corresponding visual output on the display panel 1541 according to the type of touch event. Although in Figure 9 In the embodiment, the touch panel 1531 and the display panel 1541 are used as two independent components to realize the input and output functions of the smartphone, but in some embodiments, the touch panel 1531 and the display panel 1541 can be integrated to realize the input and output functions of the smartphone.

[0205] The smartphone may also include at least one sensor 1550, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1541 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1541 and / or the backlight when the smartphone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the smartphone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the smartphone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be described here.

[0206] Audio circuit 1560, speaker 1561, and microphone 1562 provide an audio interface between the user and the smartphone. Audio circuit 1560 converts received audio data into electrical signals and transmits them to speaker 1561, which then converts them into sound signals for output. Microphone 1562, on the other hand, converts collected sound signals into electrical signals, which are then received by audio circuit 1560 and converted into audio data. The audio data is then processed by processor 1580 and transmitted to, for example, another smartphone via RF circuit 1510, or stored in memory 1520 for further processing.

[0207] WiFi is a short-range wireless transmission technology. Smartphones can help users send and receive emails, browse the web, and access streaming media through the WiFi module 1570. It provides users with wireless broadband Internet access. Figure 9 A WiFi module 1570 is shown, but it is understandable that it is not an essential component of the smartphone and can be omitted as needed without changing the essence of the invention.

[0208] Processor 1580 is the control center of the smartphone, connecting all components of the smartphone using various interfaces and circuits. It executes software programs and / or modules stored in memory 1520 and accesses data stored in memory 1520 to perform various smartphone functions and process data. Optionally, processor 1580 may include one or more processing units. Preferably, processor 1580 integrates an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 1580.

[0209] The smartphone also includes a power supply 1590 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1580 through a power management system, thereby managing functions such as charging, discharging, and power consumption through the power management system.

[0210] Although not shown, the smartphone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0211] In the embodiment of the present application, the memory 1520 included in the smart phone can store program code and transmit the program code to the processor.

[0212] The processor 1580 included in the smart phone can execute the feature vector determination method or content push method provided in the above embodiment according to the instructions in the program code.

[0213] An embodiment of the present application further provides a computer-readable storage medium for storing a computer program, wherein the computer program is used to execute the feature vector determination method or content push method provided in the above embodiment.

[0214] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the feature vector determination method or content push method provided in various optional implementations of the above aspects.

[0215] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the above-mentioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the above-mentioned storage medium can be at least one of the following media: read-only memory (English: read-only memory, abbreviated: ROM), RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0216] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0217] The above is merely one specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A content push method, characterized in that: The method comprises: Determine the object characteristics of the target object; Determining target content associated with the object feature from the undetermined content based on a degree of association between a feature vector of the undetermined content and the object feature; the target content includes first information from a first modality and second information from a second modality, the feature vector of the target content is generated based on the first modality vector and the second modality vector, the first undetermined vector corresponds to the first information, the second undetermined vector corresponds to the second information, the second modality vector is obtained by adjusting the second undetermined vector using a first activation vector, the first activation vector includes a first weight parameter determined by the criticality of information carried by the first undetermined vector, the first weight parameter is used to specifically adjust the information included in the second undetermined vector based on the emphasis of the first undetermined quantity to enhance key information in the second undetermined vector and suppress non-critical information in the second undetermined vector, the first modality vector is obtained by adjusting the first undetermined vector using the second activation vector, the second activation vector includes a second weight parameter determined by the criticality of information carried by the second undetermined vector, the second weight parameter is used to specifically adjust the information carried by the first undetermined vector based on the emphasis of the second undetermined quantity to enhance key information in the first undetermined vector and suppress non-critical information in the first undetermined vector; The target content is returned based on the target object.

2. The method according to claim 1, characterized in that If the target object is a target user, and the object feature is a user feature corresponding to the target user, returning the target content based on the target object includes: Pushing the target content to the target user; If the target object is a target keyword, and the object feature is a word feature corresponding to the target keyword, returning the target content based on the target object includes: The corresponding search results are returned based on the target keyword, and the search results include the target content.

3. A method for determining a feature vector, characterized in that: The method comprises: Acquire target content to be processed, where the target content includes first information from a first modality and second information from a second modality; Generate a first undetermined vector corresponding to the first information and a second undetermined vector corresponding to the second information; determining a first activation vector according to the first undetermined vector, wherein the first activation vector includes a first weight parameter determined by a criticality of information carried by the first undetermined vector; The second undetermined vector is adjusted by the first activation vector to obtain a second modal vector, wherein the first weight parameter is used to specifically adjust the information included in the second undetermined vector based on the emphasis of the first undetermined vector to enhance key information in the second undetermined vector and suppress non-key information in the second undetermined vector; determining a second activation vector according to the second undetermined vector, wherein the second activation vector includes a second weight parameter determined by a criticality of information carried by the second undetermined vector; The first undetermined vector is adjusted by the second activation vector to obtain a first modal vector, wherein the second weight parameter is used to specifically adjust the information included in the first undetermined vector based on the emphasis of the second undetermined vector to enhance key information in the first undetermined vector and suppress non-key information in the first undetermined vector; A feature vector corresponding to the target content is generated according to the first modal vector and the second modal vector.

4. The method according to claim 3, characterized in that The dimension of the first undetermined vector is a first dimension, the dimension of the second undetermined vector is a second dimension, and determining the first activation vector according to the first undetermined vector includes: According to a first mapping matrix, the first undetermined vector is mapped from the first dimension to the second dimension to obtain the first activation vector with the second dimension.

5. The method according to claim 3, characterized in that The dimension of the first undetermined vector is a first dimension, the dimension of the second undetermined vector is a second dimension, and determining the second activation vector according to the second undetermined vector includes: According to the second mapping matrix, the second undetermined vector is mapped from the second dimension to the first dimension to obtain the second activation vector with the first dimension.

6. The method according to any one of claims 3 to 5, characterized in that: The vector generation model includes a first feature module, a second feature module, and a first mapping matrix; generating a first undetermined vector corresponding to the first information and a second undetermined vector corresponding to the second information includes: Generate a first undetermined vector corresponding to the first information through the first feature module, and generate a second undetermined vector corresponding to the second information through the second feature module; The vector generation model is trained as follows: Acquire a training sample associated with the target content, where the training sample includes information from the first modality and information from the second modality, and has a sample label for identifying a content category to which the training sample belongs; Determining a first undetermined sample vector and a second undetermined sample vector of the training sample by using an initial vector generation model; Determining a first pending activation vector according to a first mapping matrix in the initial vector generation model and the first pending sample vector, and obtaining a second modality sample vector based on the first pending activation vector and the second pending sample vector; Using the first undetermined sample vector and the second modality sample vector as sample vectors of the training sample, and obtaining corresponding classification results through a classifier; According to the difference between the sample label and the classification result, parameters of the first feature module and the first mapping matrix are adjusted to obtain the vector generation model.

7. The method according to any one of claims 3 to 5, characterized in that: The method further comprises: The content category of the target content is determined according to the feature vector.

8. The method according to claim 3, characterized in that The target content is video content, and the video content includes text information from a title description and image information from a video frame. Generating a first undetermined vector corresponding to the first information and a second undetermined vector corresponding to the second information includes: A text vector to be determined corresponding to the text information and an image vector to be determined corresponding to the image information are generated.

9. A content push device, characterized in that: The device comprises: a determination unit and a return unit; The determining unit is configured to determine an object feature of a target object; determine target content associated with the object feature from the pending content based on a degree of association between a feature vector of the pending content and the object feature; the target content includes first information from a first modality and second information from a second modality, the feature vector of the target content is generated based on a first modality vector and a second modality vector, the first pending vector corresponds to the first information, the second pending vector corresponds to the second information, the second modality vector is obtained by adjusting the second pending vector by a first activation vector, and the first activation vector includes a first weight parameter determined by a criticality of information carried by the first pending vector. The first weight parameter is used to adjust the information included in the second undetermined vector based on the emphasis of the first undetermined quantity to enhance the key information in the second undetermined vector and suppress the non-key information in the second undetermined vector. The first modal vector is obtained by adjusting the first undetermined vector by the second activation vector. The second activation vector includes a second weight parameter determined by the criticality of the information carried by the second undetermined vector. The second weight parameter is used to adjust the information carried by the first undetermined vector based on the emphasis of the second undetermined quantity to enhance the key information in the first undetermined vector and suppress the non-key information in the first undetermined vector. The returning unit is configured to return the target content based on the target object.

10. The device according to claim 9, characterized in that If the target object is a target user and the object feature is a user feature corresponding to the target user, the returning unit is configured to: Pushing the target content to the target user; If the target object is a target keyword and the object feature is a word feature corresponding to the target keyword, the returning unit is configured to: The corresponding search results are returned based on the target keyword, and the search results include the target content.

11. A device for determining a feature vector, characterized in that: The device includes: an acquisition unit, a first generation unit, a first determination unit, an adjustment unit, and a second generation unit; The acquisition unit is configured to acquire target content to be processed, wherein the target content includes first information from a first modality and second information from a second modality; The first generating unit is configured to generate a first undetermined vector corresponding to the first information and a second undetermined vector corresponding to the second information; The first determining unit is configured to determine a first activation vector according to the first undetermined vector, where the first activation vector includes a first weight parameter determined by a criticality of information carried by the first undetermined vector; The adjustment unit is configured to adjust the second undetermined vector by using the first activation vector to obtain a second modal vector, wherein the first weight parameter is used to specifically adjust information included in the second undetermined vector based on the emphasis of the first undetermined vector to enhance key information in the second undetermined vector and suppress non-key information in the second undetermined vector; The first determining unit is further configured to determine a second activation vector based on the second undetermined vector, where the second activation vector includes a second weight parameter determined by a criticality of information carried by the second undetermined vector; The second generating unit is configured to adjust the first undetermined vector by using the second activation vector to obtain a first modal vector, wherein the second weight parameter is used to specifically adjust information included in the first undetermined vector based on the emphasis of the second undetermined vector to enhance key information in the first undetermined vector and suppress non-key information in the first undetermined vector; The second generating unit is configured to generate a feature vector corresponding to the target content according to the first modal vector and the second modal vector.

12. The device according to claim 11, characterized in that The dimension of the first quantity to be oriented is a first dimension, the dimension of the second quantity to be oriented is a second dimension, and the first determining unit is configured to: According to a first mapping matrix, the first undetermined vector is mapped from the first dimension to the second dimension to obtain the first activation vector with the second dimension.

13. The device according to claim 11, characterized in that The dimension of the first quantity to be oriented is a first dimension, the dimension of the second quantity to be oriented is a second dimension, and the first determining unit is configured to: According to the second mapping matrix, the second undetermined vector is mapped from the second dimension to the first dimension to obtain the second activation vector with the first dimension.

14. The device according to any one of claims 11 to 13, characterized in that The vector generation model includes a first feature module, a second feature module and a first mapping matrix; the first generation unit is used to: Generate a first undetermined vector corresponding to the first information through the first feature module, and generate a second undetermined vector corresponding to the second information through the second feature module; The apparatus further comprises a training unit, configured to: Acquire a training sample associated with the target content, where the training sample includes information from the first modality and information from the second modality, and has a sample label for identifying a content category to which the training sample belongs; Determining a first undetermined sample vector and a second undetermined sample vector of the training sample by using an initial vector generation model; Determining a first pending activation vector according to a first mapping matrix in the initial vector generation model and the first pending sample vector, and obtaining a second modality sample vector based on the first pending activation vector and the second pending sample vector; Using the first undetermined sample vector and the second modality sample vector as sample vectors of the training sample, and obtaining corresponding classification results through a classifier; According to the difference between the sample label and the classification result, parameters of the first feature module and the first mapping matrix are adjusted to obtain the vector generation model.

15. The device according to any one of claims 11 to 13, characterized in that The apparatus further includes a second determining unit, configured to: The content category of the target content is determined according to the feature vector.

16. The device according to claim 11, characterized in that The target content is video content, and the video content includes text information from a title description and image information from a video frame. The first generating unit is configured to: A text vector to be determined corresponding to the text information and an image vector to be determined corresponding to the image information are generated.

17. A computer device, characterized in that: The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the method of claim 1 or 2, or execute the method of any one of claims 3 to 8, according to the instructions in the program code.

18. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method according to claim 1 or 2, or execute the method according to any one of claims 3 to 8.

19. A computer program product, characterized in that The computer program product includes computer instructions, and a processor of a computer device executes the computer instructions, so that the computer device executes the method according to claim 1 or 2, or executes the method according to any one of claims 3 to 8.

Images