A multi-modal industry question-answering large model training method and system

Abstract

The present invention discloses a multimodal industry question-and-answer big model training method and system, comprising: first obtaining a multi-relationship graph and the industry description information of the multimodal entities therein, the multi-relationship graph including a multimodal question entity and a multimodal answer entity. Then, confirm the multimodal entity instance from the multi-relationship graph, and obtain the link with its instance as the core node. Then, based on the industry description information of the multimodal entity instance link, the original multimodal industry question-and-answer big model and the original feature conversion network are optimized and learned, and the overall error includes the first learning error of the focus multimodal question entity, and finally the target multimodal industry question-and-answer big model and the target feature conversion network are obtained. With such a design, the accuracy of question-and-answer is improved by considering the complex relationships between multimodal entities, the characteristics of multimodal entities are mined, the model training process is optimized, and the ability of the multimodal industry question-and-answer system to answer questions in different industry fields is thereby improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal industry question-and-answer large model training method and system. Background Art

[0002] In today's era of information explosion, knowledge data in various industries is growing massively, and question-answering systems are facing higher requirements. Traditional question-answering models can often only process single-modal data and are difficult to adapt to the needs of multimodal information fusion. Multimodal data is very common in industry knowledge expression. For example, in the field of electronic products, product reviews include both text descriptions and product pictures and review videos. When dealing with multimodal industry questions and answers, existing question-answering models have problems such as insufficient mining of multimodal entity relationships and low model training accuracy. Summary of the invention

[0003] The purpose of the present invention is to provide a multi-modal industry question-answering large model training method and system.

[0004] In a first aspect, an embodiment of the present invention provides a multimodal industry question-answering large model training method, comprising:

[0005] Obtaining a multi-relationship graph and industry description information of each multimodal entity in the multi-relationship graph, wherein the multi-relationship graph includes a plurality of multimodal entities and a connection relationship exists between two multimodal entities with a certain degree of association, wherein the plurality of multimodal entities include a plurality of multimodal question entities representing questions and a plurality of multimodal answer entities representing preferred answers, wherein the plurality of multimodal question entities include at least one focus multimodal question entity, and the focus multimodal question entity has a connection relationship with at least one multimodal answer entity and at least one other multimodal question entity;

[0006] Confirm at least one multimodal entity instance from the multi-relationship graph, and for each multimodal entity instance, obtain at least one link of a multimodal entity with the multimodal entity instance as a core node from the multi-relationship graph, wherein first-level adjacent multimodal entities on a link are all multimodal question entities or multimodal answer entities, and a link in which first-level adjacent multimodal entities are all multimodal question entities is a first link, and a link in which first-level adjacent multimodal entities are all multimodal answer entities is a second link, wherein the at least one multimodal entity instance includes at least one focus multimodal question entity, and each link of the focus multimodal question entity includes at least one first link and at least one second link;

[0007] According to the industry description information of the multimodal entity on each link of each multimodal entity instance, the original multimodal industry question-answering big model and the original feature conversion network are optimized and learned to obtain the target multimodal industry question-answering big model and the target feature conversion network;

[0008] The overall error of the optimization learning includes a first learning error of each focal multimodal problem entity, and the first learning error is used to associate the target feature of the corresponding focal multimodal problem entity in each learning cycle with the difference information between the converted features obtained by the original feature conversion network according to the initial feature conversion of the corresponding focal multimodal problem entity in each learning cycle;

[0009] The target feature is a multimodal entity feature or a significant feature; the original multimodal industry question-and-answer big model is used to input the industry description information of the multimodal entity on each link of the multimodal entity instance, and output the multimodal entity feature of the multimodal entity instance; if each link of the multimodal entity instance includes at least one first link, the multimodal entity feature includes the initial feature obtained based on the industry description information of the multimodal entity on each first link; if each link of the multimodal entity instance includes at least one second link, the multimodal entity feature includes the significant feature obtained based on the industry description information of the multimodal entity on each second link.

[0010] In a second aspect, an embodiment of the present invention provides a server system, including a server, wherein the server is used to execute the method described in the first aspect.

[0011] Compared with the prior art, the beneficial effects provided by the present invention include: adopting a multimodal industry question and answer large model training method and system disclosed by the present invention, including: first obtaining the industry description information of the multi-relationship graph and the multimodal entities therein, the multi-relationship graph contains multimodal question entities and multimodal answer entities. Then, confirm the multimodal entity instance from the multi-relationship graph, and obtain the link with its instance as the core node. Then, based on the industry description information of the multimodal entity instance link, the original multimodal industry question and answer large model and the original feature conversion network are optimized and learned, and the overall error includes the first learning error of the focus multimodal question entity, and finally the target multimodal industry question and answer large model and the target feature conversion network are obtained. With such a design, the accuracy of question and answer is improved by considering the complex relationship between multimodal entities, the characteristics of multimodal entities are mined, and the model training process is optimized, thereby improving the ability of the multimodal industry question and answer system to answer questions in different industry fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can also be obtained based on these drawings without creative work.

[0013] Figure 1A schematic diagram of the steps of the multi-modal industry question-answering large model training method provided in an embodiment of the present invention;

[0014] Figure 2 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0016] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0017] In order to solve the technical problems in the aforementioned background technology, Figure 1 A flow chart of the multimodal industry question-and-answer big model training method provided in the embodiment of the present disclosure is provided, and the multimodal industry question-and-answer big model training method is introduced in detail below.

[0018] Step S201, obtaining a multi-relationship graph and industry description information of each multimodal entity in the multi-relationship graph, wherein the multi-relationship graph includes a plurality of multimodal entities and a connection relationship exists between two multimodal entities with a certain degree of association, wherein the plurality of multimodal entities include a plurality of multimodal question entities representing questions and a plurality of multimodal answer entities representing preferred answers, wherein the plurality of multimodal question entities include at least one focus multimodal question entity, and the focus multimodal question entity has a connection relationship with at least one multimodal answer entity and at least one other multimodal question entity;

[0019] Step S202, confirming at least one multimodal entity instance from the multi-relationship graph, for each multimodal entity instance, obtaining at least one link of a multimodal entity with the multimodal entity instance as a core node from the multi-relationship graph, wherein all first-level adjacent multimodal entities on a link are multimodal question entities or multimodal answer entities, and a link in which all first-level adjacent multimodal entities are multimodal question entities is a first link, and a link in which all first-level adjacent multimodal entities are multimodal answer entities is a second link, the at least one multimodal entity instance includes at least one focus multimodal question entity, and each link of the focus multimodal question entity includes at least one first link and at least one second link;

[0020] Step S203, optimizing and learning the original multimodal industry question-answering big model and the original feature conversion network according to the industry description information of the multimodal entity on each link of each multimodal entity instance, and obtaining the target multimodal industry question-answering big model and the target feature conversion network;

[0021] The overall error of the optimization learning includes a first learning error of each focal multimodal problem entity, and the first learning error is used to associate the target feature of the corresponding focal multimodal problem entity in each learning cycle with the difference information between the converted features obtained by the original feature conversion network according to the initial feature conversion of the corresponding focal multimodal problem entity in each learning cycle;

[0022] The target feature is a multimodal entity feature or a significant feature; the original multimodal industry question-and-answer big model is used to input the industry description information of the multimodal entity on each link of the multimodal entity instance, and output the multimodal entity feature of the multimodal entity instance; if each link of the multimodal entity instance includes at least one first link, the multimodal entity feature includes the initial feature obtained based on the industry description information of the multimodal entity on each first link; if each link of the multimodal entity instance includes at least one second link, the multimodal entity feature includes the significant feature obtained based on the industry description information of the multimodal entity on each second link.

[0023] In an embodiment of the present invention, for example, it is assumed that the server is processing a question-answering model training for an electronic product manufacturing industry. In this industry, multimodal question entities can be questions about the functions, designs, manufacturing processes, etc. of electronic products. For example, a multimodal question entity "How does the screen resolution of a smartphone affect the display effect" can contain multimodal information such as text descriptions and related screen resolution charts. Multimodal answer entities are answers to these questions. For example, answers such as "The higher the screen resolution, the more pixels per inch, the clearer the image, but it will also increase energy consumption" can also be accompanied by some comparison pictures of screen display effects with different resolutions. The server identifies the connection relationship between two multimodal entities with a certain degree of association. For example, the question entity "How does the screen resolution of a smartphone affect the display effect" has a connection relationship with the answer entity "The higher the screen resolution, the more pixels per inch, the clearer the image, but it will also increase energy consumption". At the same time, this question entity also has a connection relationship with other question entities such as "How to improve the screen resolution of a smartphone" because they all revolve around topics related to smartphone screens. For the multimodal question entity "How does the resolution of smartphone screens affect display effects", its industry description information may include text information such as the screen technology development trend of the smartphone industry (such as the impact of the transition from LCD to OLED on resolution), industry standards for screen resolution (such as the definition and characteristics of standards such as HD, FHD, and QHD), and user demand for screen display effects (such as gamers' preference for high resolution), and may also include some relevant video materials, such as the effect display of screens with different resolutions when playing high-definition videos, and other multimodal industry description information. For the multimodal answer entity "The higher the screen resolution, the more pixels per inch, the clearer the image, but it also increases energy consumption", the industry description information may include information such as the general law of the relationship between energy consumption and screen resolution in the entire electronic product manufacturing industry (such as through statistical data on mobile phones of different brands), the application of quantitative standards for image clarity in the industry, and also some charts showing the comparison of energy consumption at different resolutions, and other multimodal content. When processing the multi-relationship graph of the above electronic product manufacturing industry, the server determines a focus multimodal question entity as "How to improve the screen resolution of smartphones". This focus multimodal question entity is connected to the multimodal answer entity "The resolution can be improved by improving the pixel arrangement, such as using Pentile arrangement" and other multimodal question entities "What is the impact of increasing screen resolution on battery life?" At the same time, the server will also identify other multimodal entity instances, such as the multimodal answer entity "Using more advanced display chips can help improve screen resolution" as a multimodal entity instance. For the focus multimodal question entity "How to improve the screen resolution of a smartphone", the server obtains the link of the multimodal entity with it as the core node from the multi-relationship graph.A first link may be: "How to improve the screen resolution of a smartphone" - "What is the impact of improving screen resolution on battery life" - "The relationship between battery life and screen power consumption". Here, all first-level adjacent multimodal entities are multimodal question entities. A second link may be: "How to improve the screen resolution of a smartphone" - "The resolution can be improved by improving the pixel arrangement, such as using Pentile arrangement" - "The application of different pixel arrangements in other electronic products". Here, all first-level adjacent multimodal entities are multimodal answer entities. Moreover, each link of the focus multimodal question entity "How to improve the screen resolution of a smartphone" includes at least one first link and at least one second link. For the multimodal entity instance of the multimodal answer entity "Using more advanced display chips can help improve screen resolution", its links may be: "Using more advanced display chips can help improve screen resolution" - "What advanced display chips are currently suitable for smartphones" (this is a first link), and "Using more advanced display chips can help improve screen resolution" - "How does the upgrade of display chips affect screen manufacturing costs" (this is a second link). Assume that in a learning cycle, for the focus multimodal question entity "how to improve the screen resolution of a smartphone", the server inputs the industry description information of the multimodal entities on its link (such as the industry description information of the battery life, pixel arrangement and other related links mentioned above) through the original multimodal industry question and answer big model, and outputs the multimodal entity features. If the link of the multimodal entity instance includes the first link, the initial feature is obtained according to the industry description information of the multimodal entity on the first link (such as "what is the impact of improving screen resolution on battery life", etc.). Assume that this initial feature is a vector containing the relationship between screen resolution and battery life, which contains numerical values ​​(such as the relationship between the percentage of battery life reduction and the resolution improvement) and related text description encoding and other multimodal information. If the link includes the second link, the significant feature is obtained according to the industry description information of the multimodal entity on the second link (such as "resolution can be improved by improving pixel arrangement, such as using Pentile arrangement", etc.). For example, this significant feature is a comprehensive vector of quantitative and qualitative descriptions of the effect of different pixel arrangements on resolution improvement. The server obtains the conversion feature according to the initial feature conversion of the corresponding focus multimodal problem entity in this learning cycle through the original feature conversion network. Assume that the original feature conversion network is a neural network, which performs a nonlinear transformation on the initial feature to obtain the conversion feature. The target feature can be a multimodal entity feature or a salient feature. Assuming that the target feature is a salient feature, the first learning error is the difference between the salient feature (target feature) and the conversion feature. For example, the difference between the quantized value of the resolution improvement effect of the pixel arrangement in the salient feature and the corresponding quantized value in the conversion feature, and the difference measurement between the text description encodings, etc.The server performs such operations for each link of each multimodal entity instance. For example, for the multimodal answer entity "adopting more advanced display chips helps improve screen resolution", the multimodal entity features are obtained through the original multimodal industry question and answer big model according to the industry description information on its links (such as the first link about the application of advanced display chips to smartphones and the second link about the impact of display chip upgrades on manufacturing costs), and then the relevant error information is calculated. During the entire optimization learning process, the server continuously adjusts the parameters of the original multimodal industry question and answer big model and the original feature conversion network to reduce the overall error. This overall error includes multiple error components such as the first learning error of each focal multimodal question entity. As the learning cycle continues, the original multimodal industry question and answer big model and the original feature conversion network are gradually optimized, and finally the target multimodal industry question and answer big model and the target feature conversion network are obtained. For example, after multiple learning cycles, the target multimodal industry question and answer big model can more accurately output more reasonable multimodal entity features based on the industry description information of the multimodal entity instance link, and the target feature conversion network can more accurately convert the initial features to make the error between them and the target features smaller.

[0024] In an embodiment of the present invention, the at least one multimodal entity instance also includes a multimodal answer entity; the overall error of the optimization learning also includes the second learning error of each focus multimodal question entity; in each cycle of optimization learning, the second error parameter of each focus multimodal question entity is obtained by the following process:

[0025] A multimodal answer entity having a connection relationship with the focus multimodal question entity is selected from the multi-relationship graph to be determined as a target multimodal answer entity, and a multimodal answer entity having no connection relationship with the focus multimodal question entity is selected from the multi-relationship graph to be determined as a non-target multimodal answer entity;

[0026] Polling each target multimodal answer entity and each non-target multimodal answer entity to obtain a plurality of multimodal answer entity pairs, each multimodal answer entity pair including a target multimodal answer entity and a non-target multimodal answer entity;

[0027] For each multimodal answer entity pair, a first matching coefficient of the focus multimodal question entity and the target multimodal answer entity in the multimodal answer entity pair is obtained according to the multimodal entity features of the focus multimodal question entity and the target multimodal answer entity in the multimodal answer entity pair in the current learning cycle, a second matching coefficient of the focus multimodal question entity and the non-target multimodal answer entity is obtained according to the multimodal entity features of the focus multimodal question entity and the non-target multimodal answer entity in the multimodal answer entity pair in the current learning cycle, and a matching error of the focus multimodal question entity for the multimodal answer entity pair is obtained according to the difference value of the distinguishing information between the first matching coefficient and the second matching coefficient;

[0028] A second learning error of the focus multimodal question entity is obtained according to the sum of the matching errors of the focus multimodal question entity to all multimodal answer entity pairs.

[0029] In the embodiment of the present invention, for example, continuing to take the electronic product manufacturing industry as an example, it is assumed that the server is processing the focus multimodal question entity "How to improve the screen resolution of a smartphone". In the multi-relationship graph, the multimodal answer entity (target multimodal answer entity) that has a connection relationship with "How to improve the screen resolution of a smartphone" may include "The resolution can be improved by improving the pixel arrangement, such as using Pentile arrangement" and "The use of more advanced display chips helps to improve the screen resolution". The multimodal answer entity (non-target multimodal answer entity) that has no connection relationship with "How to improve the screen resolution of a smartphone" may be "The material selection of the smartphone shell" and the like, because it is not directly related to the problem of improving the screen resolution. The server begins to poll each target multimodal answer entity and each non-target multimodal answer entity to obtain multiple multimodal answer entity pairs. For the target multimodal answer entity "The resolution can be improved by improving the pixel arrangement, such as using Pentile arrangement" and the non-target multimodal answer entity "The material selection of the smartphone shell", a multimodal answer entity pair ("The resolution can be improved by improving the pixel arrangement, such as using Pentile arrangement", "The material selection of the smartphone shell") is formed. Then, for the target multimodal answer entity "adopting more advanced display chips can help improve screen resolution" and the non-target multimodal answer entity "material selection of smartphone shell", another multimodal answer entity pair ("adopting more advanced display chips can help improve screen resolution", "material selection of smartphone shell") is formed. The server will poll all target multimodal answer entities and non-target multimodal answer entities in this way to obtain a series of multimodal answer entity pairs. For the focus multimodal question entity "how to improve smartphone screen resolution" and the target multimodal answer entity "resolution can be improved by improving pixel arrangement, such as Pentile arrangement", "material selection of smartphone shell") in the multimodal answer entity pair ("resolution can be improved by improving pixel arrangement, such as Pentile arrangement", "material selection of smartphone shell"), the server calculates the first matching coefficient based on their multimodal entity features in this round of learning cycle. Assume that in this round of learning cycle, the multimodal entity features of the focus multimodal question entity include information such as the quantitative representation of the screen resolution improvement demand (such as the expected resolution value), the text description encoding of the relevant technical improvement direction, etc. The multimodal entity features of the target multimodal answer entity include quantitative data on the resolution improvement effect of different pixel arrangements (such as the percentage of resolution that can be improved by using Pentile arrangement) and text description encoding of relevant technical principles. The server obtains the first matching coefficient through a predefined matching algorithm (for example, based on vector similarity calculation, calculating the cosine similarity of the feature vectors of the two), assuming that this coefficient is 0.8.For the non-target multimodal answer entity "material selection of smartphone shell", its multimodal entity features include the physical properties of the shell material (such as hardness, weight, etc.), quantitative data of material cost, and text description encoding of related application scenarios. The server also calculates the second matching coefficient of the focus multimodal question entity and the non-target multimodal answer entity through a predefined matching algorithm, assuming that this coefficient is 0.1. Then, the matching error of the focus multimodal question entity for this multimodal answer entity pair is obtained based on the difference value of the distinguishing information between the first matching coefficient (0.8) and the second matching coefficient (0.1). The difference value of the distinguishing information here can be the difference between the two, that is, 0.8-0.1=0.7, and this 0.7 is the matching error of this multimodal answer entity pair. The server will sum the matching errors of the focus multimodal question entity "how to improve the screen resolution of smartphones" for all multimodal answer entity pairs. Assume that in addition to the multimodal answer entity pairs mentioned above, there are other multimodal answer entity pairs, such as ("adopting more advanced display chips helps improve screen resolution", "charging interface types for smartphone batteries"), etc., and the matching errors of each multimodal answer entity pair are calculated to be 0.6, 0.5, etc. Then the second learning error of the focus multimodal question entity is the sum of all these matching errors, that is, 0.7+0.6+0.5=1.8. This second learning error will be used as part of the overall error of the optimized learning for subsequent optimization and adjustment of the original multimodal industry question and answer model and the original feature conversion network, so that the model is more accurate in distinguishing answer entities related to and unrelated to the focus multimodal question entity.

[0030] In an embodiment of the present invention, the overall error of the optimization learning further includes a third learning error of the multimodal problem entity in each multimodal entity instance;

[0031] The third learning error of the multimodal problem entity in each multimodal entity instance is obtained as follows:

[0032] For each cycle of optimization learning, the core node features of the group core nodes of each category group in this round of learning cycle and the multimodal entity features of each multimodal problem entity in this round of learning cycle are obtained, wherein the core node features of the group core nodes of each category group in the first group category division are preset parameters;

[0033] For each multimodal question entity, a matching coefficient between the multimodal question entity and the corresponding group core node is obtained according to the multimodal entity features of the multimodal question entity in the current learning cycle and the core node features of each group core node;

[0034] According to the matching coefficient between each multimodal problem entity and each group core node, the probability of each multimodal problem entity being classified into each category group is obtained;

[0035] For each multimodal question entity, a third learning error of the multimodal question entity is obtained according to difference information between a matching coefficient between the multimodal question entity and each group core node and a probability that the multimodal question entity is classified into a corresponding category group.

[0036] In an embodiment of the present invention, illustratively, taking the electronic product manufacturing industry as an example again, it is assumed that there are multiple category groups in the multi-relationship graph. For example, one category group may be a multimodal entity related to the screen (including multimodal question entities and answer entities related to screen resolution, pixel arrangement, etc.), and another category group may be a multimodal entity related to the battery (such as multimodal question entities and answer entities related to battery capacity, charging speed, etc.). During each cycle of optimization learning, the server must first obtain the core node features of the group core nodes of each category group in this round of learning cycle and the multimodal entity features of each multimodal question entity in this round of learning cycle. For the screen-related category group, it is assumed that its group core node is the multimodal entity "screen display technology". When the group category is divided for the first time, its core node features may be some preset parameters, such as text description information such as the basic type of display technology (such as LCD, OLED), the development history of display technology, and multimodal information such as image examples of some typical display technologies. As the learning cycle progresses, this core node feature will be continuously updated. For the battery-related category group, the core node of the group can be "battery energy storage principle", and its first core node feature as a preset parameter can include text descriptions of the basic energy storage principle of chemical batteries (such as the charging and discharging principle of lithium-ion batteries), some battery structure schematics and other multimodal information. At the same time, the server obtains the multimodal entity features of each multimodal problem entity in this round of learning cycle. For example, for the multimodal problem entity "how to improve the screen resolution of smartphones", its multimodal entity features can include information such as the numerical range of the current screen resolution, the degree of user demand for higher resolution (quantitative representation through user survey data), and the technical direction involved in improving resolution (such as hardware improvement, software optimization, etc. Text description encoding). For the multimodal problem entity "how to improve the battery capacity of smartphones", its multimodal entity features can include statistical data on existing battery capacity, battery volume limiting factors (such as text descriptions of the internal space layout of mobile phones), and some research progress on new battery materials (such as text encoding of relevant paper abstracts). For the multimodal problem entity "how to improve the screen resolution of smartphones", the server calculates the matching coefficient based on its multimodal entity features in this round of learning cycle and the core node features of each group core node. When calculating the matching coefficient with the group core node "Screen Display Technology" of the screen-related category group, the server will compare the display technology-related parts of the multimodal problem entity features (such as the correlation between the hardware improvement direction involved in improving resolution and the basic type and development history of display technology). Assuming that a content-based matching algorithm is used (for example, the features of both are converted into vectors, then dot product operations are performed and normalized), the matching coefficient obtained is 0.7. When matching with the group core node "Battery Energy Storage Principle" of the battery-related category group, since the correlation between the two is relatively small, the calculated matching coefficient can be 0.1.The server obtains the probability of each multimodal question entity being classified into each category group according to the matching coefficient between each multimodal question entity and each group core node. Assume that in addition to the screen-related and battery-related category groups, there are other category groups, such as the shell design category group, and the matching coefficients calculated with the core nodes of each group are 0.7 (screen-related), 0.1 (battery-related), 0.05 (shell design-related), etc. Then the probability that the multimodal question entity "How to improve the screen resolution of smartphones" is classified into the screen-related category group can be calculated by dividing its matching coefficient with the core node of the screen-related group by the sum of all matching coefficients, that is, 0.7 / (0.7+0.1+0.05)≈0.82. The probability of being classified into the battery-related category group is 0.1 / (0.7+0.1+0.05)≈0.12, and the probability of being classified into the shell design-related category group is 0.05 / (0.7+0.1+0.05)≈0.06. For the multimodal problem entity "How to improve the screen resolution of smartphones", the server obtains the third learning error based on the difference information between the matching coefficient between it and the core nodes of each group and the probability of the multimodal problem entity being classified into the corresponding category group. Calculation of the difference information of the screen-related category group: Assuming the matching coefficient is 0.7 and the classification probability is 0.82, the difference information of the two can be the absolute value of the difference between the two, that is, |0.7-0.82|=0.12. Calculation of the difference information of the battery-related category group: The matching coefficient is 0.1, the classification probability is 0.12, and the difference information is |0.1-0.12|=0.02. Calculation of the difference information of the shell design-related category group: The matching coefficient is 0.05, the classification probability is 0.06, and the difference information is |0.05-0.06|=0.01. The third learning error of the multimodal problem entity is the sum of these difference information, that is, 0.12+0.02+0.01=0.15. This third learning error will be used as part of the optimized learning error to adjust the model parameters, making the classification of multimodal question entities in category groups more accurate, thereby improving the performance of the multimodal industry question-answering model.

[0037] In this embodiment of the present invention, the first learning error of the focus multimodal question entity is obtained by the following process:

[0038] According to the similarity measurement between each group core node and the target feature of the focal multimodal problem entity in this round of learning cycle, the core node feature of each group core node is linearly superimposed to be determined as a first superimposed feature, and a feature enhancement operation is performed on the target feature according to the first superimposed feature to obtain a target feature after feature enhancement, and the target feature after feature enhancement is determined as the focal feature of the focal multimodal problem entity in this round of learning cycle;

[0039] According to the similarity measurement between each group core node and the initial features of the focal multimodal problem entity in this round of learning cycle, the core node features of each group core node are linearly superimposed to determine as second superimposed features, and according to the second superimposed features, the initial features are enhanced to obtain the initial features after feature enhancement, and the initial features after feature enhancement are transformed through the original feature transformation network to obtain the transformation features of the focal multimodal problem entity in this round of learning cycle;

[0040] According to the difference information between the focus feature and the conversion feature of the focus multimodal question entity in this round of learning cycle, a first learning error of the focus multimodal question entity is obtained.

[0041] In the embodiment of the present invention, for example, still taking the electronic product manufacturing industry as an example, it is assumed that the focus multimodal problem entity is "how to improve the screen resolution of smartphones". In this round of learning cycle, the target feature is a multimodal entity feature or a significant feature related to the problem entity. It is assumed that the target feature includes the technical requirements for improving screen resolution (such as the numerical requirements for pixel density improvement, the text description of the expectation for display effect improvement, and multimodal information such as related screen technology charts). It is assumed that there are three category groups in the multi-relationship graph: screen technology category group, battery technology category group, and mobile phone shell design category group. For the group core node of the screen technology category group (such as "screen display technology"), its core node features include multimodal information such as text and images such as the type of screen display technology (such as LCD, OLED), the upper limit of resolution of different technologies. The core node features of the group core node of the battery technology category group (such as "battery energy supply mechanism") include multimodal information such as text and data related to the voltage and capacity of the battery, and the core node features of the group core node of the mobile phone shell design category group (such as "shell material and structure") include multimodal information such as shell material characteristics and design style. The server first calculates the similarity measure between each group core node and the target feature of the focus multimodal problem entity in this round of learning cycle. For the target feature of the group core node of the screen technology category group and the focus multimodal problem entity, the server uses a similarity calculation method (such as calculating the cosine similarity of the multimodal feature vectors of the two) and assumes that the similarity measure is 0.8. The similarity measure between the group core node of the battery technology category group and the target feature is 0.1, and the similarity measure between the group core node of the mobile phone shell design category group and the target feature is 0.05. Then, according to these similarity measures, linear superposition is performed on the core node features of each group core node. Assume that the core node feature vector of the screen technology category group is (V_1), the core node feature vector of the battery technology category group is (V_2), and the core node feature vector of the mobile phone shell design category group is (V_3). Then the first superposition feature (V_{superposition 1}=0.8V_1+0.1V_2+0.05V_3). Then, according to the first superposition feature (V_{superposition 1}), a feature enhancement operation is performed on the target feature. For example, if the numerical requirement for screen resolution improvement in the target feature is (X), in the feature enhancement operation, (X) will be adjusted according to the screen technology related information in (V_{overlay 1}). Assume that after the feature enhancement operation, the numerical requirement for screen resolution improvement becomes (X'), and the related screen technology chart is also refined according to the information in (V_{overlay 1}) (such as adding resolution improvement examples related to specific screen technology), and the target feature after feature enhancement is obtained, which is determined as the focus feature of the focus multimodal problem entity in this round of learning cycle.The server then calculates the similarity measure between each group core node and the initial features of the focal multimodal problem entity in this round of learning cycle. The initial features can include some preliminary analysis information about screen resolution, such as the approximate range of screen resolutions of smartphones currently on the market, preliminary factors affecting screen resolution improvement, and other multimodal information. Assume that through calculation, the similarity measure between the group core node and the initial features of the screen technology category group is 0.7, the battery technology category group is 0.08, and the mobile phone shell design category group is 0.02. Then, according to these similarity measures, the core node features of each group core node are linearly superimposed to obtain the second superimposed feature (V_{superimposed 2}=0.7V_1+0.08V_2+0.02V_3). The initial features are enhanced according to the second superimposed feature (V_{superimposed 2}). For example, the current screen resolution range in the initial features will be adjusted according to the screen technology related information in (V_{superimposed 2}) to obtain the initial features after feature enhancement. Finally, the initial features after feature enhancement are transformed through the original feature conversion network. The original feature conversion network can be a feature conversion model based on a neural network, which performs nonlinear transformation on the initial features (vectors after multimodal information processing) after feature enhancement to obtain the conversion features of the focus multimodal problem entity in this round of learning cycle. The server obtains the first learning error based on the difference information between the focus features and the conversion features of the focus multimodal problem entity in this round of learning cycle. The focus features include multimodal information such as the screen resolution improvement requirements after enhancement and related technical diagrams, and the conversion features are information related to the initial features after conversion by the original feature conversion network. For example, the screen resolution improvement requirements in the focus features are (X'), and the corresponding information related to the screen resolution improvement after conversion in the conversion features is (Y), and the numerical difference between the two is (|X'-Y|). At the same time, for the chart part in the multimodal information, the server calculates the difference metric between the two through an image comparison algorithm. The difference information such as these numerical differences and image difference metrics are combined to obtain the first learning error of the focus multimodal problem entity. This first learning error will be used to optimize the original multimodal industry question-answering large model and the original feature conversion network to improve the model's processing accuracy of the focus multimodal problem entity.

[0042] In the embodiment of the present invention, the method of obtaining the probability of each multimodal question entity being classified into each category group according to the matching coefficient between each multimodal question entity and each group core node includes:

[0043] For each group core node, the sum of the matching coefficients of each sample multimodal problem entity corresponding to the group core node is determined as the cohesion of the category group corresponding to the group core node;

[0044] For each multimodal question entity, the importance of the multimodal question entity in the category group is obtained according to the matching coefficient between the multimodal question entity and a group core node, and the cohesion of the category group corresponding to the group core node;

[0045] For each multimodal question entity, the proportion of the importance of the multimodal question entity in a category group to the total importance of the multimodal question entity in all category groups is determined as the probability of the multimodal question entity being classified into the category group.

[0046] In the embodiment of the present invention, illustratively, continuing to take the electronic product manufacturing industry as an example, it is assumed that there are three category groups: screen technology category group, battery technology category group and mobile phone shell design category group. For the screen technology category group, its group core node is "screen display technology". Assume that there are multiple sample multimodal problem entities, such as "how to improve the screen resolution of smartphones", "what is the principle of screen color display", "the impact of different screen ratios on visual experience", etc. The server calculates the matching coefficients of these sample multimodal problem entities and the group core node "screen display technology". For "how to improve the screen resolution of smartphones", it is assumed that the matching coefficient is 0.7; for "what is the principle of screen color display", it is assumed that the matching coefficient is 0.6; for "the impact of different screen ratios on visual experience", it is assumed that the matching coefficient is 0.5. The server determines the sum of the matching coefficients of each sample multimodal problem entity corresponding to the group core node "screen display technology" as the cohesion of the category group corresponding to the group core node. In this example, the cohesion of the screen technology category group is 0.7+0.6+0.5=1.8. Similarly, for the battery technology category group, its group core node is "battery energy supply mechanism". Assuming that the matching coefficients calculated for the sample multimodal problem entities "how to improve the battery capacity of smartphones" and "factors affecting battery charging speed" are 0.8 and 0.7 respectively, the cohesion of the battery technology category group is 0.8+0.7=1.5. For the mobile phone shell design category group, its group core node is "shell material and structure". Assuming that the matching coefficients calculated for the sample multimodal problem entities "the impact of the material of the mobile phone shell on heat dissipation" and "how the shell design affects the grip of the mobile phone" are 0.6 and 0.5 respectively, the cohesion of the mobile phone shell design category group is 0.6+0.5=1.1. For the multimodal problem entity "how to improve the screen resolution of smartphones", its matching coefficient with the group core node "screen display technology" of the screen technology category group is 0.7, and the cohesion of the screen technology category group is 1.8. The server calculates the importance of the multimodal problem entity in the category group based on the matching coefficient between the multimodal problem entity and the core node of the group, and the cohesion of the category group corresponding to the core node of the group. The importance can be calculated as the ratio of the two, that is, importance = 0.7 / 1.8≈0.39. For the multimodal problem entity "How to improve the battery capacity of smartphones" and the battery technology category group, the matching coefficient is 0.8, the cohesion is 1.5, and its importance in the battery technology category group is 0.8 / 1.5≈0.53. For the multimodal problem entity "The impact of the material of the mobile phone shell on heat dissipation" and the mobile phone shell design category group, the matching coefficient is 0.6, the cohesion is 1.1, and its importance in the mobile phone shell design category group is 0.6 / 1.1≈0.55.For the multimodal problem entity "How to improve the screen resolution of smartphones", its importance in the screen technology category group is 0.39. Assume that its importance in the battery technology category group is 0.1 (the calculation process is similar to the previous one), and its importance in the mobile phone shell design category group is 0.05. The total importance of this multimodal problem entity in all category groups is 0.39+0.1+0.05=0.54. Then the probability that "How to improve the screen resolution of smartphones" is classified into the screen technology category group is 0.39 / 0.54≈0.72. For the multimodal problem entity "How to improve the battery capacity of smartphones", its importance in the battery technology category group is 0.53, assuming that its importance in the screen technology category group is 0.08, and its importance in the mobile phone shell design category group is 0.03. The total importance is 0.53+0.08+0.03=0.64, and its probability of being classified into the battery technology category group is 0.53 / 0.64≈0.83. For the multimodal question entity "The impact of the material of the mobile phone shell on heat dissipation", its importance in the mobile phone shell design category group is 0.55. It is assumed that its importance in the screen technology category group is 0.05, and its importance in the battery technology category group is 0.02. The total importance is 0.55+0.05+0.02=0.62, and its probability of being classified in the mobile phone shell design category group is 0.55 / 0.62≈0.89. In this way, the server can accurately determine the probability of each multimodal question entity being classified into each category group, which helps to further optimize the classification and association capabilities of the multimodal industry question and answer model and improve the accurate processing of multimodal entities.

[0047] In the embodiment of the present invention, each cycle optimization learning further includes:

[0048] For each multimodal question entity, determining the group core node with the highest matching coefficient corresponding to the multimodal question entity as the target group core node, and determining the category group corresponding to the target group core node as the category group corresponding to the multimodal question entity in the next learning cycle;

[0049] For each category group in the next learning cycle, the core node characteristics of the core nodes of the corresponding group are adjusted according to the overall error of the optimization learning.

[0050] In the embodiment of the present invention, illustratively, continuing to take the electronic product manufacturing industry as an example, it is assumed that there are multimodal problem entities such as "how to improve the screen resolution of smartphones", "how to extend the battery life of smartphones" and "how to improve the heat dissipation performance of mobile phone shells". For the screen technology category group, its group core node is "screen display technology"; the group core node of the battery technology category group is "battery energy supply mechanism"; the group core node of the mobile phone shell design category group is "shell material and structure". For the multimodal problem entity "how to improve the screen resolution of smartphones", the server calculates its matching coefficient with each group core node. Assume that the matching coefficient with "screen display technology" is 0.7, the matching coefficient with "battery energy supply mechanism" is 0.1, and the matching coefficient with "shell material and structure" is 0.05. Since the matching coefficient with "screen display technology" is the highest, the server determines "screen display technology" as the target group core node, and determines the screen technology category group as the category group corresponding to the multimodal problem entity "how to improve the screen resolution of smartphones" in the next learning cycle. For the multimodal problem entity "How to extend the battery life of smartphones", it is assumed that the matching coefficient with "screen display technology" is 0.08, the matching coefficient with "battery energy supply mechanism" is 0.85, and the matching coefficient with "shell material and structure" is 0.03. The server determines "battery energy supply mechanism" as the core node of the target group, and the corresponding battery technology category group is the category group corresponding to the multimodal problem entity in the next learning cycle. For the multimodal problem entity "How to improve the heat dissipation performance of mobile phone shell", it is assumed that the matching coefficient with "screen display technology" is 0.05, the matching coefficient with "battery energy supply mechanism" is 0.03, and the matching coefficient with "shell material and structure" is 0.8. The server determines "shell material and structure" as the core node of the target group, and the mobile phone shell design category group is the category group corresponding to the multimodal problem entity in the next learning cycle. After each learning cycle, the server has obtained the overall error of the optimized learning, which includes the various errors mentioned above (such as the first learning error and second learning error of the focus multimodal problem entity, and the third learning error of the multimodal problem entity). For the group core node "Screen Display Technology" of the screen technology category group, it is assumed that the overall error of the optimization learning indicates that there is a large error in the screen resolution-related questions (for example, it is not accurate enough when answering questions about the impact of screen resolution improvement on display effects). The server adjusts the core node feature of "Screen Display Technology" based on this overall error. If the overall error shows a problem in the resolution value calculation, the server will update the screen resolution standard value in the core node feature (for example, adjust the maximum resolution value under different screen technologies based on more industry data), and will update the resolution-related image examples (such as adding more comparison pictures of screen display effects under different resolutions).At the same time, if there are errors in the text description, such as inaccurate description of the principle of improving the resolution of a certain screen technology, the server will correct the relevant technical principle text description content. For the group core node "Battery Energy Supply Mechanism" of the battery technology category group, if the overall error shows that there is a problem in the answer to the relationship between battery capacity and usage time. The server will adjust the quantitative data of the relationship between battery capacity and energy output in the core node features (for example, adjust the battery usage time estimate under different power levels based on the new battery test results), and update the technical process diagram related to battery energy supply (such as the diagram of optimizing the battery charging and discharging process), and correct the relevant text description (such as the description of the battery capacity calculation method). For the group core node "Shell Material and Structure" of the mobile phone shell design category group, if the overall error shows that the answer to the impact of the shell material on the heat dissipation performance is inaccurate. The server will adjust the quantitative data such as the shell material thermal conductivity coefficient in the core node features (based on the new material test results), update the heat dissipation structure example diagram of shells of different materials, and correct the text description content about the relationship between shell material and heat dissipation. Through such adjustments, in the next learning cycle, the core node features of the group core nodes can better reflect the actual situation, making the matching between the multimodal question entities and the group core nodes more accurate, thereby improving the performance of the multimodal industry question and answer large model.

[0051] In an embodiment of the present invention, the plurality of multimodal question entities include at least one non-focus multimodal question entity, and the non-focus multimodal question entity is a multimodal question entity that has no connection relationship with the multimodal answer entity;

[0052] At least one multimodal entity instance also includes a non-focus multimodal problem entity, the links of the non-focus multimodal problem entity are all first links, and the multimodal entity features are initial features.

[0053] In an embodiment of the present invention, exemplarily, in the multi-relationship graph of the electronic product manufacturing industry, the multimodal question entity covers various issues related to electronic products. For example, the focus multimodal question entity can be "how to improve the screen resolution of a smartphone", which has a connection relationship with some multimodal answer entities (such as "resolution can be improved by improving the pixel arrangement, such as using Pentile arrangement", etc.), and has its specific link structure in the multi-relationship graph, including a first link (connected to other multimodal question entities) and a second link (connected to the multimodal answer entity). Non-focus multimodal question entities, such as "the development history of smartphone camera pixels", are multimodal question entities that have no connection relationship with multimodal answer entities. In the logical structure of the entire multi-relationship graph, it mainly focuses on the relationship between a series of question entities related to itself, and does not directly connect with the answer entity. For this non-focus multimodal question entity "the development history of smartphone camera pixels", its links are all first links. For example, it has a connection relationship with other multimodal question entities such as "what are the pixels of early smartphone cameras" and "what factors drive the growth of camera pixels". These connection relationships constitute the first link, and since there is no direct connection with the answer entity, its multimodal entity features are initial features. For the non-focus multimodal question entity "the development history of smartphone camera pixels", when processing its link, the server obtains information from each reference link in the link (since they are all first links, the reference links here are these first links) to determine the initial features. Taking the link "the development history of smartphone camera pixels"-"how many pixels were there in early smartphone cameras" as an example, the server obtains information from this link. For the multimodal entity "how many pixels were there in early smartphone cameras", the server divides the group category features of the higher-level adjacent multimodal entity (here "the development history of smartphone camera pixels") connected to it (for example, category features related to camera technology, which can include multimodal information such as text and images such as the development stage of camera technology and major technological breakthroughs at different stages) and the industry description information configured by itself (such as statistical data on early smartphone models and corresponding camera pixels, related product pictures, and other multimodal information), hierarchically to the core node multimodal entity (here "the development history of smartphone camera pixels") until the group category features of the core node multimodal entity on this reference link are obtained. For example, the server extracts information such as the changing trend of pixel values ​​and typical pixel values ​​at different stages from the statistical data of early smartphone models and corresponding camera pixels, and combines it with relevant information on the development stage of camera technology to construct a group category feature vector containing these multimodal information. This vector is the group category feature of the non-focal multimodal problem entity "development history of smartphone camera pixels" on this benchmark link.The server divides the group category features of the multimodal entity instance (here, "the development history of smartphone camera pixels") on each benchmark link with weights (for example, the weights are determined based on factors such as the importance of the link and the accuracy of relevant information), and obtains the features of the multimodal entity instance corresponding to the benchmark link. Since it is the first link, this feature is the initial feature. Assuming that the server has determined multiple benchmark links, such as "the development history of smartphone camera pixels" - "how many pixels are there in early smartphone cameras", "the development history of smartphone camera pixels" - "what factors have driven the growth of camera pixels", etc., the above operations are performed for each benchmark link, and finally the initial features of the non-focus multimodal problem entity "the development history of smartphone camera pixels" are obtained by combining these results. This initial feature contains multimodal information about the development history of smartphone camera pixels obtained from each link, such as quantitative data on pixel value changes at different stages, text descriptions of factors driving pixel growth, and related image examples, etc. This information will play a role in subsequent optimization learning of the multimodal industry question-answering model and other operations.

[0054] In the embodiment of the present invention, the initial features or salient features of each multimodal entity instance are obtained by the following process:

[0055] From each link of the multimodal entity instance being a core node multimodal entity, obtaining each reference link, the reference links are all first links or all second links;

[0056] For each reference link, starting from the highest-level multimodal entity in the reference link, according to the group category characteristics of each higher-level adjacent multimodal entity connected to the multimodal entity and the industry description information configured by the multimodal entity, the core node multimodal entity group is divided by level until the group category characteristics of the core node multimodal entity on the reference link are obtained; wherein the group category characteristics of the highest-level adjacent multimodal entity are obtained according to the industry description information configured by the highest-level adjacent multimodal entity;

[0057] Based on the local perceptron, the group category features of the multimodal entity instances on each reference link are divided into groups with weights to obtain the features of the multimodal entity instances corresponding to the reference link, wherein if the reference link is the first link, the corresponding features are initial features, and if the reference link is the second link, the corresponding features are significant features.

[0058] In the embodiment of the present invention, illustratively, the electronic product manufacturing industry is continued to be taken as an example, assuming that the multimodal entity instance is "smartphone screen display technology". This multimodal entity instance has multiple links in the multi-relationship graph. The link may include connections with other multimodal question entities, such as "smartphone screen display technology"-"the relationship between screen resolution and display effect", which is a first link; there are also connections with multimodal answer entities, such as "smartphone screen display technology"-"the advantages of OLED screens are high contrast and wide viewing angle", which is a second link. For the multimodal entity instance "smartphone screen display technology", the server obtains each reference link from each of its links. For the first link "smartphone screen display technology"-"the relationship between screen resolution and display effect", this can be used as a reference link because it is the first link. For the second link "smartphone screen display technology"-"the advantages of OLED screens are high contrast and wide viewing angle", this can also be used as a reference link because it is the second link. The server will traverse all links in this way to find all reference links that meet the requirements (all are first links or all are second links). Take the first link "smartphone screen display technology"-"the relationship between screen resolution and display effect" as an example. Starting from the highest-level multimodal entity in the benchmark link, the highest-level multimodal entity here is "the relationship between screen resolution and display effect". According to the group category characteristics of the higher-level adjacent multimodal entity connected to this multimodal entity (here is "smartphone screen display technology") and the industry description information configured by itself, it is divided into core node multimodal entities (here is "smartphone screen display technology") by level. For the multimodal entity "the relationship between screen resolution and display effect", the industry description information configured by itself can include multimodal information such as quantitative values ​​of screen resolution (such as 1080p, 2K, etc.), test data of display effects at different resolutions (such as color accuracy, clarity, etc.), and related display effect comparison images. The group category characteristics of "smartphone screen display technology" can include multimodal information such as the type of display technology (such as LCD, OLED), the development history of display technology (such as the development process from early simple liquid crystal display to the current high dynamic range display technology). When dividing the multimodal entity groups into core nodes by level, the server combines the display effect data in the "relationship between screen resolution and display effect" with the display technology type in the "smartphone screen display technology". For example, if it is OLED technology, it can have better color performance at high resolution. The server constructs the group category feature of "smartphone screen display technology" on this benchmark link based on this information. This group category feature can be a vector containing multimodal information such as display technology type, resolution value range, and display effect characteristics under this technology.Take the second link "Smartphone screen display technology" - "The advantages of OLED screens are high contrast and wide viewing angle" as an example. The highest-level multimodal entity is "The advantages of OLED screens are high contrast and wide viewing angle". Its own configured industry description information can include multimodal information such as the specific value of OLED screen contrast (such as 100000:1, etc.), the measurement data of the viewing angle range (such as 170 degrees horizontal viewing angle, etc.), and the OLED screen structure diagram. Combined with the group category feature of "smartphone screen display technology" according to the above method. For example, the contrast and viewing angle advantages of OLED screens are combined with the development trend of display technology (such as the trend of OLED gradually replacing LCD as the high-end mobile phone screen), to construct the group category feature of "smartphone screen display technology" on this benchmark link. This feature is also a vector containing multiple multimodal information. For the group category feature vector corresponding to the first link "Smartphone screen display technology" - "Relationship between screen resolution and display effect", the server divides it into weighted groups based on the local perceptron. The local perceptron may determine the weight based on pre-trained weights or based on the current learning situation. For example, the weight of the dimension of display technology type may be high because it has an important impact on the overall screen display technology; while the weight of display effect test data at some specific resolutions may be relatively low. Through this weighted grouping, the server obtains the feature corresponding to the multimodal entity instance "smartphone screen display technology" and the first link, which is the initial feature. This initial feature integrates the multimodal information in the link and is adjusted according to the weight, which can better reflect the characteristics of the multimodal entity instance under this link relationship. For the group category feature vector corresponding to the second link "smartphone screen display technology"-"the advantages of OLED screens are high contrast and wide viewing angle", weighted grouping is also performed based on the local perceptron. For example, the weight of the contrast and viewing angle data of OLED screens may be high because these are the advantages emphasized in this link; while the weight of the development history of display technology may be low. Through this weighted grouping, the server obtains the feature corresponding to the multimodal entity instance "smartphone screen display technology" and the second link, which is a significant feature. This notable feature highlights the key characteristics of the multimodal entity instance in this particular link, and can be better used for subsequent related operations of the multimodal industry question-answering large model, such as calculating matching coefficients, optimizing learning, etc.

[0059] In the embodiment of the present invention, each level of grouping in the hierarchical division of multimodal entity groups into core nodes is implemented in the following manner:

[0060] For any multimodal entity divided into target-level groups, obtaining industry description features of the multimodal entity according to industry description information of the multimodal entity;

[0061] For any multimodal entity divided into target-level groups, each adjacent multimodal entity at a higher level that is connected to the multimodal entity is determined as a reference multimodal entity, and a group category feature of the reference multimodal entity is obtained;

[0062] For any multimodal entity divided into target-level groups, according to the industry description feature of the multimodal entity and the group category feature of each benchmark multimodal entity, obtain the degree of association between the multimodal entity and each benchmark multimodal entity;

[0063] For any multimodal entity divided into target-level groups, linear superposition is performed on the group category features of each benchmark multimodal entity according to the degree of association corresponding to each benchmark multimodal entity, and the group category features of the multimodal entity are obtained based on the result of linear superposition and the industry description features of the multimodal entity.

[0064] In an embodiment of the present invention, for example, taking the multimodal entity instance "smartphone battery performance" in the electronic product manufacturing industry as an example, there are multimodal entities such as "battery capacity and battery life" in its link. For the multimodal entity "battery capacity and battery life", the server begins to divide the multimodal entity group to the core node by level. The server obtains its industry description characteristics based on the industry description information of the multimodal entity "battery capacity and battery life". Its industry description information may include specific values ​​of battery capacity of different models of smartphones (such as 3000mAh, 4000mAh, etc.), statistical data of battery life in various usage scenarios (such as continuous call duration, video playback duration, etc.), and multimodal information such as charts of the relationship between battery capacity and battery life. The server processes this information, for example, quantizes and encodes the battery capacity value and the battery life value, and converts the chart information into a part of the feature vector, thereby obtaining the industry description characteristics of the multimodal entity "battery capacity and battery life". This feature can be a comprehensive representation containing numerical, coded and vector elements, which can comprehensively describe the characteristics of the multimodal entity in terms of battery capacity and battery life. For the multimodal entity "battery capacity and battery life", the higher-level adjacent multimodal entity with a connection relationship with it is determined as the benchmark multimodal entity. In this example, assuming that the higher-level adjacent multimodal entity is "smartphone battery performance", then "smartphone battery performance" is the benchmark multimodal entity. The server obtains the group category feature of the benchmark multimodal entity "smartphone battery performance". Its group category feature can include multimodal information such as comprehensive evaluation indicators of battery performance (such as quantitative data such as battery energy density and charge and discharge efficiency), the importance weight of battery performance in the overall performance of smartphones (values ​​obtained through industry survey data), and technical development trends related to battery performance (such as text descriptions of the impact of new battery materials on performance improvement and related image information such as technology roadmaps). The server calculates the degree of association between them based on the industry description features of the multimodal entity "battery capacity and battery life" and the group category features of the benchmark multimodal entity "smartphone battery performance". For example, battery capacity and battery life directly affect battery performance, and the server can calculate the degree of correlation through a predefined algorithm. If the battery capacity is larger and the battery life is longer, the improvement in battery performance is more obvious, and the degree of correlation is higher. Suppose that it is calculated that the degree of correlation between "battery capacity and battery life" and "smartphone battery performance" is 0.8. This degree of correlation is the result of quantification based on the multimodal information of the two, and it reflects the close connection between the two. The server performs linear superposition on its group category features based on the degree of correlation (0.8) corresponding to the benchmark multimodal entity "smartphone battery performance".Assume that the group category feature vector of "smartphone battery performance" is (V=[v_1,v_2,v_3]) (where (v_1) represents the energy density value, (v_2) represents the charge and discharge efficiency value, (v_3) represents the coding of the technology development trend, etc.), and the result after linear superposition is (0.8V=[0.8v_1,0.8v_2,0.8v_3]). Then, based on the result of linear superposition and the industry description features of the multimodal entity "battery capacity and battery life", the group category features of the multimodal entity "battery capacity and battery life" are obtained. For example, the vector after linear superposition is combined or further calculated with the industry description feature vector of "battery capacity and battery life" (assuming that it is (U=[u_1,u_2,u_3]), where (u_1) represents the battery capacity value, (u_2) represents the battery life value, and (u_3) represents the coding of the relationship between battery capacity and battery life, etc.). It can be a simple vector addition (W=0.8V+U=[0.8v_1+u_1,0.8v_2+u_2,0.8v_3+u_3]), and this (W) is the new group category feature of the multimodal entity "battery capacity and battery life". This new group category feature combines its own industry description features and the relationship with the adjacent multimodal entities at a higher level, which can more accurately reflect its characteristics in the entire link, and with the continuous division of the levels, gradually construct accurate group category features towards the core node multimodal entity (such as "smartphone battery performance"), thereby providing more valuable information for the subsequent operation of the multimodal industry question-and-answer large model.

[0065] In an embodiment of the present invention, the method further includes:

[0066] Constructing a multi-relationship graph and obtaining industry description information of each multimodal entity in the multi-relationship graph, wherein the multi-relationship graph includes a plurality of multimodal entities and a connection relationship exists between two multimodal entities with a certain degree of association, wherein the plurality of multimodal entities include a plurality of multimodal question entities representing questions and a plurality of multimodal answer entities representing undetermined preferred answers; and the plurality of multimodal question entities include a target multimodal question entity representing a target question;

[0067] Acquire in the multi-relationship graph at least one third link with the target multimodal question entity as a core node multimodal entity, and at least one fourth link with the multimodal answer entity of the to-be-determined preferred answer as a core node multimodal entity;

[0068] Input each third link corresponding to the target multimodal question entity and the industry description information of the multimodal entity on each third link into the target multimodal industry question-answering big model to obtain the multimodal entity features of the target multimodal question entity, input each fourth link corresponding to each pending preferred answer and the industry description information of the multimodal entity on each fourth link into the target multimodal industry question-answering big model to obtain the multimodal entity features of each pending preferred answer;

[0069] If all the multimodal entities connected to the target multimodal question entity are multimodal answer entities, then each third link of the target question is obtained as a first link, and the multimodal entity features of the target multimodal question entity are converted through a target conversion network to obtain conversion features, and for each pending preferred answer, a matching coefficient between the target question and the pending preferred answer is obtained based on the conversion features and the multimodal entity features of the pending preferred answer;

[0070] According to the matching coefficient corresponding to each pending preferred answer, a target preferred answer recommended to the target question is obtained from each pending preferred answer.

[0071] In an embodiment of the present invention, exemplarily, the server starts to construct a multi-relationship graph, in which the multimodal entities include various contents related to smartphones. Multimodal question entities may include "how to improve the running speed of smartphones", "how to take high-quality photos with the camera of smartphones", etc. Multimodal answer entities include "increasing memory can improve the running speed, such as from 4GB to 8GB", "adjusting camera parameters, such as increasing ISO value can take better photos in low-light environments", etc. For the multimodal question entity "how to improve the running speed of smartphones", its industry description information may include knowledge of smartphone hardware architecture (such as text and diagram information on the working principles of CPU, memory, storage, etc.), the impact of software on running speed (multimodal information such as text and code examples such as resource management mechanisms of operating systems), and user demand for running speed (quantitative representation through user survey data and text information of user feedback), etc. For the multimodal answer entity "Increasing memory can improve running speed, such as from 4GB to 8GB", the industry description information can include the actual effect of memory upgrade on different models of mobile phones (such as statistical data of performance improvement percentage), cost-effectiveness analysis of memory cost and running speed improvement (through cost data and performance improvement data charts), and the technical principles of the impact of different memory types (such as LPDDR4, LPDDR5) on running speed (multimodal information such as text and schematic diagrams). Assume that the target multimodal question entity is "How to improve the running speed of smartphones". In the multi-relationship graph, the server obtains at least one third link with "How to improve the running speed of smartphones" as the core node multimodal entity. For example, a third link can be "How to improve the running speed of smartphones"-"The impact of CPU performance on running speed"-"Performance comparison of different CPU architectures". For pending preferred answers, assume that a pending preferred answer is "Optimizing mobile phone system software can improve running speed". The server obtains at least one fourth link with this pending preferred answer as the core node multimodal entity, such as "Optimizing mobile phone system software can improve running speed"-"Specific methods of software optimization, such as clearing cache"-"The impact of clearing cache on different software". The server inputs each third link (such as the link mentioned above) corresponding to the target multimodal question entity "How to improve the running speed of a smartphone" and the industry description information of the multimodal entity on each third link into the target multimodal industry question and answer big model. For example, for the third link "How to improve the running speed of a smartphone" - "The impact of CPU performance on running speed" - "Performance comparison of different CPU architectures", the question and answer big model processes the multimodal information contained therein, such as hardware architecture knowledge and performance impact relationships. Based on this input information, the question and answer big model outputs the multimodal entity features of the target multimodal question entity through the internal neural network structure (such as multi-layer perceptron for text information processing, convolutional neural network for image information processing and other multimodal processing mechanisms).This feature can be a vector representation that integrates multiple factors related to the improvement of running speed, such as hardware, software, and user needs, which includes multimodal elements such as quantitative values ​​(such as hardware performance weights) and text encodings (such as key description encodings of user needs). The server inputs each fourth link (such as the link mentioned above) corresponding to each pending preferred answer (such as "Optimizing mobile phone system software can improve running speed") and the industry description information of the multimodal entity on each fourth link into the target multimodal industry question and answer big model. For example, for the fourth link "Optimizing mobile phone system software can improve running speed"-"Specific methods of software optimization, such as clearing cache"-"The impact of clearing cache on different software", the question and answer big model processes the multimodal information such as software optimization methods and impact range. The question and answer big model outputs the multimodal entity features of each pending preferred answer. This feature is also a vector representation that integrates multiple factors related to software optimization, including the quantitative effect of software operation (such as the performance improvement value after clearing cache), text description encoding of software functions, and other multimodal elements. If all the multimodal entities that are connected to the target multimodal question entity "how to improve the running speed of a smartphone" are multimodal answer entities, and each third link of the target question is a first link. The server converts the multimodal entity features of the target multimodal question entity through the target conversion network to obtain conversion features. The target conversion network can be a neural network structure specifically used for feature conversion, which performs nonlinear transformation on multimodal entity features, for example, converting a vector containing multiple modal information into a feature vector that is more suitable for comparison. For each pending preferred answer (such as "optimizing mobile phone system software can improve running speed"), the server obtains the matching coefficient between the target question and the pending preferred answer based on the conversion features and the multimodal entity features of the pending preferred answer. For example, the matching coefficient can be obtained by calculating the cosine similarity between the feature vectors of the two. If the conversion feature vector is (A), the multimodal entity feature vector of the pending preferred answer is (B), and the matching coefficient (C=cos(A,B)). The server obtains the target preferred answer recommended to the target question from each pending preferred answer based on the matching coefficient corresponding to each pending preferred answer. Assume that there are multiple pending preferred answers, such as "Increasing memory can increase running speed, such as from 4GB to 8GB", "Optimizing mobile phone system software can increase running speed", "Closing unnecessary background applications can increase running speed", etc., and their corresponding matching coefficients are 0.6, 0.8, and 0.7 respectively. The server will select the pending preferred answer with the highest matching coefficient (here "Optimizing mobile phone system software can increase running speed") as the target preferred answer and recommend it to the target multimodal question entity "How to increase the running speed of a smartphone". In this way, through the construction of multi-relationship graphs, link information processing, application of the question-answering large model, and calculation of matching coefficients, the server can find the most appropriate answer for the target multimodal question entity.

[0072] In an embodiment of the present invention, it also includes:

[0073] If the multimodal entities that are connected to the target multimodal question entity include a multimodal answer entity and a multimodal question entity, then each third link of the target question is obtained, including the first link and the second link. For each pending preferred answer, the matching coefficient between the target question and the pending preferred answer is obtained based on the multimodal entity features of the target question and the pending preferred answer.

[0074] In the embodiment of the present invention, illustratively, it is assumed that the target multimodal question entity is "how to improve the energy efficiency of the smart home system". In the multi-relationship graph, the multimodal entities that have a connection relationship with this target multimodal question entity include multimodal answer entities and multimodal question entities. For example, the multimodal answer entity may have "using energy-saving devices, such as low-power smart sensors", and the multimodal question entity may have "which smart home devices have high power consumption", etc. The server obtains each third link of the target question "how to improve the energy efficiency of the smart home system", and these third links include the first link and the second link. A possible third link is: "how to improve the energy efficiency of the smart home system"-"which smart home devices have high power consumption" (this is the first link, because the adjacent multimodal entity is the question entity)-"power consumption of the smart lighting system" (this is still the first link)-"using energy-saving devices, such as low-power smart sensors" (this is the second link, because the adjacent multimodal entity is the answer entity). The server inputs each third link corresponding to the target multimodal question entity "how to improve the energy efficiency of the smart home system" and the industry description information of the multimodal entity on each third link into the target multimodal industry question and answer big model. For multimodal question entities in the third link, such as "Which smart home devices have high power consumption", its industry description information can include power consumption statistics of various smart home devices (smart speakers, smart cameras, smart sockets, etc.) (presenting the power consumption values ​​of different devices in different working modes in the form of charts), power consumption test standards (such as text descriptions of specific smart home device power consumption test specifications), and the proportion of power consumption of different devices in the energy consumption of the entire smart home system (represented in the form of pie charts, etc.). For multimodal answer entities in the third link, such as "Use energy-saving devices, such as low-power smart sensors", its industry description information can include power consumption values ​​of different types of smart sensors (temperature sensors, light sensors, etc.) (listing the power consumption of sensors of different brands and models in a table), energy-saving effect evaluation methods (such as comparing the reduction ratio of energy consumption of the entire smart home system before and after using energy-saving sensors) and the working principle of energy-saving sensors (multimodal information combining schematic diagrams and text descriptions). Based on these rich multimodal information, the question-answering model outputs the multimodal entity features of the target multimodal question entity. This feature can be a vector representation that integrates the current status of smart home system energy consumption, device power consumption, energy-saving methods, etc. For example, the vector can contain multimodal elements such as the power consumption weights of different devices (based on their importance in the system and actual power consumption), quantitative indicators of energy-saving potential (such as the expected percentage of energy consumption reduction), and key text description encodings related to energy efficiency.For each pending preferred answer, the industry description information of each corresponding fourth link and the multimodal entity on each fourth link is also input into the target multimodal industry question-answering model. Assume that a pending preferred answer is "Optimizing the working mode of the equipment can improve energy efficiency, such as the timer switch function of the intelligent lighting system". The multimodal entity industry description information in its fourth link includes energy consumption data of different working modes of the intelligent lighting system (such as always on, automatically adjusting the brightness according to the light, timer switch, etc.) (using a line graph to represent the energy consumption changes of different modes within a day), technical implementation of working mode switching (such as through the text description of the smart home control protocol and the relevant control flow diagram) and the impact of different working modes on user experience (using multimodal information such as user feedback text and rating data).

[0075] The question-answering model outputs the multimodal entity features of the pending preferred answer based on this information. This feature is also a vector representation that integrates multiple factors such as the working mode, energy consumption, and user experience of the intelligent lighting system, including the quantitative value of the impact of working mode switching on energy consumption, and multimodal elements such as text description encoding related to user experience. The server obtains the matching coefficient between the target question and the pending preferred answer based on the multimodal entity features of the target question and the pending preferred answer. A variety of methods can be used to calculate the matching coefficient, such as a method based on a vector space model. The cosine similarity between the two can be calculated as the matching coefficient. In addition to cosine similarity, other distance measurement methods, such as Euclidean distance, can also be used, and then converted into a matching coefficient based on the distance measurement result (for example, the smaller the distance, the higher the matching coefficient). In this way, the server can accurately evaluate the degree of matching between the target question and the pending preferred answer, so as to provide the user with the most appropriate answer in subsequent operations or perform further optimization learning and other operations.

[0076] In an embodiment of the present invention, the multimodal entity instance is obtained in the following manner, including:

[0077] For the target industry field, obtain the benchmark answer content that has been marked with the target value, and obtain the unmarked pending answer content;

[0078] Combining the respective benchmark answer contents with the original multimodal questions obtained for the target industry field to generate corresponding original question description contents, and outputting the inferred target values ​​corresponding to the respective original question description contents through the first integrated model, and obtaining the analysis results of the first integrated model in the target industry field based on the inferred target values ​​and the corresponding description target values;

[0079] When the analysis result does not meet the preset requirements, each pre-configured content added to the training description is added to the original multimodal problem to generate a corresponding pending multimodal problem, and in each pending multimodal problem, a target multimodal problem that makes the analysis result of the first integrated model meet the preset requirements is confirmed;

[0080] Each pending answer content is combined with the target multimodal question to generate a corresponding question description content instance, and through the first integrated model, the target value instance corresponding to each question description content instance is output respectively, and a multimodal entity instance set is constructed based on the question description content instances and the corresponding target value instances.

[0081] In an embodiment of the present invention, exemplarily, in the e-commerce industry, the target may be an answer to a question related to product evaluation. The server obtains various benchmark answer contents that have been marked with description target values ​​for this target industry field. For example, for the product evaluation question "How comfortable is a certain brand of sports shoes?", the benchmark answer content may be "This pair of sports shoes is very comfortable, the sole is soft, and the upper is breathable", and the marked description target value is "positive evaluation". Another benchmark answer content may be "This pair of shoes is not very comfortable to wear, and my feet hurt after walking for a long time", and the marked description target value is "negative evaluation". These benchmark answer contents are contents that have been marked with clear description target values ​​by manual or other reliable methods, and can be used for model evaluation and training. At the same time, the server obtains various unmarked pending answer contents. On the e-commerce platform, these may be newly collected user reviews, but they have not yet been classified and marked. For example, "The shoes are not bad, but a little tight" or "The design of this pair of shoes is very fashionable, and it feels good to wear". The server combines each benchmark answer content with the original multimodal question obtained for the e-commerce industry to generate the corresponding original question description content. For example, the original multimodal question is "Please evaluate a certain brand of sports shoes". After being combined with the benchmark answer content "This sports shoe is very comfortable, the sole is soft, and the upper is breathable", the original question description content becomes "Please evaluate a certain brand of sports shoes: This sports shoe is very comfortable, the sole is soft, and the upper is breathable". Through the first integrated model (this model can be a classification model based on a machine learning algorithm, which can classify question-answer combinations), the inferred target values ​​corresponding to each original question description content are output respectively. Assume that the target value inferred by the first integrated model for the original question description content "Please evaluate a certain brand of sports shoes: This sports shoe is very comfortable, the sole is soft, and the upper is breathable" is "positive evaluation". Then, based on each inferred target value and the corresponding description target value, the server obtains the analysis result of the first integrated model in the field of e-commerce. For example, the performance of the model can be evaluated by calculating indicators such as accuracy and recall. If the proportion of correctly inferred evaluations (the inferred target value is consistent with the description target value) in a set of test data accounts for a certain threshold (such as 80%) of the total test data, the model performance is considered to be good; if it is lower than this threshold, the analysis result is considered to fail to meet the preset requirements. When the analysis results do not meet the preset requirements, the server adds the pre-configured training description content to the original multimodal problem to generate the corresponding pending multimodal problem. For example, the training description content can be more attribute descriptions about sports shoes, such as "Please evaluate a certain brand of sports shoes from the aspects of sole elasticity, upper material and shoe shape", which generates a pending multimodal problem. Among the pending multimodal problems, the server confirms the target multimodal problem that makes the analysis results of the first integrated model meet the preset requirements.Through continuous adjustment and testing, a pending multimodal question is found, so that the first integrated model can improve the matching accuracy of the inferred target value and the description target value of the benchmark answer content under this new question form to a level that meets the requirements. For example, when the pending multimodal question "Please evaluate a certain brand of sports shoes from the aspects of sole elasticity, upper material and shoe shape" is used, the accuracy of the model reaches 85%, then this pending multimodal question is determined as the target multimodal question. The server combines each pending answer content with the target multimodal question to generate the corresponding problem description content instance. For example, for the pending answer content "The shoes are not bad, but a little tight" and the target multimodal question "Please evaluate a certain brand of sports shoes from the aspects of sole elasticity, upper material and shoe shape", the combined problem description content instance is "Please evaluate a certain brand of sports shoes from the aspects of sole elasticity, upper material and shoe shape: the shoes are not bad, but a little tight". Through the first integrated model, the target value instances corresponding to each problem description content instance are output respectively. Assume that for the above problem description content instance, the target value instance output by the first integrated model is "neutral evaluation". Finally, based on each question description content instance and the corresponding target value instance, the server constructs a multimodal entity instance set. Each element in this set is a multimodal entity instance, including a question description content instance (including text and possibly related images, such as product images and other multimodal information) and a target value instance (evaluation classification result). This multimodal entity instance set will be used for subsequent multimodal industry question-answering model training and optimization operations to improve the model's accuracy and classification capabilities in answering questions related to product evaluation in the e-commerce industry.

[0082] In the embodiment of the present invention, the step of obtaining unmarked pending answer contents includes:

[0083] Obtain the unmarked initial answer content;

[0084] According to the matching coefficient between every two initial answer contents, the initial answer contents meeting similar conditions are clustered to obtain initial answer content groups, and one initial answer content is randomly confirmed in each initial answer content group to be determined as the pending answer content.

[0085] In an embodiment of the present invention, exemplarily, in the online education industry, the server obtains unlabeled initial answer content from various channels. These channels may include student feedback from online courses, discussions in forums, and comments on online education courses on social media. For example, student comments on a mathematics course collected from an online course platform: "The explanation of this mathematics course is very clear, but there are a few exercises", "I think the teacher of this mathematics course is very patient, but the course progress is a bit fast", "The content of this mathematics course is very practical, but the exam review materials are not enough", etc. These are all unlabeled initial answer contents, which contain students' views on different aspects of online education courses. The server performs subsequent operations based on the matching coefficient between each two initial answer contents. The method for calculating the matching coefficient can be based on the similarity calculation of text content. For example, for the two initial answer contents "The explanation of this mathematics course is very clear, but there are a few exercises" and "I think the content of this mathematics course is very practical, but the exam review materials are not enough", the server will decompose them into vector representations of words or phrases. Bag-of-Words or more complex word vector models (such as Word2Vec) can be used to represent these texts. Then, the matching coefficient is determined by calculating the cosine similarity between the vectors. Assume that the matching coefficient between the two answer contents is 0.3, indicating that they have a certain degree of similarity. The server clusters the initial answer contents that meet the similarity conditions according to these matching coefficients to obtain each initial answer content group. For example, if the similarity condition is set to a matching coefficient greater than 0.2, then for the initial answer contents "The explanation of this math course is very clear, but there are a few exercises", "The content of this math course is very practical, but the exam review materials are not enough", and "The knowledge points of this course are explained very well, but the homework assignments are not very reasonable", the matching coefficients between them are all greater than 0.2, so they will be clustered into an initial answer content group. And "I think the teacher of this math course is very patient, but the course progress is a bit fast" may be divided into another initial answer content group because the matching coefficient with the previous answer content is lower than 0.2. After obtaining each initial answer content group, the server randomly confirms an initial answer content in each initial answer content group and determines it as the pending answer content. For example, for the group of answers mentioned above that includes "The explanation of this math course is very clear, but there are not enough exercises", the server randomly selects "The content of this math course is very practical, but the exam review materials are not enough" as the pending answer content. For other initial answer content groups, random selection is performed in the same way. The pending answer content determined in this way will be used for subsequent combination with the target multimodal question to generate a question description content instance, and then participate in related operations of the multimodal industry question and answer large model, such as model training and optimization.This method can reduce the amount of data to a certain extent, while retaining the representativeness of different types of answer content and improving the efficiency and accuracy of model processing.

[0086] In an embodiment of the present invention, after outputting the target value instances corresponding to the problem description content instances respectively, and before constructing a multimodal entity instance set based on the problem description content instances and the corresponding target value instances, the method further includes:

[0087] When there are other initial answer contents except one pending answer content in the initial answer content group to be obtained, the other initial answer contents are respectively combined with the target multimodal question to generate a corresponding question description content instance, and the target value instance obtained corresponding to the one pending answer content is respectively determined as the target value instance of each question description content instance generated corresponding to the initial answer content group.

[0088] In an embodiment of the present invention, illustratively, in the tourism industry, it is assumed that an initial answer content group is obtained by clustering the matching coefficients. The initial answer contents in this group are all about the tourists' feedback on a certain tourist attraction. For example, this initial answer content group contains the following initial answer contents: "The scenery of this attraction is beautiful, but the transportation is not very convenient." "The scenery is really good, but there are too few dining options around." "The scenery is beautiful, but the accommodation is a bit far from the attraction." According to the previous steps, a pending answer content is randomly determined from this group, assuming that it is "The scenery of this attraction is beautiful, but the transportation is not very convenient." The target multimodal question can be "Please comprehensively evaluate the experience of playing at this tourist attraction." The server obtains the other initial answer contents in the initial answer content group except for the pending answer content "The scenery of this attraction is beautiful, but the transportation is not very convenient.", that is, "The scenery is really good, but there are too few dining options around." and "The scenery is beautiful, but the accommodation is a bit far from the attraction." The server combines these other initial answer contents with the target multimodal question respectively to generate corresponding question description content instances. After combining "The scenery is really good, but there are too few dining options around." with the target multimodal question "Please give a comprehensive evaluation of the experience of visiting this tourist attraction", the obtained problem description content instance is "Please give a comprehensive evaluation of the experience of visiting this tourist attraction: The scenery is really good, but there are too few dining options around." After combining "The scenery is beautiful, but the accommodation is a bit far from the attraction." with the target multimodal question, the obtained problem description content instance is "Please give a comprehensive evaluation of the experience of visiting this tourist attraction: The scenery is beautiful, but the accommodation is a bit far from the attraction." Previously, for the pending answer content "The scenery of this attraction is beautiful, but the transportation is not very convenient.", a target value instance was output through the first integrated model, assuming that this target value instance is "medium evaluation". The server determines the target value instance "medium evaluation" obtained for the pending answer content as the target value instance of each problem description content instance generated by the initial answer content group (i.e., "Please give a comprehensive evaluation of the experience of visiting this tourist attraction: The scenery is really good, but there are too few dining options around." and "Please give a comprehensive evaluation of the experience of visiting this tourist attraction: The scenery is beautiful, but the accommodation is a bit far from the attraction."). The purpose of doing this is based on the assumption that the answer content in the same initial answer content group has a certain similarity. Since the target value instance has been determined for a pending answer content in the group, it can be considered to a certain extent that other similar answer contents in this group also have the same target value instance. This helps to more comprehensively cover answer contents with different expressions but similar properties when constructing a set of multimodal entity instances, while reducing the amount of computation required to infer the target value for each answer content separately, thereby improving the efficiency of the entire multimodal industry question and answer model construction process. Based on previous operations, the server already has multiple question description content instances and corresponding target value instances.For example, in addition to the content in the above example about tourist attractions, there may be other question description content instances and target value instances about different tourist attractions or different aspects of the same tourist attraction. These contents together constitute a set of multimodal entity instances. In this set, each multimodal entity instance contains a question description content instance (which may contain text descriptions, or multimodal information such as pictures and videos related to tourism, such as pictures of attractions, videos of tourists playing, etc.) and a target value instance (such as positive reviews, medium reviews, negative reviews, etc.). This set of multimodal entity instances will be used for subsequent training, optimization and other operations of the multimodal industry question-and-answer large model, so that the model can better understand the various question-and-answer relationships in the tourism industry and improve the accuracy of answers to tourism-related questions.

[0089] In the embodiment of the present invention, the step of obtaining unmarked pending answer contents includes:

[0090] Obtaining each unmarked original answer content, and obtaining each core content and the corresponding similar identifier and similar content identifier; the similar identifier and similar content identifier corresponding to a core content is used to associate the associated content corresponding to the core content;

[0091] Based on the operation of replacing the corresponding core content with the associated content, the original answer content including the core content is processed respectively to obtain each enhanced answer content after processing;

[0092] The original answer contents and the enhanced answer contents are regarded as pending answer contents.

[0093] In an embodiment of the present invention, exemplarily, in the catering industry, the server obtains unmarked original answer content from various sources. These sources can be food review websites, online evaluation platforms of restaurants, or discussions about catering experiences on social media. For example, the original answer content may include: "The steak in this restaurant is delicious, but the side dishes are a bit small", "I like the pasta in this restaurant, but the sauce is a bit light", "The environment of this restaurant is very good, but the service is a bit slow", etc. At the same time, the server obtains each core content and the corresponding similar identifier (similar content identifier). In the context of catering evaluation, the core content can be the name of the dish, service attributes, environmental characteristics, etc. For example, for "the steak in this restaurant is delicious, but the side dishes are a little less", "steak" is a core content, and its similar identifiers may be associated with other similar main dishes, such as "pork chop" and "lamb chop"; for "I like the pasta in this restaurant, but the sauce is a little light", "pasta" is the core content, and its similar identifiers may be associated with other pasta, such as "macaroni" and "spiral noodles"; for "the environment of this restaurant is very good, but the service is a little slow", "environment" and "service" are core contents, and the similar identifiers of "environment" may be associated with similar contents such as the restaurant's decoration style and seating layout, and the similar identifiers of "service" may be associated with related contents such as the waiter's attitude and serving speed. Based on the operation of replacing the corresponding core content with the associated content, the server processes the original answer content containing the core content respectively, and obtains the enhanced answer content after processing. For "the steak in this restaurant is delicious, but the side dishes are a little less", according to the similar identifier of "steak", "steak" is replaced with "pork chop" to obtain the enhanced answer content "the pork chop in this restaurant is delicious, but the side dishes are a little less". Similarly, replacing "steak" with "lamb chops" can also get enhanced answers such as "The lamb chops in this restaurant are delicious, but the side dishes are a bit small." For "I like the pasta in this restaurant, but the sauce is a bit light," replace "pasta" with "macaroni" to get "I like the macaroni in this restaurant, but the sauce is a bit light"; replace "pasta" with "spirulina" to get enhanced answers such as "I like the spiral noodles in this restaurant, but the sauce is a bit light." For "The environment of this restaurant is very good, but the service is a bit slow," if the similar identifier of "environment" contains "decoration style," replace "environment" with "decoration style" to get enhanced answers such as "The decoration style of this restaurant is very good, but the service is a bit slow." The server uses each original answer content and each enhanced answer content as each pending answer content.In this example of the catering industry, the pending answers include the original evaluation, such as "The steak in this restaurant is delicious, but the side dishes are a little less", "I like the pasta in this restaurant, but the sauce is a little light", "The environment of this restaurant is very good, but the service is a little slow", etc., as well as the enhanced answer content obtained by replacing the core content, such as "The pork chop in this restaurant is delicious, but the side dishes are a little less", "I like the macaroni in this restaurant, but the sauce is a little light", "The decoration style of this restaurant is very good, but the service is a little slow", etc. These pending answers will be used for subsequent operations such as combining with the target multimodal question to generate question description content instances. For example, the target multimodal question can be "Please give a comprehensive evaluation of this restaurant". After these pending answers are combined with the target multimodal question, the target value instance will be output through the first integrated model, and then a multimodal entity instance set will be constructed, providing a data basis for operations such as training of the multimodal industry question and answer large model. This method increases the diversity of data, enables the model to better handle various possible expressions, and improves the accuracy of question and answer.

[0094] In the embodiment of the present invention, the operation based on replacing the corresponding core content with the associated content processes the original answer content including the core content to obtain each enhanced answer content after processing, including:

[0095] Based on the core contents, matching and identifying are performed in an original answer content;

[0096] When obtaining the matching target core content based on the recognition result, obtaining the corresponding similar core contents based on the similar identifier corresponding to the target core content, and replacing the target core content in the original answer content with the similar core contents to obtain the pending answer contents;

[0097] The content matching coefficients between each of the pending answer contents and the one original answer content are calculated respectively, and the pending answer contents whose content matching coefficients meet the selection conditions are used as enhanced answer contents.

[0098] In an embodiment of the present invention, illustratively, in the electronic product industry, it is assumed that the server obtains an original answer content: "The camera of this smart phone has high pixels and good photo effects, but the battery life is not ideal." At the same time, the core content obtained by the server may include "camera", "battery", etc. These core contents are determined from the perspective of different functional components of electronic products, and each core content has a corresponding similar identifier. For example, the similar identifiers of "camera" may include "lens", "image sensor", etc., and the similar identifiers of "battery" may include "power supply", "energy storage device", etc. The server performs matching and identification in this original answer content based on these core contents. For the core content "camera", the server will search in the original answer content whether there is a matching word. In this example, "camera" appears directly in the original answer content, so it is identified as the matching target core content. Similarly, such a matching and identification operation is also performed for "battery". When the matching target core content "camera" is obtained based on the identification result, the server obtains the corresponding similar core contents, namely "lens" and "image sensor", based on the similar identifiers corresponding to "camera". The server replaces the target core content "camera" in the original answer content with these similar core contents respectively to obtain each pending answer content. After replacing "camera" with "lens", the pending answer content is obtained: "The lens of this smartphone has high pixels and takes good photos, but the battery life is not ideal." After replacing "camera" with "image sensor", the pending answer content is obtained: "The image sensor of this smartphone has high pixels and takes good photos, but the battery life is not ideal." The server calculates the content matching coefficient between each pending answer content and the original answer content. There are many methods to calculate the content matching coefficient, such as the word vector-based method. For the original answer content "The camera of this smartphone has high pixels and takes good photos, but the battery life is not ideal." and the pending answer content "The lens of this smartphone has high pixels and takes good photos, but the battery life is not ideal." First, convert the two sentences into word vector representations. Assume that a pre-trained word vector model is used to map each word in the sentence to a vector space of fixed dimension. Then, the content matching coefficient can be obtained by calculating the cosine similarity between the vectors. For example, after calculation, the content matching coefficient between the two sentences is 0.8. For the original answer and the pending answer "This smartphone has a high-pixel image sensor and a good photo effect, but the battery life is not ideal.", after the same calculation, the content matching coefficient is 0.7. Assume that the selection condition is that the content matching coefficient is greater than 0.6. Since 0.8 and 0.7 are both greater than 0.6, the two pending answers "This smartphone has a high-pixel image sensor and a good photo effect, but the battery life is not ideal." and "This smartphone has a high-pixel image sensor and takes great photos, but its battery life is less than ideal." are all selected as enhanced answer content. These enhanced answer contents will be used together with the original answer content as pending answer content, and will be used in subsequent combination with the target multimodal question to generate question description content instances, and then participate in the construction and optimization process of the multimodal industry question and answer big model. This method of replacing the target core content with similar core content and screening the enhanced answer content increases the diversity and richness of the data, and helps to improve the multimodal industry question and answer big model's ability to understand and process different expressions.

[0099] In the embodiment of the present invention, the respective similar core contents are used to replace the target core content in the original answer content to obtain the enhanced answer content, including:

[0100] Replacing the target core content in the original answer content with the respective similar core contents to obtain respective pending answer contents;

[0101] The content matching coefficients between each of the pending answer contents and the one original answer content are calculated respectively, and the pending answer contents whose content matching coefficients meet the selection conditions are used as enhanced answer contents.

[0102] In the embodiment of the present invention, for example, assume that the server obtains an original answer content about a car: "The engine of this car is very powerful, but the fuel consumption is a bit high." In the automobile industry, the core content may include "engine" and the like. Similar identifiers corresponding to "engine" may include "engine", "power unit" and the like. The server replaces the target core content "engine" in the original answer content with the similar core content. When "engine" is replaced by "engine", the pending answer is obtained: "The engine of this car is very powerful, but the fuel consumption is a bit high." When "power unit" is replaced by "engine", the pending answer is obtained: "The power unit of this car is very powerful, but the fuel consumption is a bit high." For the original answer "The engine of this car is very powerful, but the fuel consumption is a bit high." and the pending answer "The engine of this car is very powerful, but the fuel consumption is a bit high." First, the server preprocesses the sentence, such as word segmentation, to obtain word sequences such as "this" car's" engine" is "powerful", "but" the fuel consumption is "a bit" high" and "this" car's" engine" is "powerful", "but" the fuel consumption is "a bit" high. Then, a pre-trained word vector model (such as Word2Vec) is used to map each word to a vector space of a fixed dimension (for example, 300 dimensions). For each word vector in the sentence, it can be combined into a sentence vector by summing, averaging, etc. Assume that the sentence vectors of the original answer content and the pending answer content are obtained by averaging. Finally, the cosine similarity between the two sentence vectors is calculated to obtain the content matching coefficient. Assume that after calculation, the content matching coefficient is 0.9. The same operation is performed for the original answer content and the pending answer content "The power unit of this car is very powerful, but the fuel consumption is a bit high." After word segmentation, word vector mapping, and sentence vector calculation, the content matching coefficient is 0.7. Assume that the selection condition is that the content matching coefficient is greater than 0.6. Since both 0.9 and 0.7 are greater than 0.6, the pending answer contents "The engine of this car is very powerful, but the fuel consumption is a bit high." and "The power unit of this car is very powerful, but the fuel consumption is a bit high." are both used as enhanced answer content. This method replaces the target core content in the original answer content by introducing similar core content, and then selects the enhanced answer content that meets the conditions according to the content matching coefficient. It helps to enrich the way data is expressed. In the multimodal industry question-answering model, it enables the model to better understand the semantic content under different expressions, thereby improving the accuracy and robustness of the model's answers to various automotive-related questions. For example, when answering questions about the automotive power system, the model can accurately associate and process the content of different expressions such as "engine", "engine", and "power unit".

[0103] In the embodiment of the present invention, the step of determining, among the pending multimodal problems, a target multimodal problem that makes the analysis result of the first integrated model meet the preset requirements includes:

[0104] Combining each of the benchmark answer contents with an undetermined multimodal question to generate corresponding undetermined question description contents, and outputting the inferred target values ​​corresponding to each of the undetermined question description contents through the first integrated model, and obtaining the analysis results of the first integrated model in the target industry field based on each inferred target value and the corresponding description target value;

[0105] Among the analysis results, a target analysis result that meets the preset requirements is selected, and the undetermined multimodal problem corresponding to the target analysis result is used as the target multimodal problem.

[0106] In the embodiment of the present invention, for example, in the clothing industry, it is assumed that some benchmark answer contents have been obtained, such as: "The style of this dress is very fashionable, but the color is a bit dark." "The material of these pants is very comfortable, but the version is a bit loose." At the same time, there are some pending multimodal questions, such as "Evaluate this dress from the aspects of style, color, material and version." The server combines each benchmark answer content with the pending multimodal question to generate corresponding pending question description content. For the baseline answer content "The style of this dress is very fashionable, but the color is a bit dark." After combining it with the pending multimodal question "Evaluate this dress from the aspects of style, color, material and version", the description of the pending question is obtained: "Evaluate this dress from the aspects of style, color, material and version: The style of this dress is very fashionable, but the color is a bit dark." For the baseline answer content "The material of these pants is very comfortable, but the version is a bit loose." After combining it with the pending multimodal question, the description of the pending question is obtained: "Evaluate this dress from the aspects of style, color, material and version: The material of these pants is very comfortable, but the version is a bit loose." (Here it is assumed that in the context of the overall evaluation of clothing, the evaluation of pants can also be used for the evaluation-related operations of this clothing). Through the first integrated model (this model can be a text classification model based on deep learning, which has been pre-trained and can be fine-tuned for the clothing industry), the inferred target values ​​corresponding to the descriptions of each pending question are output respectively. Assume that the inferred target value inferred by the first integrated model for the description of the pending question "Evaluate this dress from the aspects of style, color, material and version: The style of this dress is very fashionable, but the color is a bit dark." is "medium evaluation". For the description of the pending question "Evaluate this dress from the aspects of style, color, material and version: The material of these pants is very comfortable, but the version is a bit loose." The inferred target value inferred is "positive evaluation". Then, based on each inferred target value and the corresponding description target value (assuming that the description target values ​​of these benchmark answer contents that were previously manually marked as "medium evaluation" and "positive evaluation" respectively), the server obtains the analysis results of the first integrated model in the field of clothing industry. This analysis result can be measured by calculating indicators such as accuracy and recall. For example, if there are 10 such pending question descriptions, of which 8 inferred target values ​​are consistent with the description target value, then the accuracy is 80%. Assume that the preset requirement is that the accuracy reaches more than 85%. If the accuracy of the first integrated model does not meet this requirement in the above calculation, the server will continue to try other pending multimodal questions. For example, there is another pending multimodal question, "Evaluate this dress in terms of fashion sense, color matching, fabric comfort, and tailoring." The server again combines the baseline answer content with this pending multimodal question to generate a new pending question description content, obtains a new inference target value through the first integrated model, and calculates new analysis results (accuracy and other indicators).Among the analysis results, once the target analysis result that meets the preset requirements (accuracy of more than 85%) is found. Assume that when using the pending multimodal question "Evaluate this dress from the aspects of fashion sense, color matching, fabric comfort and tailoring", the accuracy obtained by calculation is 88%, which meets the preset requirements. The server will use the pending multimodal question corresponding to this target analysis result, that is, "Evaluate this dress from the aspects of fashion sense, color matching, fabric comfort and tailoring", as the target multimodal question. This target multimodal question will be used to combine with the pending answer content to generate a question description content instance, and then participate in the construction of a multimodal entity instance set and other operations to support the construction and optimization of the multimodal industry question and answer large model, and improve the accuracy of the model's answers to questions related to the clothing industry.

[0107] In an embodiment of the present invention, constructing a multimodal entity instance set based on each problem description content instance and the corresponding target value instance includes:

[0108] Obtaining a target value instance corresponding to a question description content instance, and obtaining each similar target value including the target value instance based on a pre-configured similar target value identifier, and combining each similar target value with the question description content instance to obtain each multimodal entity instance; the similar target value identifier is used to associate other target values ​​similar to a target value;

[0109] The obtained multimodal entity instances are combined to obtain a multimodal entity instance set.

[0110] In an embodiment of the present invention, illustratively, in the hotel industry, it is assumed that there is already a problem description content instance: "Please evaluate the service of this hotel: the waiter's attitude is very enthusiastic, but the check-in procedure is a bit slow." And its corresponding target value instance is "medium evaluation". The server pre-configures a similar target value identifier for associating other target values ​​similar to a target value. In the context of hotel evaluation, the similar target value identifier may associate "medium evaluation" with "slightly below average evaluation", "close to average evaluation", etc. Therefore, each similar target value containing "medium evaluation" is "slightly below average evaluation" and "close to average evaluation". The server combines these similar target values ​​with this problem description content instance respectively to obtain each multimodal entity instance. When "slightly below average evaluation" is combined with the problem description content instance "Please evaluate the service of this hotel: the waiter's attitude is very enthusiastic, but the check-in procedure is a bit slow.", a multimodal entity instance is obtained: ("Please evaluate the service of this hotel: the waiter's attitude is very enthusiastic, but the check-in procedure is a bit slow.", "slightly below average evaluation"). When "close to the average evaluation" is combined with the problem description content instance, a multimodal entity instance is obtained: ("Please evaluate the service of this hotel: the waiter is very enthusiastic, but the check-in procedure is a bit slow.", "Close to the average evaluation"). The server combines the obtained multimodal entity instances to obtain a set of multimodal entity instances. Assume that in addition to the above-mentioned multimodal entity instances about hotel service evaluation, there are other multimodal entity instances constructed by problem description content instances about hotels and their corresponding target value instances. For example, for the problem description content instance "Please evaluate the room of this hotel: the room is clean, but the space is a bit small." The corresponding target value instance is "slightly below the average evaluation". The similar target values ​​"close to the average evaluation" and "medium evaluation" are obtained through the similar target value identifier, and the constructed multimodal entity instances are ("Please evaluate the room of this hotel: the room is clean, but the space is a bit small.", "Close to the average evaluation") and ("Please evaluate the room of this hotel: the room is clean, but the space is a bit small.", "medium evaluation"). All these multimodal entity instances combined together constitute a set of multimodal entity instances. Each element in this set is a multimodal entity instance, which includes a question description content instance (which can contain multimodal information such as text, pictures, etc., such as photos of hotel rooms, etc.) and a target value instance (evaluation result). This multimodal entity instance set will be used for subsequent multimodal industry question-answering large model training, optimization and other operations. For example, during the training process, the model can learn the relationship between different question descriptions and different evaluation results based on the data in the multimodal entity instance set, thereby improving the accuracy and rationality of answers to hotel industry-related questions.

[0111] In the embodiment of the present invention, obtaining the analysis result of the first integrated model in the target industry field based on each inference target value and the corresponding description target value includes:

[0112] Based on each inferred target value and the corresponding descriptive target value, the precision corresponding to the first integrated model is calculated, and the recall rate of the first integrated model under various descriptive target values ​​in the target industry field is calculated; the average recall rate obtained based on each recall rate and the precision rate are used as the analysis results of the first integrated model in the target industry field.

[0113] In an embodiment of the present invention, illustratively, in the field of financial investment, it is assumed that there are a number of benchmark answer contents that have been marked with description target values. For example, for the question "Is a certain stock worth investing in?", the benchmark answer contents may include "This stock has a good financial situation, an optimistic industry outlook, and is worth investing" (the description target value is "positive evaluation-worth investing"), "This stock has unstable performance, high risk, and is not worth investing" (the description target value is "negative evaluation-not worth investing"), etc. The server combines these benchmark answer contents with the original multimodal question and obtains each inferred target value through the first integrated model. Assume that there are 100 such combinations, of which 80 combinations have inferred target values ​​that are consistent with the corresponding description target values. The calculation formula for the accuracy is: accuracy = number of correctly predicted samples (number of samples whose inferred target values ​​are consistent with the description target values) / number of predicted samples (here is the total number of combinations). Therefore, accuracy = 80 / 100 = 0.8 or 80%. This means that the first integrated model has an accuracy of 80% when predicting the target values ​​of these combinations. Continuing with the example of the financial investment field, the target values ​​may be described in categories such as "positive evaluation-worth investing", "negative evaluation-not worth investing" and "neutral evaluation-wait and see". For the "positive evaluation-worth investing" category: assume that among all the benchmark answers that are actually "positive evaluation-worth investing" (assuming there are 50 in total), the first integrated model correctly inferred 40 of them. The calculation formula for the recall rate is: recall rate = the number of samples of this category that are correctly predicted / the actual number of samples of this category. Therefore, the recall rate for the "positive evaluation-worth investing" category = 40 / 50 = 0.8 or 80%. For the "negative evaluation-not worth investing" category: assume that there are actually 30 such benchmark answers, and the model correctly inferred 20 of them. Then the recall rate = 20 / 30≈0.67 or 67%. For the "neutral evaluation-wait and see" category: assume that there are actually 20, and the model correctly inferred 15 of them. Recall rate = 15 / 20 = 0.75 or 75%. The average recall is calculated by summing the recalls under each type of descriptive target value and dividing it by the number of categories. In this example, the average recall = (0.8 + 0.67 + 0.75) / 3 ≈ 0.74 or 74%. The server will use the average recall (74%) obtained based on each recall and the previously calculated precision (80%) as the analysis result of the first integrated model in the financial investment field. This analysis result can comprehensively evaluate the performance of the first integrated model in this target industry field. The precision reflects the proportion of correct predictions made by the model, while the average recall reflects the model's coverage ability under different target value categories.If both the precision and average recall are high, it means that the model performs well in question-answering in the field of financial investment. If one or both of these indicators are low, the model may need to be adjusted, such as adding training data, optimizing the model structure, etc., to improve the accuracy and comprehensiveness of the model in multimodal question-answering in the field of financial investment.

[0114] In the embodiment of the present invention, after constructing the multimodal entity instance set, the method further includes:

[0115] Obtain a set of multimodal entity instances constructed respectively for each target industry field;

[0116] Adding the original guidance unit corresponding to the guidance optimization module to the second integrated model to obtain an initial description target value analysis model; the model complexity of the first integrated model exceeds that of the second integrated model;

[0117] Based on each set of multimodal entity instances, multiple rounds of optimization learning are performed on the original guidance unit in the initial description target value analysis model to obtain a target description target value analysis model consisting of the trained target guidance unit and the second integrated model.

[0118] In the embodiment of the present invention, for example, it is assumed that there are two target industry fields, one is the smartphone industry and the other is the smart car industry. For the smartphone industry, a set of multimodal entity instances has been constructed in accordance with the steps. The multimodal entity instances in this set include a combination of smartphone-related questions (such as performance evaluation, photo effects, etc.) and answers (from user comments, official documents, etc.), as well as corresponding target value instances (such as target values ​​such as accuracy and relevance of answers). For example, a multimodal entity instance is: the question "What factors affect the battery life of smartphones?", and the answer "Battery capacity, screen resolution, processor power consumption, etc. will affect battery life. Large-capacity batteries can store more power, and high-resolution screens and high-performance processors will consume more power." The target value instance is "high" for accuracy and "high" for relevance. For the smart car industry, a set of multimodal entity instances is also constructed. The examples involve issues such as the autonomous driving function and battery life of smart cars. For example, the question "How are the levels of autonomous driving for smart cars divided?" is answered with "According to international standards, autonomous driving is divided into L0-L5 levels, L0 is without autonomous driving assistance, L5 is fully autonomous driving, etc.", and the target value instance is "medium" accuracy and "high" relevance. The guidance optimization module includes an original guidance unit, which can be a simple rule engine or a small neural network structure, and its function is to preliminarily guide and process the data. The second integrated model is a relatively simple model, for example, it can be composed of several basic linear models or simple decision tree models. Compared with the first integrated model, its model complexity is lower. The first integrated model can include complex deep learning architectures, such as a combination of multi-layer convolutional neural networks and recurrent neural networks, for processing complex multimodal information. The server adds the original guidance unit to the second integrated model. For example, if the second integrated model is a model composed of two linear regression models for predicting target values ​​(such as answer accuracy), the original guidance unit preprocesses the input data (data in the multimodal entity instance set) according to some pre-set rules, such as simple classification or feature screening of the data. The initial description target value analysis model obtained after the addition has the ability to preliminarily process multimodal entity instance data. In the first round of optimization learning, the server inputs data from the multimodal entity instance sets of the smartphone industry and the smart car industry into the initial description target value analysis model. For multimodal entity instance data in the smartphone industry, such as the aforementioned instance of battery life, the original guide unit in the model first processes the data. It may extract keywords from questions and answers according to pre-set rules, such as extracting keywords such as "battery life" and "battery capacity". Then, the second integrated model makes predictions based on these processed data to predict the target value of the answer (accuracy, relevance, etc.).Since the prediction of the initial model may be inaccurate, errors will occur. This error will be fed back to the original guidance unit, and the original guidance unit will adjust its own rules or parameters according to the error. For example, if the predicted accuracy target value is significantly different from the actual accuracy target value, the original guidance unit will adjust the keyword extraction rules, add some relevant keywords or adjust the weight of the keywords. In subsequent rounds, the server continues to use the data in the multimodal entity instance set to optimize the model. As the rounds increase, the original guidance unit continues to adjust its parameters and rules, gradually improving its data processing capabilities. After multiple rounds of optimization learning, the original guidance unit becomes the target guidance unit that completes the training. At this time, the target description target value analysis model composed of this target guidance unit and the second integrated model can more accurately process the multimodal entity instance data and make more accurate predictions on the target value (accuracy, relevance, etc.) of the answer. For example, for multimodal entity instance data in the new smartphone industry or smart car industry, the target description target value analysis model can give a target value prediction that is closer to the actual situation, thereby providing more effective support for the training and optimization of the multimodal industry question and answer large model.

[0119] In the embodiment of the present invention, after obtaining the target description target value analysis model composed of the trained target guidance unit and the second integrated model, the method further includes:

[0120] Obtain candidate answer content collected for each target industry field;

[0121] After optimizing the matching target multimodal problem according to the description target value range corresponding to the target industry field, a candidate content is combined with the optimized target multimodal problem to generate the corresponding candidate problem description content, and based on the target description target value analysis model, the corresponding description target value is output for the candidate problem description content.

[0122] In an embodiment of the present invention, illustratively, the server collects candidate answer content from multiple data sources for the smartphone industry. For example, a large amount of answer content is obtained from the customer service records of mobile phone manufacturers. These answer contents can be answers from customer service personnel to various questions about smartphones for users, such as "If your phone is heating up, it may be because there are too many programs running in the background. You can solve this problem by closing some unnecessary background programs." At the same time, the server will also collect answer content generated by exchanges between users from some technical forums. For example, "I found that adjusting the screen refresh rate of the mobile phone to 60Hz can save power, and at the same time has little effect on the visual experience of daily use." In addition, content will also be obtained from the answer part of some professional technology media review articles. For example, "The appearance design of this smartphone adopts a brand-new process, the texture of the glass back panel is very good, and it is not easy to get fingerprints." These collected contents constitute the candidate answer content of the smartphone industry. In the smart car industry, the server obtains candidate answer content from the records of the after-sales service center of the car manufacturer. For example, "If your smart car's automatic driving assistance system fails, first check whether the sensor in front of the vehicle is blocked." From car-related forums and communities, collect user-shared content as candidate answers, such as "I think the operating interface of the smart car's in-car entertainment system can be more concise and intuitive. The operation steps of some functions are too cumbersome now." Content is also obtained from car review media, such as "The battery life of this smart car can reach about 400 kilometers in urban conditions in actual tests, but it will decrease in highway conditions." For the smartphone industry, it is assumed that there is a range of descriptive target values. For example, in terms of accuracy, the accuracy target value of the answer content is required to be above 0.8 (the accuracy here can be determined by previously trained models or manual annotation). Target multimodal questions such as "How to improve the battery life of smartphones?" If the accuracy of the answer to this question is found to be improved in the previous analysis, the server will tune it. The optimization method can be to add more restrictions, such as changing it to "How to improve the battery life of smartphones without affecting the performance of the phone?" Then, the server selects a candidate answer content, such as "You can improve the battery life by reducing the screen brightness, turning off unnecessary notifications and background applications. At the same time, you can also turn on the phone's power saving mode, but it will have some impact on the user experience of some functions." This candidate answer content is combined with the optimized target multimodal question to generate a candidate question description content: "For the question 'How to improve the battery life of smartphones without affecting the performance of the phone?', the answer is 'You can improve the battery life by reducing the screen brightness, turning off unnecessary notifications and background applications. At the same time, you can also turn on the phone's power saving mode, but it will have some impact on the user experience of some functions.'"In the smart car industry, it is assumed that the description target value range is required to reach 0.7 or above in terms of correlation. For example, for a target multimodal question such as "In what aspects is the safety performance of a smart car reflected?" If it is found that the correlation between this question and some answers is not ideal, the server will tune it. For example, change it to "In what aspects is the safety performance of a smart car reflected in active safety (such as anti-collision system, lane keeping, etc.) and passive safety (such as airbags, body structure, etc.)?" Select a candidate answer content, such as "The active safety performance of a smart car is reflected in its anti-collision warning system, which can detect obstacles in front in advance and brake in time, and the lane keeping system can prevent the vehicle from deviating from the lane. In terms of passive safety, the high-strength body structure and multiple airbags can protect the passengers in the car in the event of a collision." Combine this candidate answer content with the tuned target multimodal question to generate a candidate question description content: "For the question 'In what aspects is the safety performance of a smart car reflected in active safety (such as anti-collision system, lane keeping, etc.) and passive safety (such as airbags, body structure, etc.)? ', the answer is 'The active safety performance of smart cars is reflected in the anti-collision warning system that can detect obstacles ahead and brake in time, and the lane keeping system can prevent the vehicle from deviating from the lane. In terms of passive safety, the high-strength body structure and multiple airbags can protect the passengers in the car in the event of a collision. The server inputs the candidate question description content generated by the smartphone industry into the target description target value analysis model. The target guidance unit in this model performs preliminary processing on the input content, such as extracting key information, such as keywords such as "lower screen brightness" and "close background applications", as well as concepts related to battery life. Then, the second integrated model outputs the description target value corresponding to the candidate question description content based on the processed information and the parameters and rules obtained by previous training. For example, 0.85 is output in terms of accuracy, indicating that the answer meets the requirements in terms of accuracy; 0.9 is output in terms of relevance, indicating that the answer is highly relevant to the question. The candidate question description content for the smart car industry is also input into the target description target value analysis model. The target guidance unit processes the input, such as identifying key information such as "anti-collision warning system" and "airbag" and concepts related to automobile safety performance. The second integrated model outputs the description target value based on this information. For example, 0.8 is output in terms of accuracy, indicating that the answer is highly accurate; 0.75 is output in terms of relevance, indicating that the answer is highly relevant to the question. These output description target values ​​can be used to further evaluate the quality of the candidate answer content and provide a basis for the optimization or screening of the question-answering system.

[0123] The embodiment of the present invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned multi-modal industry question-answering large model training method and system. Figure 2 As shown, Figure 2 The block diagram of the computer device 100 provided in the embodiment of the present invention. The computer device 100 includes a memory 111, a processor 112 and a communication unit 113. To achieve data transmission or interaction, the memory 111, the processor 112 and the communication unit 113 are directly or indirectly electrically connected to each other. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0124] For illustrative purposes, the foregoing description is made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Numerous modifications and variations are possible in accordance with the above teachings. These embodiments are selected and described in order to best illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can best utilize the present disclosure and utilize various embodiments with different modifications to suit the intended specific application.

Claims

1. A multi-modal industry question-answering large model training method, characterized in that: include: Obtaining a multi-relationship graph and industry description information of each multimodal entity in the multi-relationship graph, wherein the multi-relationship graph includes a plurality of multimodal entities and there is a connection relationship between two multimodal entities with a certain degree of association, wherein the plurality of multimodal entities include a plurality of multimodal question entities representing questions and a plurality of multimodal answer entities representing pending preferred answers, wherein the plurality of multimodal question entities include at least one focus multimodal question entity, and the focus multimodal question entity has a connection relationship with at least one multimodal answer entity and at least one other multimodal question entity, wherein the industry description information is multimodal description information of any one of the electronic product manufacturing industry, online education industry, tourism industry, catering industry, e-commerce industry, clothing industry, hotel industry, financial investment industry, smart car industry and smart phone industry, and the multimodal description information includes text description information, picture description information and video description information; Confirm at least one multimodal entity instance from the multi-relation graph, for each multimodal entity instance, obtain at least one link of a multimodal entity with the multimodal entity instance as a core node from the multi-relation graph, the first-level adjacent multimodal entities on a link are all multimodal question entities or multimodal answer entities, and the link in which the first-level adjacent multimodal entities are all multimodal question entities is a first link, and the link in which the first-level adjacent multimodal entities are all multimodal answer entities is a second link, the at least one multimodal entity instance includes at least one focus multimodal question entity, each link of the focus multimodal question entity includes at least one first link and at least one second link, and obtain features of the multimodal entity instance corresponding to a reference link, wherein if the reference link is the first link, the corresponding features are initial features, and if the reference link is the second link, the corresponding features are significant features; According to the industry description information of the multimodal entity on each link of each multimodal entity instance, the original multimodal industry question-answering big model and the original feature conversion network are optimized and learned to obtain the target multimodal industry question-answering big model and the target feature conversion network; Among them, the overall error of the optimized learning includes the first learning error of each focal multimodal problem entity, and the first learning error is used to associate the target feature of the corresponding focal multimodal problem entity in each learning cycle with the difference information between the converted features obtained by the original feature conversion network according to the initial feature conversion of the corresponding focal multimodal problem entity in each learning cycle; the first learning error of the focal multimodal problem entity is obtained by the following process: according to the similarity measurement of each group core node and the target feature of the focal multimodal problem entity in this round of learning cycle, the core node features of each group core node are linearly superimposed to determine them as the first superimposed features, and the target features are enhanced according to the first superimposed features to obtain the target features after feature enhancement, and then The target feature after feature enhancement is determined as the focus feature of the focus multimodal problem entity in this round of learning cycle; according to the similarity measurement between each group core node and the initial feature of the focus multimodal problem entity in this round of learning cycle, the core node feature of each group core node is linearly superimposed to be determined as the second superimposed feature; according to the second superimposed feature, the initial feature is subjected to feature enhancement operation to obtain the initial feature after feature enhancement; the initial feature after feature enhancement is transformed through the original feature conversion network to obtain the conversion feature of the focus multimodal problem entity in this round of learning cycle; according to the difference information between the focus feature and the conversion feature of the focus multimodal problem entity in this round of learning cycle, the first learning error of the focus multimodal problem entity is obtained; The target feature is a multimodal entity feature; the original multimodal industry question-and-answer big model is used to input the industry description information of the multimodal entity on each link of the multimodal entity instance, and output the multimodal entity feature of the multimodal entity instance; if each link of the multimodal entity instance includes at least one first link, then the multimodal entity feature includes the initial feature obtained based on the industry description information of the multimodal entity on each first link; if each link of the multimodal entity instance includes at least one second link, then the multimodal entity feature includes the significant feature obtained based on the industry description information of the multimodal entity on each second link.

2. The method according to claim 1, characterized in that The at least one multimodal entity instance further comprises a multimodal answer entity; The overall error of the optimization learning also includes the second learning error of each focal multimodal problem entity; In each cycle of optimization learning, the second error parameter of each focal multimodal problem entity is obtained by the following process: A multimodal answer entity having a connection relationship with the focus multimodal question entity is selected from the multi-relationship graph to be determined as a target multimodal answer entity, and a multimodal answer entity having no connection relationship with the focus multimodal question entity is selected from the multi-relationship graph to be determined as a non-target multimodal answer entity; Polling each target multimodal answer entity and each non-target multimodal answer entity to obtain a plurality of multimodal answer entity pairs, each multimodal answer entity pair including a target multimodal answer entity and a non-target multimodal answer entity; For each multimodal answer entity pair, a first matching coefficient of the focus multimodal question entity and the target multimodal answer entity in the multimodal answer entity pair is obtained according to the multimodal entity features of the focus multimodal question entity and the target multimodal answer entity in the multimodal answer entity pair in the current learning cycle, a second matching coefficient of the focus multimodal question entity and the non-target multimodal answer entity is obtained according to the multimodal entity features of the focus multimodal question entity and the non-target multimodal answer entity in the multimodal answer entity pair in the current learning cycle, and a matching error of the focus multimodal question entity for the multimodal answer entity pair is obtained according to the difference value of the distinguishing information between the first matching coefficient and the second matching coefficient; A second learning error of the focus multimodal question entity is obtained according to the sum of the matching errors of the focus multimodal question entity to all multimodal answer entity pairs.

3. The method according to claim 1, characterized in that The overall error of the optimization learning also includes a third learning error of the multimodal problem entity in each multimodal entity instance; The third learning error of the multimodal problem entity in each multimodal entity instance is obtained as follows: For each cycle of optimization learning, the core node features of the group core nodes of each category group in this round of learning cycle and the multimodal entity features of each multimodal problem entity in this round of learning cycle are obtained, wherein the core node features of the group core nodes of each category group in the first group category division are preset parameters; For each multimodal question entity, a matching coefficient between the multimodal question entity and the corresponding group core node is obtained according to the multimodal entity features of the multimodal question entity in the current learning cycle and the core node features of each group core node; For each group core node, the sum of the matching coefficients of each sample multimodal problem entity corresponding to the group core node is determined as the cohesion of the category group corresponding to the group core node; For each multimodal question entity, the importance of the multimodal question entity in the category group is obtained according to the matching coefficient between the multimodal question entity and a core node of a group, and the cohesion of the category group corresponding to the core node of the group; For each multimodal question entity, the proportion of the importance of the multimodal question entity in a category group to the total importance of the multimodal question entity in all category groups is determined as the probability that the multimodal question entity is classified into the category group; For each multimodal question entity, obtaining a third learning error of the multimodal question entity according to difference information between a matching coefficient between the multimodal question entity and each group core node and a probability that the multimodal question entity is classified into a corresponding category group; Each cycle of optimization learning also includes: For each multimodal question entity, determining the group core node with the highest matching coefficient corresponding to the multimodal question entity as the target group core node, and determining the category group corresponding to the target group core node as the category group corresponding to the multimodal question entity in the next learning cycle; For each category group in the next learning cycle, the core node characteristics of the core nodes of the corresponding group are adjusted according to the overall error of the optimization learning.

4. The method according to claim 1, characterized in that: The multiple multimodal question entities include at least one non-focus multimodal question entity, and the non-focus multimodal question entity is a multimodal question entity that has no connection relationship with the multimodal answer entity; At least one multimodal entity instance also includes a non-focus multimodal problem entity, the links of the non-focus multimodal problem entity are all first links, and the multimodal entity features are initial features; The initial features or salient features of each multimodal entity instance are obtained by the following process: From each link of the multimodal entity instance being a core node multimodal entity, obtaining each reference link, the reference links are all first links or all second links; For each reference link, starting from the highest-level multimodal entity in the reference link, according to the group category characteristics of each higher-level adjacent multimodal entity connected to the multimodal entity and the industry description information configured by the multimodal entity, the core node multimodal entity group is divided by level until the group category characteristics of the core node multimodal entity on the reference link are obtained; wherein the group category characteristics of the highest-level adjacent multimodal entity are obtained according to the industry description information configured by the highest-level adjacent multimodal entity; Based on the local perceptron, the group category features of multimodal entity instances on each benchmark link are divided into groups with weights; Each level of grouping in the hierarchical multimodal entity grouping of core nodes is implemented in the following way: For any multimodal entity divided into target-level groups, obtaining industry description features of the multimodal entity according to industry description information of the multimodal entity; For any multimodal entity divided into target-level groups, each adjacent multimodal entity at a higher level that is connected to the multimodal entity is determined as a reference multimodal entity, and a group category feature of the reference multimodal entity is obtained; For any multimodal entity divided into target-level groups, according to the industry description feature of the multimodal entity and the group category feature of each benchmark multimodal entity, obtain the degree of association between the multimodal entity and each benchmark multimodal entity; For any multimodal entity divided into target-level groups, linear superposition is performed on the group category features of each benchmark multimodal entity according to the degree of association corresponding to each benchmark multimodal entity, and the group category features of the multimodal entity are obtained based on the result of linear superposition and the industry description features of the multimodal entity.

5. The method according to claim 1, characterized in that The method further comprises: Constructing a multi-relationship graph and obtaining industry description information of each multimodal entity in the multi-relationship graph, wherein the plurality of multimodal problem entities include a target multimodal problem entity representing a target problem; Acquire in the multi-relationship graph at least one third link with the target multimodal question entity as a core node multimodal entity, and at least one fourth link with the multimodal answer entity of the to-be-determined preferred answer as a core node multimodal entity; Input each third link corresponding to the target multimodal question entity and the industry description information of the multimodal entity on each third link into the target multimodal industry question-answering big model to obtain the multimodal entity features of the target multimodal question entity, input each fourth link corresponding to each pending preferred answer and the industry description information of the multimodal entity on each fourth link into the target multimodal industry question-answering big model to obtain the multimodal entity features of each pending preferred answer; If all the multimodal entities connected to the target multimodal question entity are multimodal answer entities, then each third link of the target question is obtained as a first link, and the multimodal entity features of the target multimodal question entity are converted through a target conversion network to obtain conversion features, and for each pending preferred answer, a matching coefficient between the target question and the pending preferred answer is obtained based on the conversion features and the multimodal entity features of the pending preferred answer; According to the matching coefficient corresponding to each pending preferred answer, a target preferred answer recommended to the target question is obtained from each pending preferred answer; If the multimodal entities that are connected to the target multimodal question entity include a multimodal answer entity and a multimodal question entity, then each third link of the target question is obtained, including the first link and the second link. For each pending preferred answer, the matching coefficient between the target question and the pending preferred answer is obtained based on the multimodal entity features of the target question and the pending preferred answer.

6. The method according to claim 1, characterized in that The multimodal entity instance is obtained by: For the target industry field, obtain the benchmark answer content that has been marked with the target value, and obtain the unmarked pending answer content; The respective benchmark answer contents are combined with the original multimodal questions obtained for the target industry field to generate corresponding original question description contents, and the inferred target values ​​corresponding to the respective original question description contents are outputted through the first integrated model, and the precision corresponding to the first integrated model is calculated based on the inferred target values ​​and the corresponding description target values, and the recall of the first integrated model under various description target values ​​in the target industry field is calculated; the average recall obtained based on the recall rates and the precision are used as the analysis results of the first integrated model in the target industry field; When the analysis result does not meet the preset requirements, each pre-configured content added to the training description is added to the original multimodal question to generate a corresponding pending multimodal question, and each benchmark answer content is combined with a pending multimodal question to generate a corresponding pending question description content, and the inferred target value corresponding to each pending question description content is output through the first integrated model, and the precision rate corresponding to the first integrated model is calculated based on each inferred target value and the corresponding description target value, and the recall rate of the first integrated model under various description target values ​​in the target industry field is calculated; the average recall rate obtained based on each recall rate and the precision rate are used as the analysis result of the first integrated model in the target industry field; Among the analysis results, a target analysis result that meets the preset requirements is selected, and the undetermined multimodal problem corresponding to the target analysis result is used as the target multimodal problem; Combining each pending answer content with the target multimodal question to generate a corresponding question description content instance, and outputting a target value instance corresponding to each question description content instance through the first integrated model, and obtaining a target value instance corresponding to a question description content instance, and obtaining each similar target value containing the target value instance based on a pre-configured similar target value identifier, and combining each similar target value with the question description content instance to obtain each multimodal entity instance; the similar target value identifier is used to associate other target values ​​similar to a target value; The obtained multimodal entity instances are combined to obtain a multimodal entity instance set.

7. The method according to claim 6, characterized in that The obtaining of unmarked pending answer contents includes: Obtain the unmarked initial answer content; According to the matching coefficient between each two initial answer contents, the initial answer contents meeting similar conditions are clustered to obtain initial answer content groups, and one initial answer content is randomly confirmed in each initial answer content group to be determined as the pending answer content; After outputting the target value instances corresponding to the problem description content instances respectively and before obtaining the target value instance corresponding to the problem description content instance, the method further includes: When there are other initial answer contents except one pending answer content in the initial answer content group to be obtained, the other initial answer contents are respectively combined with the target multimodal question to generate a corresponding question description content instance, and the target value instance obtained corresponding to the one pending answer content is respectively determined as the target value instance of each question description content instance generated corresponding to the initial answer content group.

8. The method according to claim 6, characterized in that The obtaining of unmarked pending answer contents includes: Obtaining each unmarked initial answer content, and obtaining each core content and the corresponding similar identifiers and similar content identifiers; the similar identifier and similar content identifier corresponding to a core content is used to associate the associated content corresponding to the core content; Based on the core contents, matching and identifying are performed in an initial answer content; When obtaining the matching target core content based on the recognition result, obtaining the corresponding similar core contents based on the similar identifier corresponding to the target core content, and replacing the target core content in the initial answer content with the similar core contents to obtain the pending answer contents; Calculating the content matching coefficients between each of the pending answer contents and the one initial answer content respectively, and taking the pending answer content whose content matching coefficient meets the selection condition as the enhanced answer content; The initial answer contents and the enhanced answer contents are regarded as pending answer contents.

9. The method according to claim 6, characterized in that After the combination obtains a set of multimodal entity instances, the method further includes: Obtain a set of multimodal entity instances constructed respectively for each target industry field; Adding the original guidance unit corresponding to the guidance optimization module to the second integrated model to obtain an initial description target value analysis model; the model complexity of the first integrated model exceeds that of the second integrated model; Based on each multimodal entity instance set, performing multiple rounds of optimization learning on the original guide unit in the initial description target value analysis model to obtain a target description target value analysis model consisting of the trained target guide unit and the second integrated model; Obtain candidate answer content collected for each target industry field; After optimizing the matching target multimodal problem according to the description target value range corresponding to the target industry field, a candidate content is combined with the optimized target multimodal problem to generate the corresponding candidate problem description content, and based on the target description target value analysis model, the corresponding description target value is output for the candidate problem description content.

10. A server system, characterized in that: The method comprises a server, wherein the server is used to execute the method described in any one of claims 1 to 9.

Images