Complaint detection method and device based on multi-modal model, equipment and storage medium

Abstract

The present application relates to the technical field of artificial intelligence, and can be applied to the field of financial technology, and discloses a complaint detection method and device based on a multi-modal model, equipment and a storage medium, the method comprising: determining a first training data set and a second training data set based on preset audio data, preset video data and preset text data in preset evaluation data; obtaining a first loss value of the multi-modal model on the first training data set, and obtaining a second loss value of the multi-modal model on the second training data set; when the first loss value or the second loss value meets a preset condition, obtaining a predicted complaint detection result output by the multi-modal model based on the preset evaluation data; training the multi-modal model based on the difference between the predicted complaint detection result and an actual complaint detection result; and obtaining a current complaint detection result output by the trained multi-modal model based on current evaluation data. The present application is beneficial to improving the complaint detection efficiency of the current evaluation data.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology and can be applied to the field of financial technology. In particular, this invention relates to a complaint detection method, apparatus, device, and storage medium based on a multimodal model. Background Technology

[0002] As evaluation data gains increasing influence on purchasing decisions, monitoring current evaluation data for target products through complaint analysis can reveal specific issues raised by users. By addressing these issues and improving the target product, companies can respond quickly to and resolve user pain points, thereby enhancing the user experience.

[0003] However, the current complaint detection process for evaluation data is cumbersome, hindering the improvement of complaint detection efficiency. This is because existing technology relies on manual operation to detect complaints on the current evaluation data, which consumes significant human and time resources, thus increasing the time required for complaint detection and consequently hindering efficiency. Summary of the Invention

[0004] This invention provides a complaint detection method, apparatus, computer equipment, and storage medium based on a multimodal model to solve the technical problem that the current complaint detection process for evaluation data is cumbersome and not conducive to improving the efficiency of complaint detection results.

[0005] Firstly, a complaint detection method based on a multimodal model is provided, including:

[0006] Obtain the preset evaluation data of the target product and the actual complaint test results corresponding to the preset evaluation data;

[0007] Based on the preset audio data, preset video data, and preset text data in the preset evaluation data, a first training dataset and a second training dataset are determined.

[0008] Obtain the first loss value of the preset multimodal model on the first training dataset, and obtain the second loss value of the multimodal model on the second training dataset;

[0009] When the first loss value or the second loss value meets the preset conditions, the predicted complaint detection result output by the multimodal model based on the preset evaluation data is obtained;

[0010] Based on the difference between the predicted complaint detection results and the actual complaint detection results, the multimodal model is trained, and the trained multimodal model is determined.

[0011] Obtain the current evaluation data of the target product, and obtain the current complaint detection result output by the trained multimodal model based on the current evaluation data.

[0012] Further, determining the first training dataset and the second training dataset based on the preset audio data, preset video data, and preset text data in the preset evaluation data includes:

[0013] Based on the preset text data, obtain multiple positive sample pairs and multiple negative sample pairs corresponding to the preset audio data, and obtain multiple positive sample pairs and multiple negative sample pairs corresponding to the preset video data;

[0014] A first training dataset is formed based on multiple positive sample pairs and multiple negative sample pairs corresponding to the preset audio data, and a second training dataset is formed based on multiple positive sample pairs and multiple negative sample pairs corresponding to the preset video data.

[0015] Further, obtaining the first loss value of the preset multimodal model on the first training dataset and obtaining the second loss value of the multimodal model on the second training dataset includes:

[0016] The first training dataset is input into a preset multimodal model, and a preset contrastive loss function is used to obtain the first loss value of the preset multimodal model on the first training dataset.

[0017] The second training dataset is input into the multimodal model, and the contrastive loss function is used to obtain the second loss value of the multimodal model on the second training dataset.

[0018] Further, the step of obtaining the predicted complaint detection result output by the multimodal model based on the preset evaluation data when the first loss value or the second loss value meets the preset conditions includes:

[0019] When the first loss value or the second loss value is less than the first preset value, feature extraction is performed on the preset audio data to obtain the first audio feature, feature extraction is performed on the pre-video data to obtain the first video feature, and feature extraction is performed on the preset text data to obtain the first text feature.

[0020] The first audio feature, the first video feature, and the first text feature are fused to obtain the first target feature. The first target feature is then input into the multimodal model to obtain the predicted complaint detection result output by the multimodal model based on the first target feature.

[0021] Further, the step of training the multimodal model based on the difference between the predicted complaint detection result and the actual complaint detection result, and determining the trained multimodal model, includes:

[0022] Obtain the predicted complaint status identifier and predicted complaint category identifier from the predicted complaint detection result; obtain the actual complaint status identifier and actual complaint category identifier from the actual complaint detection result; obtain the third loss value between the predicted complaint status identifier and the actual complaint status identifier; and obtain the fourth loss value between the predicted complaint category identifier and the actual complaint category identifier.

[0023] The first loss value, the second loss value, the third loss value, and the fourth loss value are added together to generate a total loss value. The multimodal model is trained based on the total loss value. When the total loss value is less than a second preset value, the training of the multimodal model is stopped, and the trained multimodal model is saved.

[0024] Further, obtaining the current evaluation data of the target product and obtaining the current complaint detection result output by the trained multimodal model based on the current evaluation data includes:

[0025] Obtain the current evaluation data of the target product, extract features from the current audio data in the current evaluation data to obtain the second audio feature, extract features from the current video data in the current evaluation data to obtain the second video feature, and extract features from the current text data in the current evaluation data to obtain the second text feature;

[0026] The second audio feature, the second video feature, and the second text feature are fused to obtain the second target feature. The second target feature is then input into the trained multimodal model to obtain the current complaint detection result output by the trained multimodal model based on the second target feature.

[0027] Further, after obtaining the current evaluation data of the target product and the current complaint detection result output by the trained multimodal model based on the current evaluation data, the complaint detection method includes:

[0028] Obtain the display page corresponding to the target product, and display the current complaint detection result on the display page. The current complaint detection result includes the current complaint status identifier and the current complaint category identifier.

[0029] Secondly, a complaint detection device based on a multimodal model is provided, comprising:

[0030] The first acquisition module is used to acquire preset evaluation data of the target product and the actual complaint detection results corresponding to the preset evaluation data;

[0031] The first determining module is used to determine the first training dataset and the second training dataset based on the preset audio data, preset video data and preset text data in the preset evaluation data;

[0032] The second acquisition module is used to acquire a first loss value of a preset multimodal model on the first training dataset and to acquire a second loss value of the multimodal model on the second training dataset.

[0033] The third acquisition module is used to acquire the predicted complaint detection result output by the multimodal model based on the preset evaluation data when the first loss value or the second loss value meets the preset conditions.

[0034] The second determining module is used to train the multimodal model based on the difference between the predicted complaint detection result and the actual complaint detection result, and to determine the trained multimodal model.

[0035] The detection module is used to obtain the current evaluation data of the target product and the current complaint detection result output by the trained multimodal model based on the current evaluation data.

[0036] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described complaint detection method.

[0037] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described complaint detection method.

[0038] This application provides a complaint detection method, apparatus, computer device, and storage medium based on a multimodal model. The advantages are twofold: firstly, by acquiring the current evaluation data of the target product and obtaining the current complaint detection result output by the trained multimodal model based on the current evaluation data, the time required for complaint detection of the current evaluation data is reduced, thus improving the efficiency of complaint detection; secondly, because the multimodal model is not affected by human intervention, the reliability of the current complaint detection result obtained from the current evaluation data is improved. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a schematic diagram of an application environment for a complaint detection method according to an embodiment of the present invention;

[0041] Figure 2 This is a schematic flowchart of a complaint detection method provided in an embodiment of the present invention;

[0042] Figure 3 yes Figure 1 A flowchart illustrating a specific implementation of step S23;

[0043] Figure 4 yes Figure 1 A schematic diagram of a specific implementation method for step S25;

[0044] Figure 5 yes Figure 1 A schematic diagram of a specific implementation method for step S26;

[0045] Figure 6 This is a schematic diagram of a complaint detection device according to an embodiment of the present invention;

[0046] Figure 7 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0047] Figure 8 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Please see Figure 1 , Figure 1 This is a schematic diagram of an application environment for a complaint detection method according to an embodiment of the present invention. The complaint detection method provided in this embodiment of the present invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network.

[0050] The server obtains the preset evaluation data of the target product and the actual complaint test results corresponding to the preset evaluation data through the client.

[0051] Based on the preset audio data, preset video data, and preset text data in the preset evaluation data, a first training dataset and a second training dataset are determined.

[0052] Obtain the first loss value of the preset multimodal model on the first training dataset, and obtain the second loss value of the multimodal model on the second training dataset;

[0053] When the first loss value or the second loss value meets the preset conditions, the predicted complaint detection result output by the multimodal model based on the preset evaluation data is obtained;

[0054] Based on the difference between the predicted complaint detection results and the actual complaint detection results, the multimodal model is trained, and the trained multimodal model is determined.

[0055] Obtain the current evaluation data of the target product, and obtain the current complaint detection result output by the trained multimodal model based on the current evaluation data.

[0056] The beneficial effects of the above-mentioned complaint detection method, apparatus, equipment, and medium are twofold. Firstly, by acquiring the current evaluation data of the target product and obtaining the current complaint detection result output by the trained multimodal model based on the current evaluation data, the complaint detection time of the current evaluation data is reduced since no manual operation is required, thus improving the complaint detection efficiency of the current evaluation data. Secondly, since the multimodal model is not affected by human intervention, the reliability of the current complaint detection result of the acquired current evaluation data is improved.

[0057] The device running the client is referred to as the client device.

[0058] Among them, the equipment that runs the server is referred to as: server equipment.

[0059] The client devices may include, but are not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices.

[0060] The server-side device can be implemented using an independent task database or a task database cluster consisting of multiple task databases. The invention will now be described in detail through specific embodiments.

[0061] Please see Figure 2 , Figure 2A flowchart illustrating a complaint detection method according to an embodiment of the present invention includes the following steps:

[0062] S21, Obtain the preset evaluation data of the target product and the actual complaint detection results corresponding to the preset evaluation data;

[0063] The target products include one or a combination of insurance products, financial products, and technology products.

[0064] Insurance products are a portfolio of committed services offered by insurance companies to the market, designed to meet consumers' needs for risk reduction and transfer. Financial products include, but are not limited to, life insurance, health insurance, and property insurance.

[0065] Financial products refer to various financial instruments and services offered by financial institutions or financial markets to meet the investment, financing, risk management, and asset allocation needs of different entities such as individuals, businesses, and governments. Financial products include, but are not limited to, bank deposits, bonds, stocks, and foreign exchange.

[0066] Among them, technology products refer to various new products or equipment developed and manufactured using modern science and technology, including but not limited to smartphones, tablets, wearable devices, etc.

[0067] S22, based on the preset audio data, preset video data, and preset text data in the preset evaluation data, determine the first training dataset and the second training dataset;

[0068] The step of determining the first training dataset and the second training dataset based on the preset audio data, preset video data, and preset text data in the preset evaluation data includes:

[0069] Based on the preset text data, obtain multiple positive sample pairs and multiple negative sample pairs corresponding to the preset audio data, and obtain multiple positive sample pairs and multiple negative sample pairs corresponding to the preset video data;

[0070] A first training dataset is formed based on multiple positive sample pairs and multiple negative sample pairs corresponding to the preset audio data, and a second training dataset is formed based on multiple positive sample pairs and multiple negative sample pairs corresponding to the preset video data.

[0071] It should be noted that, based on the preset text data, obtaining multiple positive sample pairs and multiple negative sample pairs corresponding to the preset audio data, and obtaining multiple positive sample pairs and multiple negative sample pairs corresponding to the preset video data, includes:

[0072] In the preset text data, a text description that matches the preset audio data is selected as the first text description, a text description that does not match the preset audio data is selected as the second text description, a text description that matches the preset video data is selected as the third text description, and a text description that does not match the preset video data is selected as the fourth text description.

[0073] Based on the preset audio data and multiple first text descriptions, multiple positive sample pairs corresponding to the preset audio data are constructed. Based on the preset audio data and multiple second text descriptions, multiple negative sample pairs corresponding to the preset audio data are constructed. Based on the multiple positive sample pairs and the multiple negative sample pairs corresponding to the preset audio data, a first training dataset is formed.

[0074] Based on the preset video data and multiple third text descriptions, multiple positive sample pairs corresponding to the preset video data are constructed. Based on the preset video data and multiple fourth text descriptions, multiple negative sample pairs corresponding to the preset video data are constructed. Based on the multiple positive sample pairs and multiple negative sample pairs corresponding to the preset video data, a second training dataset is formed.

[0075] Among them, the first loss value is an important indicator for measuring the alignment between the preset audio data and the first text description. By minimizing the first loss value, the multimodal model can learn a more accurate alignment method, making the preset audio data and the first text description closer in the feature space.

[0076] The second loss value is an important indicator for measuring the alignment between the preset video data and the third text description. By minimizing the second loss value, the multimodal model can learn a more accurate alignment method, making the preset video data and the third text description closer in the feature space.

[0077] Specifically, obtaining the first loss value of the preset multimodal model on the first training dataset can effectively promote the alignment between the preset audio data and the first text description of the multimodal model. Obtaining the second loss value of the multimodal model on the second training dataset can effectively promote the alignment between the preset video data and the third text description of the multimodal model. This cross-modal alignment enables the multimodal model to better adapt to complex and ever-changing real-world application scenarios.

[0078] S23, obtain the first loss value of the preset multimodal model on the first training dataset, and obtain the second loss value of the multimodal model on the second training dataset;

[0079] S24, when the first loss value or the second loss value meets the preset conditions, obtain the predicted complaint detection result output by the multimodal model based on the preset evaluation data;

[0080] Wherein, the step of obtaining the predicted complaint detection result output by the multimodal model based on the preset evaluation data when the first loss value or the second loss value meets the preset conditions includes:

[0081] When the first loss value or the second loss value is less than the first preset value, feature extraction is performed on the preset audio data to obtain the first audio feature, feature extraction is performed on the pre-video data to obtain the first video feature, and feature extraction is performed on the preset text data to obtain the first text feature.

[0082] The first audio feature, the first video feature, and the first text feature are fused to obtain the first target feature. The first target feature is then input into the multimodal model to obtain the predicted complaint detection result output by the multimodal model based on the first target feature.

[0083] S25, based on the difference between the predicted complaint detection result and the actual complaint detection result, train the multimodal model and determine the trained multimodal model;

[0084] The multimodal model is trained based on the difference between the predicted complaint detection results and the actual complaint detection results. This optimizes the predictive ability of the multimodal model because the model can gradually reduce the difference by continuously comparing and correcting it, thereby improving the accuracy and reliability of the multimodal model's predictions.

[0085] S26, obtain the current evaluation data of the target product, and obtain the current complaint detection result output by the trained multimodal model based on the current evaluation data.

[0086] The target products include one or a combination of insurance products, financial products, and technology products.

[0087] For ease of explanation, the following example is provided:

[0088] For example, if the current evaluation data is the current video content, and the target product is an insurance product, and the insurance product is already being promoted on various video platforms, with users making various comments, complaints, or negative messages regarding the current video content, this method can be applied to video marketing scenarios for insurance products. It can pinpoint and detect user comments and complaints about the current video content related to the insurance product, thereby optimizing the insurance product and improving service quality.

[0089] For example, if the current evaluation data is the current video content, and the target product is a financial product, and the financial product is already being promoted on various video platforms, with users making various comments, complaints, or negative messages regarding the current video content, this method can be applied to video marketing scenarios for financial products. It can pinpoint and detect user comments and complaints about the current video content, thereby optimizing the financial product and improving service quality.

[0090] For example, if the current evaluation data is the current video content, and the target product is a technology product, and the technology product is already being promoted on various video platforms, with users making various comments, complaints, or negative messages regarding the current video content, this method can be applied to video marketing scenarios for technology products. It can pinpoint and detect user comments and complaints about the current video content, thereby optimizing the technology product and improving service quality.

[0091] Wherein, after obtaining the current evaluation data of the target product and obtaining the current complaint detection result output by the trained multimodal model based on the current evaluation data, the complaint detection method includes:

[0092] Obtain the display page corresponding to the target product, and display the current complaint detection result on the display page. The current complaint detection result includes the current complaint status identifier and the current complaint category identifier.

[0093] The current complaint detection result is the complaint detection result predicted by the trained multimodal model.

[0094] The current complaint status identifier is the complaint status identifier predicted by the trained multimodal model.

[0095] The complaint status is indicated as either "complained" or "not complained".

[0096] The current complaint category identifier is the complaint category identifier predicted by the trained multimodal model.

[0097] The complaint category identifiers include, but are not limited to, identifiers for product quality, product description discrepancies, price issues, after-sales service, and pre-sales service.

[0098] In this embodiment of the invention, the beneficial effects are twofold. Firstly, by acquiring the current evaluation data of the target product and obtaining the current complaint detection result output by the trained multimodal model based on the current evaluation data, the complaint detection time of the current evaluation data is reduced since no manual operation is required, thus improving the complaint detection efficiency of the current evaluation data. Secondly, since the multimodal model is not affected by human intervention, the reliability of the current complaint detection result of the acquired current evaluation data is improved.

[0099] Please see Figure 3 , Figure 3 yes Figure 1 A detailed flowchart of a specific implementation method for step S23 is described below:

[0100] S31, input the first training dataset into the preset multimodal model, and use the preset contrastive loss function to obtain the first loss value of the preset multimodal model on the first training dataset;

[0101] S32, input the second training dataset into the multimodal model, and use the contrastive loss function to obtain the second loss value of the multimodal model on the second training dataset.

[0102] In this embodiment of the invention, the first loss value and the second loss value are important indicators for measuring the degree of alignment between multimodal data. By minimizing the first loss value and the second loss value, the multimodal model can learn a more accurate alignment method, thereby improving the multimodal model's ability to understand and process multimodal data.

[0103] Please see Figure 4 , Figure 4 yes Figure 1 A detailed flowchart of a specific implementation method for step S25 is described below:

[0104] S41, obtain the predicted complaint status identifier and predicted complaint category identifier from the predicted complaint detection result, obtain the actual complaint status identifier and actual complaint category identifier from the actual complaint detection result, obtain the third loss value between the predicted complaint status identifier and the actual complaint status identifier, and obtain the fourth loss value between the predicted complaint category identifier and the actual complaint category identifier.

[0105] Among them, the predicted complaint detection results are the complaint detection results predicted by the multimodal model that has not yet completed training.

[0106] Among them, the predicted complaint status identifier is the complaint status identifier predicted by the multimodal model that has not yet completed training.

[0107] Among them, the predicted complaint category identifier is the complaint category identifier predicted by the multimodal model that has not yet completed training.

[0108] The actual complaint test results are those marked as complaint test results.

[0109] The actual complaint status is indicated by the marked complaint status.

[0110] The actual complaint category is identified by the labeled complaint category identifier.

[0111] S42, add the first loss value, the second loss value, the third loss value, and the fourth loss value to generate a total loss value, train the multimodal model based on the total loss value, and stop training the multimodal model when the total loss value is less than a second preset value, and save the trained multimodal model.

[0112] For ease of explanation, the following example is provided:

[0113] Based on a preset loss value generation model, the first loss value, the second loss value, the third loss value, and the fourth loss value are added together to generate a total loss value. The loss value generation model is as follows:

[0114]

[0115] in, This represents the total loss value. Indicates the first loss value, Indicates the second loss value, Indicates the third loss value, This represents the fourth loss value.

[0116] In this embodiment of the invention, the multimodal model that has been trained can be saved and deployed at any time without having to go through the time-consuming and resource-intensive training process again, which greatly improves work efficiency.

[0117] Please see Figure 5 , Figure 5 yes Figure 1 A detailed flowchart of a specific implementation method for step S26 is described below:

[0118] S51, obtain the current evaluation data of the target product, extract features from the current audio data in the current evaluation data to obtain the second audio feature, extract features from the current video data in the current evaluation data to obtain the second video feature, and extract features from the current text data in the current evaluation data to obtain the second text feature;

[0119] S52, the second audio feature, the second video feature, and the second text feature are fused to obtain the second target feature. The second target feature is then input into the trained multimodal model to obtain the current complaint detection result output by the trained multimodal model based on the second target feature.

[0120] In this embodiment of the invention, the current complaint detection result output by the trained multimodal model based on the second target feature is obtained. Since no manual operation is required, the complaint detection time of the current evaluation data is reduced, which is beneficial to improving the complaint detection efficiency of the current evaluation data.

[0121] Please see Figure 6 , Figure 6 This is a schematic diagram of a complaint detection device according to an embodiment of the present invention, as shown below. Figure 6 As shown, the complaint detection device includes a first acquisition module 101, a first determination module 102, a second acquisition module 103, a third acquisition module 104, a second determination module 105, and a detection module 106. Detailed descriptions of each functional module are as follows:

[0122] The first acquisition module 101 is used to acquire preset evaluation data of the target product and the actual complaint detection results corresponding to the preset evaluation data;

[0123] The first determining module 102 is used to determine the first training dataset and the second training dataset based on the preset audio data, preset video data and preset text data in the preset evaluation data;

[0124] The second acquisition module 103 is used to acquire a first loss value of a preset multimodal model on the first training dataset and to acquire a second loss value of the multimodal model on the second training dataset.

[0125] The third acquisition module 104 is used to acquire the predicted complaint detection result output by the multimodal model based on the preset evaluation data when the first loss value or the second loss value meets the preset conditions.

[0126] The second determining module 105 is used to train the multimodal model based on the difference between the predicted complaint detection result and the actual complaint detection result, and to determine the trained multimodal model.

[0127] The detection module 106 is used to obtain the current evaluation data of the target product and obtain the current complaint detection result output by the trained multimodal model based on the current evaluation data.

[0128] In one embodiment, the first determining module 102 includes:

[0129] The first acquisition subunit is used to acquire, based on the preset text data, multiple positive sample pairs and multiple negative sample pairs corresponding to the preset audio data, and multiple positive sample pairs and multiple negative sample pairs corresponding to the preset video data;

[0130] The sub-unit is used to form a first training dataset based on multiple positive sample pairs and multiple negative sample pairs corresponding to the preset audio data, and to form a second training dataset based on multiple positive sample pairs and multiple negative sample pairs corresponding to the preset video data.

[0131] In one embodiment, the second acquisition module 103 includes:

[0132] The second acquisition subunit is used to input the first training dataset into a preset multimodal model and use a preset contrastive loss function to obtain the first loss value of the preset multimodal model on the first training dataset.

[0133] The third acquisition subunit is used to input the second training dataset into the multimodal model and use the contrastive loss function to obtain the second loss value of the multimodal model on the second training dataset.

[0134] In one embodiment, the third acquisition module 104 includes:

[0135] The first extraction subunit is configured to extract features from the preset audio data to obtain first audio features, extract features from the preset video data to obtain first video features, and extract features from the preset text data to obtain first text features when the first loss value or the second loss value is less than a first preset value.

[0136] The input subunit is used to fuse the first audio feature, the first video feature, and the first text feature to obtain the first target feature, input the first target feature into the multimodal model, and obtain the predicted complaint detection result output by the multimodal model based on the first target feature.

[0137] In one embodiment, the second determining module 105 includes:

[0138] The fourth acquisition subunit is used to acquire the predicted complaint status identifier and the predicted complaint category identifier in the predicted complaint detection result, acquire the actual complaint status identifier and the actual complaint category identifier in the actual complaint detection result, acquire the third loss value between the predicted complaint status identifier and the actual complaint status identifier, and acquire the fourth loss value between the predicted complaint category identifier and the actual complaint category identifier.

[0139] A sub-unit is generated to add the first loss value, the second loss value, the third loss value, and the fourth loss value to generate a total loss value. The multimodal model is trained based on the total loss value. When the total loss value is less than a second preset value, the training of the multimodal model is stopped, and the trained multimodal model is saved.

[0140] In one embodiment, the detection module 106 includes:

[0141] The second extraction subunit is used to obtain the current evaluation data of the target product, extract features from the current audio data in the current evaluation data to obtain the second audio feature, extract features from the current video data in the current evaluation data to obtain the second video feature, and extract features from the current text data in the current evaluation data to obtain the second text feature.

[0142] The detection subunit is used to fuse the second audio feature, the second video feature, and the second text feature to obtain the second target feature, input the second target feature into the trained multimodal model, and obtain the current complaint detection result output by the trained multimodal model based on the second target feature.

[0143] In one embodiment, the complaint detection device further includes:

[0144] The display module is used to obtain the display page corresponding to the target product and display the current complaint detection result through the display page. The current complaint detection result includes the current complaint status identifier and the current complaint category identifier.

[0145] In this embodiment of the invention, the beneficial effects are twofold. Firstly, by acquiring the current evaluation data of the target product and obtaining the current complaint detection result output by the trained multimodal model based on the current evaluation data, the complaint detection time of the current evaluation data is reduced since no manual operation is required, thus improving the complaint detection efficiency of the current evaluation data. Secondly, since the multimodal model is not affected by human intervention, the reliability of the current complaint detection result of the acquired current evaluation data is improved.

[0146] For specific limitations on complaint detection devices, please refer to the limitations on complaint detection methods mentioned above, which will not be repeated here.

[0147] Each module in the aforementioned complaint detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0148] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a computer device according to one embodiment of the present invention. In one embodiment, a computer device is provided, which may be a server device, and its internal structure diagram may be as shown below. Figure 7 As shown. This computer device includes a processor, memory, network interface, and database connected via a system bus.

[0149] The processor of this computer device provides computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is used for communication with external clients via a network connection. When the computer program is executed by the processor, it can implement the functions or steps of a complaint detection method based on a multimodal model on the server device.

[0150] Please see Figure 8 , Figure 8 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. In one embodiment, a computer device is provided, which may be a client device, and its internal structure diagram may be as follows. Figure 8 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external task database via a network connection. When the computer program is executed by the processor, it can implement the functions or steps of a complaint detection method based on a multimodal model on a client device.

[0151] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor.

[0152] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0153] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), graphics processing units (GPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0154] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software may depend on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of the embodiments of this disclosure.

Claims

1. A complaint detection method based on a multimodal model, characterized in that, include: Obtain the preset evaluation data of the target product and the actual complaint detection results corresponding to the preset evaluation data. The preset evaluation data consists of videos of the target product used for promotion and their corresponding comments. Based on the preset audio data, preset video data, and preset text data in the preset evaluation data, a first training dataset and a second training dataset are determined. Obtain the first loss value of the preset multimodal model on the first training dataset, and obtain the second loss value of the multimodal model on the second training dataset; When the first loss value or the second loss value meets the preset conditions, the predicted complaint detection result output by the multimodal model based on the preset evaluation data is obtained; Based on the difference between the predicted complaint detection results and the actual complaint detection results, the multimodal model is trained, and the trained multimodal model is determined. Obtain the current evaluation data of the target product, and obtain the current complaint detection result output by the trained multimodal model based on the current evaluation data; Based on the difference between the predicted complaint detection results and the actual complaint detection results, the multimodal model is trained, including: Obtain a third loss value between the predicted complaint status identifier and the actual complaint status identifier, obtain a fourth loss value between the predicted complaint category identifier and the actual complaint category identifier, add the first loss value, the second loss value, the third loss value and the fourth loss value together to generate a total loss value, and train the multimodal model based on the total loss value; Obtaining a first loss value of a preset multimodal model on the first training dataset, and obtaining a second loss value of the multimodal model on the second training dataset, includes: Obtain a first loss value of the preset multimodal model on the first training dataset, and obtain a second loss value of the multimodal model on the second training dataset; wherein, the first loss value is an important indicator for measuring the alignment between preset audio data and first text description; wherein, the second loss value is an important indicator for measuring the alignment between preset video data and third text description; The process of obtaining the first and third text descriptions is as follows: In the preset text data, the text description matching the preset audio data is selected as the first text description, and the text description matching the preset video data is selected as the third text description.

2. The complaint detection method according to claim 1, characterized in that, The step of determining the first training dataset and the second training dataset based on the preset audio data, preset video data, and preset text data in the preset evaluation data includes: Based on the preset text data, obtain multiple positive sample pairs and multiple negative sample pairs corresponding to the preset audio data, and obtain multiple positive sample pairs and multiple negative sample pairs corresponding to the preset video data; A first training dataset is formed based on multiple positive sample pairs and multiple negative sample pairs corresponding to the preset audio data, and a second training dataset is formed based on multiple positive sample pairs and multiple negative sample pairs corresponding to the preset video data.

3. The complaint detection method according to claim 1, characterized in that, The step of obtaining a first loss value of a preset multimodal model on the first training dataset and obtaining a second loss value of the multimodal model on the second training dataset includes: The first training dataset is input into a preset multimodal model, and a preset contrastive loss function is used to obtain the first loss value of the preset multimodal model on the first training dataset. The second training dataset is input into the multimodal model, and the contrastive loss function is used to obtain the second loss value of the multimodal model on the second training dataset.

4. The complaint detection method according to claim 1, characterized in that, When the first loss value or the second loss value meets a preset condition, the step of obtaining the predicted complaint detection result output by the multimodal model based on the preset evaluation data includes: When the first loss value or the second loss value is less than the first preset value, feature extraction is performed on the preset audio data to obtain the first audio feature, feature extraction is performed on the preset video data to obtain the first video feature, and feature extraction is performed on the preset text data to obtain the first text feature. The first audio feature, the first video feature, and the first text feature are fused to obtain the first target feature. The first target feature is then input into the multimodal model to obtain the predicted complaint detection result output by the multimodal model based on the first target feature.

5. The complaint detection method according to claim 1, characterized in that, The step of training the multimodal model based on the difference between the predicted complaint detection result and the actual complaint detection result, and determining the trained multimodal model, includes: Obtain the predicted complaint status identifier and predicted complaint category identifier from the predicted complaint detection result; obtain the actual complaint status identifier and actual complaint category identifier from the actual complaint detection result; obtain the third loss value between the predicted complaint status identifier and the actual complaint status identifier; and obtain the fourth loss value between the predicted complaint category identifier and the actual complaint category identifier. The first loss value, the second loss value, the third loss value, and the fourth loss value are added together to generate a total loss value. The multimodal model is trained based on the total loss value. When the total loss value is less than a second preset value, the training of the multimodal model is stopped, and the trained multimodal model is saved.

6. The complaint detection method according to claim 1, characterized in that, The step of obtaining the current evaluation data of the target product and obtaining the current complaint detection result output by the trained multimodal model based on the current evaluation data includes: Obtain the current evaluation data of the target product, extract features from the current audio data in the current evaluation data to obtain the second audio feature, extract features from the current video data in the current evaluation data to obtain the second video feature, and extract features from the current text data in the current evaluation data to obtain the second text feature; The second audio feature, the second video feature, and the second text feature are fused to obtain the second target feature. The second target feature is then input into the trained multimodal model to obtain the current complaint detection result output by the trained multimodal model based on the second target feature.

7. The complaint detection method according to any one of claims 1 to 6, characterized in that, After obtaining the current evaluation data of the target product and the current complaint detection result output by the trained multimodal model based on the current evaluation data, the complaint detection method includes: Obtain the display page corresponding to the target product, and display the current complaint detection result on the display page. The current complaint detection result includes the current complaint status identifier and the current complaint category identifier.

8. A complaint detection device based on a multimodal model, characterized in that, include: The first acquisition module is used to acquire preset evaluation data of the target product and the actual complaint detection results corresponding to the preset evaluation data. The preset evaluation data consists of videos of the target product used for promotion and their corresponding comments. The first determining module is used to determine the first training dataset and the second training dataset based on the preset audio data, preset video data and preset text data in the preset evaluation data; The second acquisition module is used to acquire a first loss value of a preset multimodal model on the first training dataset and to acquire a second loss value of the multimodal model on the second training dataset. The third acquisition module is used to acquire the predicted complaint detection result output by the multimodal model based on the preset evaluation data when the first loss value or the second loss value meets the preset conditions. The second determining module is used to train the multimodal model based on the difference between the predicted complaint detection result and the actual complaint detection result, and to determine the trained multimodal model. The detection module is used to obtain the current evaluation data of the target product and obtain the current complaint detection result output by the trained multimodal model based on the current evaluation data; Based on the difference between the predicted complaint detection results and the actual complaint detection results, the multimodal model is trained, including: Obtain a third loss value between the predicted complaint status identifier and the actual complaint status identifier, obtain a fourth loss value between the predicted complaint category identifier and the actual complaint category identifier, add the first loss value, the second loss value, the third loss value and the fourth loss value together to generate a total loss value, and train the multimodal model based on the total loss value; Obtaining a first loss value of a preset multimodal model on the first training dataset, and obtaining a second loss value of the multimodal model on the second training dataset, includes: Obtain a first loss value of the preset multimodal model on the first training dataset, and obtain a second loss value of the multimodal model on the second training dataset; wherein, the first loss value is an important indicator for measuring the alignment between preset audio data and first text description; wherein, the second loss value is an important indicator for measuring the alignment between preset video data and third text description; The process of obtaining the first and third text descriptions is as follows: In the preset text data, the text description matching the preset audio data is selected as the first text description, and the text description matching the preset video data is selected as the third text description.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the complaint detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the complaint detection method as described in any one of claims 1 to 7.

Images